Propose to remove workingDir setting from kaniko buildstrategy

[mattcui]

A real use case is one of our end user met a problem with building a Dockerfile image with Source-to-image (based on Shipwright/build), his app source code to be built as an image is in a subdirectory (e.g. /subdir) of a git repo, and Dockerfile used to build the image is in the subdir too, but there is also a Dockerfile under the repo root dir. When he used our Source-to-image to build the image for his app, he specified subdir as the context and Dockerfile for dockerfile property, but the image was built based on the Dockerfile from git repo root directory rather than /subdir, which is not expected by him. He tried with docker build command to build image, run the command under /subdir directory, the image was built with the Dockerfile from /subdir as expected. @SaschaSchwarze0 / @qu1queee and me did lots of investigations on this case, we found for this scenario, docker build and kaniko actually handle the image building with the same behavior, the reason of why the result looked different from user's test is that the command of docker build and kaniko were run from different directories - workingDir.

Talk more here about workingDir: for our shipwright build, the workingDir is specified in buildstrategy, for examples workingDir: /workspace/source, it means kaniko command is always run from /workspace/source as the work dir. According to the rule of resolving Dockerfile location of kaniko, kaniko will try to find the Dockerfile from the work dir firstly, in this case, the user only specified Dockerifle as dockerfile property, so kaniko tried to find <workingDir>/Dockerfile, workingDir is /workspace/source as hardcoded in buildstrategy, and as we know, the git repo source code is also saved in /workspace/source, so kaniko tried to find the Dockerfile from git repo root dir, in user's case, a Dockerfile happened to be there. For the test with docker build by the user, he ran docker build from subdir, so docker could find the Dockerfile from subdir directly, we already verified that we could get the same result as kankio if we run docker build command from git repo root dir. So in this case, if user specifies subdir/Dockerfile as --dockerfile property, the kaniko could resolve the correct
Dockerfile from subdir, but from the user experience perspective, it's not good as the user already specified subdir as --context, so it may bring some confusion that the user has to specify subdir in --dockerfile property again.

To address this use case, we would like to leverage the rule of resolving Dockerfile location of kaniko, we don't want to set /workspace/source as workingDir, we could remove this setting from buildstrategy, tekton hardcoded to set /workspace as workingDir by default. So the work dir is change to /workspace, if we still specify subdir as --context and Dockerfile as --dockerfile, kaniko will still try to find Dockerfile from the work dir, currently, since it's changed to /workspace, there is no Dockerfile, then kankio will try to find the Dockerfile from the context dir, it can find the Dockerfile under subdir, which is what we want to see, as it works same as docker build, so we can have a comparison for 2 basic scenarios between kaniko and docker build to see if they have same behaviors

The 2 basic scenarios:

1. Both source code and Dockerfile are in the git repo root dir (/workspace/source)
   In this case, running kaniko from /workspace has the same result as running docker build with same context and dockerfile. Both kaniko and docker build can find the Dockerfile from git repo root dir (/workspace/source) properly:

/workspace# docker run -ti --rm -v /workspace/source:/workspace/source -w /workspace gcr.io/kaniko-project/executor:v1.3.0 --skip-tls-verify=true --dockerfile=Dockerfile --context=/workspace/source --no-push

/workspace/source# docker build /workspace/source -f Dockerfile

2. Both source code and Dockerfile are in a subdir ( /workspace/source/subdir ) of git repo, it’s the use case we talked at the beginning of this issue. We also can get the same user experience (same context and dockerfile) with kaniko and docker build

/workspace# docker run -ti --rm -v /workspace/source:/workspace/source -w /workspace gcr.io/kaniko-project/executor:v1.3.0 --skip-tls-verify=true --dockerfile=Dockerfile --context=/workspace/source/subdir --no-push

/workspace/source# docker build /workspace/source/subdir -f Dockerfile

There are another 2 scenarios, which we don't think used often by end users, for example:

3. Source code is in root dir (/workspace/source) of git repo, Dockerfile is in a subdir ( /workspace/source/subdir ) of git repo

/workspace# docker run -ti --rm -v /workspace/source:/workspace/source -w /workspace gcr.io/kaniko-project/executor:v1.3.0 --skip-tls-verify=true --dockerfile=subdir/Dockerfile --context=/workspace/source --no-push

/workspace/source# docker build /workspace/source -f subdir/Dockerfile

We can see we still use the same context and dockerfile to keep consistent between kaniko and docker build, but for the other one, we can't guarantee to have the same user experience (same context and dockerfile) , for example

4. Source code is in a subdir ( /workspace/source/subdir ) of git repo, Dockerfile is in root dir (/workspace/source) of git repo, need to specify --context to subdir

/workspace# docker run -ti --rm -v /workspace/source:/workspace/source -w /workspace gcr.io/kaniko-project/executor:v1.3.0 --skip-tls-verify=true --dockerfile=../Dockerfile --context=/workspace/source/subdir --no-push

/workspace/source# docker build /workspace/source/subdir -f Dockerfile

We can see there needs a trick to specify ../Dockerfile for --dockerfile option, because according to the Dockerfile resolution rule of kaniko, it tries to find the Dockerfile from workingDir (/workspace) firstly, the source code is actually in /workspace/source/subdir and Dockerfile is in /workspace/source, so it can't find Dockerfile from /workspace/Dockerfile, then it will try to find the Dockerfile from <context-dir>/<dockerfile>, it's /workspace/source/subdir/../Dockerfile

Although the dockerfile option is different between kaniko and docker build, we thought, as it's not a common use case, we can accept this difference.

So the conclusion is that we propose to remove workingDir: /workspace/source from kaniko buildstrategy.

Sorry for the long story and description, and thanks for your patience, appreciate if you could give us your comments/opinions. Thanks.

In the end, specially much thank Sasha (@SaschaSchwarze0) and Enrique (@qu1queee) for the contribution to this issue, Sasha did most of investigations, created a test repo and have a good summary in Readme, Enrique proposed the solution to remove workdingDir from buildstrategy.

/cc @zhangtbj @xiujuan95