USER command should not require user or group IDs to exist

[hrobertson]

It is common for a Dockerfile to contain a USER directive of the form USER 1000:1000. docker build does not require a user and group with those ids to exist, nor does it create them. However, Kaniko errors.

Interestingly, while creating a minimal dockerfile to demonstrate this I discovered that Kaniko errors in two different ways:

Working example using Docker:

$ cat usertest1
FROM alpine
USER 1000:1000
CMD ["whoami"]
$ docker build --file usertest1 -t usertest:1 .
Sending build context to Docker daemon  23.46MB
Step 1/3 : FROM alpine
 ---> 196d12cf6ab1
Step 2/3 : USER 1000:1000
 ---> Using cache
 ---> 4794c572f720
Step 3/3 : CMD ["whoami"]
 ---> Using cache
 ---> 6ab640ab2641
Successfully built 6ab640ab2641
Successfully tagged usertest:1
$ docker run --rm usertest:1
whoami: unknown uid 1000

Kaniko failure mode 1

$ cat usertest1
FROM alpine
USER 1000:1000
CMD ["whoami"]

$ docker run --rm -it -v $PWD:/workspace gcr.io/kaniko-project/executor:latest --dockerfile=usertest1 --no-push --context=dir:///workspace
INFO[0000] Downloading base image alpine                
2018/12/04 13:25:52 No matching credentials were found, falling back on anonymous
INFO[0001] Taking snapshot of full filesystem...        
INFO[0001] Skipping paths under /dev, as it is a whitelisted directory 
INFO[0001] Skipping paths under /kaniko, as it is a whitelisted directory 
INFO[0001] Skipping paths under /proc, as it is a whitelisted directory 
INFO[0001] Skipping paths under /sys, as it is a whitelisted directory 
INFO[0001] Skipping paths under /workspace, as it is a whitelisted directory 
INFO[0001] USER 1000:1000                               
INFO[0001] cmd: USER                                    
error building image: error building stage: open /etc/passwd: no such file or directory

Kaniko failure mode 2:
Note the addition of the RUN directive and the different error message from Kaniko

$ cat usertest2
FROM alpine
RUN test 0
USER 1000:1000
CMD ["whoami"]

$ docker run --rm -it -v $PWD:/workspace gcr.io/kaniko-project/executor:latest --dockerfile=usertest2 --no-push --context=dir:///workspace
INFO[0000] Downloading base image alpine                
2018/12/04 13:32:00 No matching credentials were found, falling back on anonymous
INFO[0001] Unpacking rootfs as cmd RUN test 0 requires it. 
INFO[0001] Taking snapshot of full filesystem...        
INFO[0001] Skipping paths under /dev, as it is a whitelisted directory 
INFO[0001] Skipping paths under /kaniko, as it is a whitelisted directory 
INFO[0001] Skipping paths under /proc, as it is a whitelisted directory 
INFO[0001] Skipping paths under /sys, as it is a whitelisted directory 
INFO[0001] Skipping paths under /workspace, as it is a whitelisted directory 
INFO[0001] RUN test 0                                   
INFO[0001] cmd: /bin/sh                                 
INFO[0001] args: [-c test 0]                            
INFO[0001] Taking snapshot of full filesystem...        
INFO[0001] Skipping paths under /dev, as it is a whitelisted directory 
INFO[0001] Skipping paths under /kaniko, as it is a whitelisted directory 
INFO[0001] Skipping paths under /proc, as it is a whitelisted directory 
INFO[0001] Skipping paths under /sys, as it is a whitelisted directory 
INFO[0001] Skipping paths under /workspace, as it is a whitelisted directory 
INFO[0001] No files were changed, appending empty layer to config. No layer added to image. 
INFO[0001] USER 1000:1000                               
INFO[0001] cmd: USER                                    
error building image: error building stage: user: unknown userid 1000

[miguelitoq76]

@hrobertson could be so kind to test the build with this modification?

add a FROM scratch at the top of the dockerfile

FROM scratch
FROM alpine
USER 1000:1000
CMD ["whoami"]

Please post your result thanks a lot, it would help to find the error!!

[hrobertson]

@miguelitoq76 I assume you didn't mean to have two FROMs?

$ cat usertest3
FROM scratch
USER 1000:1000
CMD ["whoami"]
$
$ docker run --rm -it -v $PWD:/workspace gcr.io/kaniko-project/executor:latest --dockerfile=usertest3 --no-push --context=dir:///workspace
INFO[0000] No base image, nothing to extract            
INFO[0000] Taking snapshot of full filesystem...        
INFO[0000] Skipping paths under /dev, as it is a whitelisted directory 
INFO[0000] Skipping paths under /kaniko, as it is a whitelisted directory 
INFO[0000] Skipping paths under /proc, as it is a whitelisted directory 
INFO[0000] Skipping paths under /sys, as it is a whitelisted directory 
INFO[0000] Skipping paths under /workspace, as it is a whitelisted directory 
INFO[0000] USER 1000:1000                               
INFO[0000] cmd: USER                                    
error building image: error building stage: open /etc/passwd: no such file or directory

In case you really did mean to have the two FROMs:

$ cat usertest4
FROM scratch
FROM alpine
USER 1000:1000
CMD ["whoami"]
$
$ docker run --rm -it -v $PWD:/workspace gcr.io/kaniko-project/executor:latest --dockerfile=usertest4 --no-push --context=dir:///workspace
INFO[0000] No base image, nothing to extract            
INFO[0000] Taking snapshot of full filesystem...        
INFO[0000] Skipping paths under /dev, as it is a whitelisted directory 
INFO[0000] Skipping paths under /kaniko, as it is a whitelisted directory 
INFO[0000] Skipping paths under /proc, as it is a whitelisted directory 
INFO[0000] Skipping paths under /sys, as it is a whitelisted directory 
INFO[0000] Skipping paths under /workspace, as it is a whitelisted directory 
INFO[0000] Deleting filesystem...                       
INFO[0000] Downloading base image alpine                
2018/12/05 12:44:45 No matching credentials were found, falling back on anonymous
INFO[0002] Taking snapshot of full filesystem...        
INFO[0002] Skipping paths under /dev, as it is a whitelisted directory 
INFO[0002] Skipping paths under /kaniko, as it is a whitelisted directory 
INFO[0002] Skipping paths under /proc, as it is a whitelisted directory 
INFO[0002] Skipping paths under /sys, as it is a whitelisted directory 
INFO[0002] Skipping paths under /workspace, as it is a whitelisted directory 
INFO[0002] USER 1000:1000                               
INFO[0002] cmd: USER                                    
error building image: error building stage: open /etc/passwd: no such file or directory

[hrobertson]

I have done some further investigation and the behaviour is even more erroneous than I first thought!

Here I build three images containing the kaniko executor binary. One pulls the binary into a scratch base, the next into an alpine base, and the next creates a user within the image.

$ cat Dockerfile
FROM gcr.io/kaniko-project/executor:latest as kaniko

FROM scratch as scratch
COPY --from=kaniko /kaniko/executor /kaniko/executor
COPY --from=kaniko /kaniko/ssl/certs/ca-certificates.crt /kaniko/ssl/certs/
ENV SSL_CERT_DIR=/kaniko/ssl/certs
WORKDIR /workspace

ENTRYPOINT ["/kaniko/executor"]

FROM alpine as alpine
COPY --from=kaniko /kaniko/executor /kaniko/executor
COPY --from=kaniko /kaniko/ssl/certs/ca-certificates.crt /kaniko/ssl/certs/
ENV SSL_CERT_DIR=/kaniko/ssl/certs
WORKDIR /workspace

ENTRYPOINT ["/kaniko/executor"]

FROM alpine as alpine-with-user
RUN addgroup -g 1000 test && adduser -D -u 1000 -G test test

I build the three separate images:

$ docker build -t kaniko:scratch --target scratch .
$ docker build -t kaniko:alpine --target alpine .
$ docker build -t kaniko:alpine-with-user --target alpine-with-user .

And here I build the usertest3 image using kaniko inside each of those images.

$ cat usertest3
FROM scratch
USER 1000:1000
CMD ["whoami"]

kaniko binary in scratch

$ docker run --rm -it -v $PWD:/workspace --entrypoint=/kaniko/executor kaniko:scratch --dockerfile=usertest3 --context=dir:///workspace --no-push
INFO[0000] No base image, nothing to extract            
INFO[0000] Taking snapshot of full filesystem...        
INFO[0000] Skipping paths under /dev, as it is a whitelisted directory 
INFO[0000] Skipping paths under /kaniko, as it is a whitelisted directory 
INFO[0000] Skipping paths under /proc, as it is a whitelisted directory 
INFO[0000] Skipping paths under /sys, as it is a whitelisted directory 
INFO[0000] Skipping paths under /workspace, as it is a whitelisted directory 
INFO[0000] USER 1000:1000                               
INFO[0000] cmd: USER                                    
error building image: error building stage: open /etc/passwd: no such file or directory

kaniko binary in alpine

$ docker run --rm -it -v $PWD/test:/workspace --entrypoint=/kaniko/executor kaniko:alpine --dockerfile=usertest3 --context=dir:///workspace --no-push 
INFO[0000] No base image, nothing to extract            
INFO[0000] Taking snapshot of full filesystem...        
INFO[0000] Skipping paths under /dev, as it is a whitelisted directory 
INFO[0000] Skipping paths under /kaniko, as it is a whitelisted directory 
INFO[0000] Skipping paths under /proc, as it is a whitelisted directory 
INFO[0000] Skipping paths under /sys, as it is a whitelisted directory 
INFO[0000] Skipping paths under /var/run, as it is a whitelisted directory 
INFO[0000] Skipping paths under /workspace, as it is a whitelisted directory 
INFO[0000] USER 1000:1000                               
INFO[0000] cmd: USER                                    
error building image: error building stage: user: unknown userid 1000

As you can see, kaniko is erroneously looking for the /etc/passwd file in the environment in which kaniko is running, not the filesystem of the image it's building! Note it shouldn't even look for a passwd file at all when IDs are specified rather than names.

kaniko binary in alpine with an existing user:
Note this user does not exist in the image kaniko is building, but in the environment in which kaniko itself is running.

$ docker run -it --rm -v $PWD:/workspace kaniko:alpine-with-user --dockerfile=usertest3 --context=dir:///workspace --no-push
INFO[0000] Downloading base image alpine                
2018/12/05 13:32:25 Unable to read "/root/.docker/config.json": open /root/.docker/config.json: no such file or directory
2018/12/05 13:32:25 No matching credentials were found, falling back on anonymous
INFO[0001] Taking snapshot of full filesystem...        
INFO[0001] Skipping paths under /dev, as it is a whitelisted directory 
INFO[0001] Skipping paths under /kaniko, as it is a whitelisted directory 
INFO[0001] Skipping paths under /proc, as it is a whitelisted directory 
INFO[0001] Skipping paths under /sys, as it is a whitelisted directory 
INFO[0001] Skipping paths under /var/run, as it is a whitelisted directory 
INFO[0001] Skipping paths under /workspace, as it is a whitelisted directory 
INFO[0001] USER 1000:1000                               
INFO[0001] cmd: USER                                    
INFO[0001] CMD ["whoami"]                               
INFO[0001] Skipping push to container registry due to --no-push flag

Until the first RUN directive causes Unpacking rootfs , kaniko is using the filesystem of the environment that Kaniko is in rather than that of the base image.
As you can see below, using kaniko in the kaniko:alpine-with-user image to build the usertest2 dockerfile (see the initial post), again fails to find the user. Contrast this with the above output building the usertest3 dockerfile.

$ docker run -it --rm -v $PWD:/workspace kaniko:alpine-with-user --dockerfile=usertest2 --context=dir:///workspace --no-push
INFO[0000] Downloading base image alpine                
2018/12/05 19:12:18 No matching credentials were found, falling back on anonymous
INFO[0001] Unpacking rootfs as cmd RUN test 0 requires it. 
INFO[0001] Taking snapshot of full filesystem...        
INFO[0001] Skipping paths under /dev, as it is a whitelisted directory 
INFO[0001] Skipping paths under /kaniko, as it is a whitelisted directory 
INFO[0001] Skipping paths under /proc, as it is a whitelisted directory 
INFO[0001] Skipping paths under /sys, as it is a whitelisted directory 
INFO[0001] Skipping paths under /var/run, as it is a whitelisted directory 
INFO[0001] Skipping paths under /workspace, as it is a whitelisted directory 
INFO[0002] RUN test 0                                   
INFO[0002] cmd: /bin/sh                                 
INFO[0002] args: [-c test 0]                            
INFO[0002] Taking snapshot of full filesystem...        
INFO[0002] Skipping paths under /dev, as it is a whitelisted directory 
INFO[0002] Skipping paths under /kaniko, as it is a whitelisted directory 
INFO[0002] Skipping paths under /proc, as it is a whitelisted directory 
INFO[0002] Skipping paths under /sys, as it is a whitelisted directory 
INFO[0002] Skipping paths under /var/run, as it is a whitelisted directory 
INFO[0002] Skipping paths under /workspace, as it is a whitelisted directory 
INFO[0002] No files were changed, appending empty layer to config. No layer added to image. 
INFO[0002] USER 1000:1000                               
INFO[0002] cmd: USER                                    
error building image: error building stage: user: unknown userid 1000

[MMeent]

It looks like this is also an issue when using --cache=true: When reaching a USER statement on a cached layer that is not available locally, it fails:

INFO[0000] Downloading base image openjdk:11            
2018/12/13 10:41:49 No matching credentials were found, falling back on anonymous
WARN[0001] Error while retrieving image from cache: getting image from path: open /cache/sha256:c7381bfd53670f1211314885b03b98f5e13fddf6958afeec61092b07c56ddef1: no such file or directory 
2018/12/13 10:41:50 No matching credentials were found, falling back on anonymous
INFO[0002] Executing 0 build triggers                   
INFO[0002] Checking for cached layer docker.company.io/project/cache:3bee254e641f46abcfa0822e3d1600938df2905e0aadba41756a7c6884cbd16d... 
INFO[0002] Using caching version of cmd: RUN groupadd -r $APPLICATION_USER && useradd --no-log-init -r -g $APPLICATION_USER $APPLICATION_USER 
INFO[0002] Checking for cached layer docker.company.io/project/cache:a1875d7dd74f1d18a37dab3e6e75f2f3f08cc58f1ded0d6d92e35647883c0686... 
INFO[0002] Using caching version of cmd: RUN mkdir /app 
INFO[0002] Checking for cached layer docker.company.io/project/cache:b9fafa07e45363ebd9db5304781b7536e15ad7bf028cdd60867fc55151c3cb4f... 
error building image: error building stage: open /etc/passwd: no such file or directory
INFO[0002] Using caching version of cmd: RUN chown -R $APPLICATION_USER /app 
INFO[0002] cmd: USER                                    

[keaaa]

Anyone been able to work around this?

[pbuszka]

I encountered the same problem with a Dockerfile which has relevant lines like:

FROM MyCustomImage
...
FROM node:8.11.4-slim
USER node
...

and it fails with the same problem as above.

error building image: error building stage: open /etc/passwd: no such file or directory

I think it is a regression introduced somewhere between 3 Oct and 15 Nov. I use kaniko invoked from skaffold and skaffold v0.18 works fine (it uses kaniko v0.4 from 3 Oct) but v0.19 fails (it uses kaniko @ commit 0c29413 from Nov 15). skaffold v20 with kaniko v0.7 fails as well with the same problem.

Maybe this might help to pinpoint the problem.

[pmcq]

I think it's just this line

kaniko/pkg/util/command_util.go

Line 309 in b0b36ed

userObj, err = user.LookupId(userStr)

that calls os.user.LookupId which results in trying to read from /etc/passwd and similarly

kaniko/pkg/util/command_util.go

Line 309 in b0b36ed

userObj, err = user.LookupId(userStr)

does this for the group id.

Since all the code does is gets a User object and then returns the UID (which would match the userStr that we looked up in the first place) I feel like the correct behavior would be something like:

1. If userStr is numeric, just use that as a userId (optionally try user.LookupId(userStr) and log a warning or something if the userid isn't found?)
2. If it is not numeric string try looking up userStr with user.Lookup, and if it is not found then return an error
   does that seem like it would give consistent behavior?

edit: playing around with this a little bit one issue seems to be that environment variables like HOME won't be set automatically if you use a numeric userid but that can be worked around if needed.

[everflux]

Maybe unrelated, but using
USER 0
to switch to the root user did not work without any error message. (I would have expected a similar message as when using USER root)

[mamoit]

It happened to me with the cache enabled, as soon as I disabled it showed Unpacking rootfs as cmd USER root requires it and started working.

[matti]

same as @mamoit building first without USER & cache enabled and then using USER will trigger this

[timowuttke]

For me, a USER command in a cached layer will produce the already mentioned error message. Is there some kind of workaround for this?

[aberres]

Happens for us as well and stops us from using Kaniko.