Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

USER command should not require user or group IDs to exist #477

Open
hrobertson opened this issue Dec 4, 2018 · 17 comments

Comments

Projects
None yet
@hrobertson
Copy link

commented Dec 4, 2018

It is common for a Dockerfile to contain a USER directive of the form USER 1000:1000. docker build does not require a user and group with those ids to exist, nor does it create them. However, Kaniko errors.

Interestingly, while creating a minimal dockerfile to demonstrate this I discovered that Kaniko errors in two different ways:

Working example using Docker:

$ cat usertest1
FROM alpine
USER 1000:1000
CMD ["whoami"]
$ docker build --file usertest1 -t usertest:1 .
Sending build context to Docker daemon  23.46MB
Step 1/3 : FROM alpine
 ---> 196d12cf6ab1
Step 2/3 : USER 1000:1000
 ---> Using cache
 ---> 4794c572f720
Step 3/3 : CMD ["whoami"]
 ---> Using cache
 ---> 6ab640ab2641
Successfully built 6ab640ab2641
Successfully tagged usertest:1
$ docker run --rm usertest:1
whoami: unknown uid 1000

Kaniko failure mode 1

$ cat usertest1
FROM alpine
USER 1000:1000
CMD ["whoami"]

$ docker run --rm -it -v $PWD:/workspace gcr.io/kaniko-project/executor:latest --dockerfile=usertest1 --no-push --context=dir:///workspace
INFO[0000] Downloading base image alpine                
2018/12/04 13:25:52 No matching credentials were found, falling back on anonymous
INFO[0001] Taking snapshot of full filesystem...        
INFO[0001] Skipping paths under /dev, as it is a whitelisted directory 
INFO[0001] Skipping paths under /kaniko, as it is a whitelisted directory 
INFO[0001] Skipping paths under /proc, as it is a whitelisted directory 
INFO[0001] Skipping paths under /sys, as it is a whitelisted directory 
INFO[0001] Skipping paths under /workspace, as it is a whitelisted directory 
INFO[0001] USER 1000:1000                               
INFO[0001] cmd: USER                                    
error building image: error building stage: open /etc/passwd: no such file or directory

Kaniko failure mode 2:
Note the addition of the RUN directive and the different error message from Kaniko

$ cat usertest2
FROM alpine
RUN test 0
USER 1000:1000
CMD ["whoami"]

$ docker run --rm -it -v $PWD:/workspace gcr.io/kaniko-project/executor:latest --dockerfile=usertest2 --no-push --context=dir:///workspace
INFO[0000] Downloading base image alpine                
2018/12/04 13:32:00 No matching credentials were found, falling back on anonymous
INFO[0001] Unpacking rootfs as cmd RUN test 0 requires it. 
INFO[0001] Taking snapshot of full filesystem...        
INFO[0001] Skipping paths under /dev, as it is a whitelisted directory 
INFO[0001] Skipping paths under /kaniko, as it is a whitelisted directory 
INFO[0001] Skipping paths under /proc, as it is a whitelisted directory 
INFO[0001] Skipping paths under /sys, as it is a whitelisted directory 
INFO[0001] Skipping paths under /workspace, as it is a whitelisted directory 
INFO[0001] RUN test 0                                   
INFO[0001] cmd: /bin/sh                                 
INFO[0001] args: [-c test 0]                            
INFO[0001] Taking snapshot of full filesystem...        
INFO[0001] Skipping paths under /dev, as it is a whitelisted directory 
INFO[0001] Skipping paths under /kaniko, as it is a whitelisted directory 
INFO[0001] Skipping paths under /proc, as it is a whitelisted directory 
INFO[0001] Skipping paths under /sys, as it is a whitelisted directory 
INFO[0001] Skipping paths under /workspace, as it is a whitelisted directory 
INFO[0001] No files were changed, appending empty layer to config. No layer added to image. 
INFO[0001] USER 1000:1000                               
INFO[0001] cmd: USER                                    
error building image: error building stage: user: unknown userid 1000
@miguelitoq76

This comment has been minimized.

Copy link

commented Dec 4, 2018

@hrobertson could be so kind to test the build with this modification?

add a FROM scratch at the top of the dockerfile

FROM scratch
FROM alpine
USER 1000:1000
CMD ["whoami"]

Please post your result thanks a lot, it would help to find the error!!

@priyawadhwa priyawadhwa added the bug label Dec 4, 2018

@hrobertson

This comment has been minimized.

Copy link
Author

commented Dec 5, 2018

@miguelitoq76 I assume you didn't mean to have two FROMs?

$ cat usertest3
FROM scratch
USER 1000:1000
CMD ["whoami"]
$
$ docker run --rm -it -v $PWD:/workspace gcr.io/kaniko-project/executor:latest --dockerfile=usertest3 --no-push --context=dir:///workspace
INFO[0000] No base image, nothing to extract            
INFO[0000] Taking snapshot of full filesystem...        
INFO[0000] Skipping paths under /dev, as it is a whitelisted directory 
INFO[0000] Skipping paths under /kaniko, as it is a whitelisted directory 
INFO[0000] Skipping paths under /proc, as it is a whitelisted directory 
INFO[0000] Skipping paths under /sys, as it is a whitelisted directory 
INFO[0000] Skipping paths under /workspace, as it is a whitelisted directory 
INFO[0000] USER 1000:1000                               
INFO[0000] cmd: USER                                    
error building image: error building stage: open /etc/passwd: no such file or directory

In case you really did mean to have the two FROMs:

$ cat usertest4
FROM scratch
FROM alpine
USER 1000:1000
CMD ["whoami"]
$
$ docker run --rm -it -v $PWD:/workspace gcr.io/kaniko-project/executor:latest --dockerfile=usertest4 --no-push --context=dir:///workspace
INFO[0000] No base image, nothing to extract            
INFO[0000] Taking snapshot of full filesystem...        
INFO[0000] Skipping paths under /dev, as it is a whitelisted directory 
INFO[0000] Skipping paths under /kaniko, as it is a whitelisted directory 
INFO[0000] Skipping paths under /proc, as it is a whitelisted directory 
INFO[0000] Skipping paths under /sys, as it is a whitelisted directory 
INFO[0000] Skipping paths under /workspace, as it is a whitelisted directory 
INFO[0000] Deleting filesystem...                       
INFO[0000] Downloading base image alpine                
2018/12/05 12:44:45 No matching credentials were found, falling back on anonymous
INFO[0002] Taking snapshot of full filesystem...        
INFO[0002] Skipping paths under /dev, as it is a whitelisted directory 
INFO[0002] Skipping paths under /kaniko, as it is a whitelisted directory 
INFO[0002] Skipping paths under /proc, as it is a whitelisted directory 
INFO[0002] Skipping paths under /sys, as it is a whitelisted directory 
INFO[0002] Skipping paths under /workspace, as it is a whitelisted directory 
INFO[0002] USER 1000:1000                               
INFO[0002] cmd: USER                                    
error building image: error building stage: open /etc/passwd: no such file or directory
@hrobertson

This comment has been minimized.

Copy link
Author

commented Dec 5, 2018

I have done some further investigation and the behaviour is even more erroneous than I first thought!

Here I build three images containing the kaniko executor binary. One pulls the binary into a scratch base, the next into an alpine base, and the next creates a user within the image.

$ cat Dockerfile
FROM gcr.io/kaniko-project/executor:latest as kaniko

FROM scratch as scratch
COPY --from=kaniko /kaniko/executor /kaniko/executor
COPY --from=kaniko /kaniko/ssl/certs/ca-certificates.crt /kaniko/ssl/certs/
ENV SSL_CERT_DIR=/kaniko/ssl/certs
WORKDIR /workspace

ENTRYPOINT ["/kaniko/executor"]

FROM alpine as alpine
COPY --from=kaniko /kaniko/executor /kaniko/executor
COPY --from=kaniko /kaniko/ssl/certs/ca-certificates.crt /kaniko/ssl/certs/
ENV SSL_CERT_DIR=/kaniko/ssl/certs
WORKDIR /workspace

ENTRYPOINT ["/kaniko/executor"]

FROM alpine as alpine-with-user
RUN addgroup -g 1000 test && adduser -D -u 1000 -G test test

I build the three separate images:

$ docker build -t kaniko:scratch --target scratch .
$ docker build -t kaniko:alpine --target alpine .
$ docker build -t kaniko:alpine-with-user --target alpine-with-user .

And here I build the usertest3 image using kaniko inside each of those images.

$ cat usertest3
FROM scratch
USER 1000:1000
CMD ["whoami"]

kaniko binary in scratch

$ docker run --rm -it -v $PWD:/workspace --entrypoint=/kaniko/executor kaniko:scratch --dockerfile=usertest3 --context=dir:///workspace --no-push
INFO[0000] No base image, nothing to extract            
INFO[0000] Taking snapshot of full filesystem...        
INFO[0000] Skipping paths under /dev, as it is a whitelisted directory 
INFO[0000] Skipping paths under /kaniko, as it is a whitelisted directory 
INFO[0000] Skipping paths under /proc, as it is a whitelisted directory 
INFO[0000] Skipping paths under /sys, as it is a whitelisted directory 
INFO[0000] Skipping paths under /workspace, as it is a whitelisted directory 
INFO[0000] USER 1000:1000                               
INFO[0000] cmd: USER                                    
error building image: error building stage: open /etc/passwd: no such file or directory

kaniko binary in alpine

$ docker run --rm -it -v $PWD/test:/workspace --entrypoint=/kaniko/executor kaniko:alpine --dockerfile=usertest3 --context=dir:///workspace --no-push 
INFO[0000] No base image, nothing to extract            
INFO[0000] Taking snapshot of full filesystem...        
INFO[0000] Skipping paths under /dev, as it is a whitelisted directory 
INFO[0000] Skipping paths under /kaniko, as it is a whitelisted directory 
INFO[0000] Skipping paths under /proc, as it is a whitelisted directory 
INFO[0000] Skipping paths under /sys, as it is a whitelisted directory 
INFO[0000] Skipping paths under /var/run, as it is a whitelisted directory 
INFO[0000] Skipping paths under /workspace, as it is a whitelisted directory 
INFO[0000] USER 1000:1000                               
INFO[0000] cmd: USER                                    
error building image: error building stage: user: unknown userid 1000

As you can see, kaniko is erroneously looking for the /etc/passwd file in the environment in which kaniko is running, not the filesystem of the image it's building! Note it shouldn't even look for a passwd file at all when IDs are specified rather than names.

kaniko binary in alpine with an existing user:
Note this user does not exist in the image kaniko is building, but in the environment in which kaniko itself is running.

$ docker run -it --rm -v $PWD:/workspace kaniko:alpine-with-user --dockerfile=usertest3 --context=dir:///workspace --no-push
INFO[0000] Downloading base image alpine                
2018/12/05 13:32:25 Unable to read "/root/.docker/config.json": open /root/.docker/config.json: no such file or directory
2018/12/05 13:32:25 No matching credentials were found, falling back on anonymous
INFO[0001] Taking snapshot of full filesystem...        
INFO[0001] Skipping paths under /dev, as it is a whitelisted directory 
INFO[0001] Skipping paths under /kaniko, as it is a whitelisted directory 
INFO[0001] Skipping paths under /proc, as it is a whitelisted directory 
INFO[0001] Skipping paths under /sys, as it is a whitelisted directory 
INFO[0001] Skipping paths under /var/run, as it is a whitelisted directory 
INFO[0001] Skipping paths under /workspace, as it is a whitelisted directory 
INFO[0001] USER 1000:1000                               
INFO[0001] cmd: USER                                    
INFO[0001] CMD ["whoami"]                               
INFO[0001] Skipping push to container registry due to --no-push flag

Until the first RUN directive causes Unpacking rootfs , kaniko is using the filesystem of the environment that Kaniko is in rather than that of the base image.
As you can see below, using kaniko in the kaniko:alpine-with-user image to build the usertest2 dockerfile (see the initial post), again fails to find the user. Contrast this with the above output building the usertest3 dockerfile.

$ docker run -it --rm -v $PWD:/workspace kaniko:alpine-with-user --dockerfile=usertest2 --context=dir:///workspace --no-push
INFO[0000] Downloading base image alpine                
2018/12/05 19:12:18 No matching credentials were found, falling back on anonymous
INFO[0001] Unpacking rootfs as cmd RUN test 0 requires it. 
INFO[0001] Taking snapshot of full filesystem...        
INFO[0001] Skipping paths under /dev, as it is a whitelisted directory 
INFO[0001] Skipping paths under /kaniko, as it is a whitelisted directory 
INFO[0001] Skipping paths under /proc, as it is a whitelisted directory 
INFO[0001] Skipping paths under /sys, as it is a whitelisted directory 
INFO[0001] Skipping paths under /var/run, as it is a whitelisted directory 
INFO[0001] Skipping paths under /workspace, as it is a whitelisted directory 
INFO[0002] RUN test 0                                   
INFO[0002] cmd: /bin/sh                                 
INFO[0002] args: [-c test 0]                            
INFO[0002] Taking snapshot of full filesystem...        
INFO[0002] Skipping paths under /dev, as it is a whitelisted directory 
INFO[0002] Skipping paths under /kaniko, as it is a whitelisted directory 
INFO[0002] Skipping paths under /proc, as it is a whitelisted directory 
INFO[0002] Skipping paths under /sys, as it is a whitelisted directory 
INFO[0002] Skipping paths under /var/run, as it is a whitelisted directory 
INFO[0002] Skipping paths under /workspace, as it is a whitelisted directory 
INFO[0002] No files were changed, appending empty layer to config. No layer added to image. 
INFO[0002] USER 1000:1000                               
INFO[0002] cmd: USER                                    
error building image: error building stage: user: unknown userid 1000
@MMeent

This comment has been minimized.

Copy link
Contributor

commented Dec 13, 2018

It looks like this is also an issue when using --cache=true: When reaching a USER statement on a cached layer that is not available locally, it fails:

INFO[0000] Downloading base image openjdk:11            
2018/12/13 10:41:49 No matching credentials were found, falling back on anonymous
WARN[0001] Error while retrieving image from cache: getting image from path: open /cache/sha256:c7381bfd53670f1211314885b03b98f5e13fddf6958afeec61092b07c56ddef1: no such file or directory 
2018/12/13 10:41:50 No matching credentials were found, falling back on anonymous
INFO[0002] Executing 0 build triggers                   
INFO[0002] Checking for cached layer docker.company.io/project/cache:3bee254e641f46abcfa0822e3d1600938df2905e0aadba41756a7c6884cbd16d... 
INFO[0002] Using caching version of cmd: RUN groupadd -r $APPLICATION_USER && useradd --no-log-init -r -g $APPLICATION_USER $APPLICATION_USER 
INFO[0002] Checking for cached layer docker.company.io/project/cache:a1875d7dd74f1d18a37dab3e6e75f2f3f08cc58f1ded0d6d92e35647883c0686... 
INFO[0002] Using caching version of cmd: RUN mkdir /app 
INFO[0002] Checking for cached layer docker.company.io/project/cache:b9fafa07e45363ebd9db5304781b7536e15ad7bf028cdd60867fc55151c3cb4f... 
error building image: error building stage: open /etc/passwd: no such file or directory
INFO[0002] Using caching version of cmd: RUN chown -R $APPLICATION_USER /app 
INFO[0002] cmd: USER                                    
@keaaa

This comment has been minimized.

Copy link

commented Jan 3, 2019

Anyone been able to work around this?

@pbuszka

This comment has been minimized.

Copy link

commented Jan 6, 2019

I encountered the same problem with a Dockerfile which has relevant lines like:

FROM MyCustomImage
...
FROM node:8.11.4-slim
USER node
...

and it fails with the same problem as above.

error building image: error building stage: open /etc/passwd: no such file or directory

I think it is a regression introduced somewhere between 3 Oct and 15 Nov. I use kaniko invoked from skaffold and skaffold v0.18 works fine (it uses kaniko v0.4 from 3 Oct) but v0.19 fails (it uses kaniko @ commit 0c29413 from Nov 15). skaffold v20 with kaniko v0.7 fails as well with the same problem.

Maybe this might help to pinpoint the problem.

pmcq referenced this issue in algorithmiaio/kaniko Jan 8, 2019

The rootfs needs to be unpacked in order to properly resolve uids/gid…
…s for chowning purposes, otherwise /etc/passwd isn't found, or wouldn't have the users/groups from the base image
@pmcq

This comment has been minimized.

Copy link

commented Jan 8, 2019

I think it's just this line

userObj, err = user.LookupId(userStr)
that calls os.user.LookupId which results in trying to read from /etc/passwd and similarly
userObj, err = user.LookupId(userStr)
does this for the group id.

Since all the code does is gets a User object and then returns the UID (which would match the userStr that we looked up in the first place) I feel like the correct behavior would be something like:

  1. If userStr is numeric, just use that as a userId (optionally try user.LookupId(userStr) and log a warning or something if the userid isn't found?)
  2. If it is not numeric string try looking up userStr with user.Lookup, and if it is not found then return an error
    does that seem like it would give consistent behavior?

edit: playing around with this a little bit one issue seems to be that environment variables like HOME won't be set automatically if you use a numeric userid but that can be worked around if needed.

@everflux

This comment has been minimized.

Copy link

commented Feb 2, 2019

Maybe unrelated, but using
USER 0
to switch to the root user did not work without any error message. (I would have expected a similar message as when using USER root)

@mamoit

This comment has been minimized.

Copy link

commented Feb 6, 2019

It happened to me with the cache enabled, as soon as I disabled it showed Unpacking rootfs as cmd USER root requires it and started working.

@matti

This comment has been minimized.

Copy link

commented Feb 18, 2019

same as @mamoit building first without USER & cache enabled and then using USER will trigger this

@twuttk

This comment has been minimized.

Copy link

commented Feb 19, 2019

For me, a USER command in a cached layer will produce the already mentioned error message. Is there some kind of workaround for this?

@aberres

This comment has been minimized.

Copy link

commented Feb 21, 2019

Happens for us as well and stops us from using Kaniko.

@marcomancuso

This comment has been minimized.

Copy link

commented Apr 3, 2019

Any update on this?

@geekofalltrades

This comment has been minimized.

Copy link

commented Apr 5, 2019

I think I was able to work around this just now by adding a minimal /etc/passwd to my container before the USER instruction.

Dockerfile:

ARG UPSTREAM_IMAGE=prom/alertmanager
ARG UPSTREAM_TAG=v0.15.3

FROM $UPSTREAM_IMAGE:$UPSTREAM_TAG as upstream

FROM scratch

LABEL project=<my-project>

COPY --from=upstream /bin/alertmanager /bin/alertmanager
COPY --from=upstream /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
COPY --from=upstream /etc/nsswitch.conf /etc/nsswitch.conf

# Work around Kaniko bug.
COPY --chown=0:0 passwd /etc/passwd

EXPOSE 9093

USER 4191718

ENTRYPOINT ["/bin/alertmanager"]
CMD        [ "--config.file=/etc/alertmanager/config.yml", \
             "--storage.path=/alertmanager" ]

passwd:

alertmanager:x:4191718:4191718:Alertmanager Service User::/bin/false

Kaniko build log:

INFO[0000] Downloading base image prom/alertmanager:v0.15.3 
2019/04/05 20:05:50 No matching credentials were found, falling back on anonymous
INFO[0001] Error while retrieving image from cache: getting file info: stat /cache/sha256:196af0317d3449c1300aa26ff0366f68c67d04581d2c9f8609cbb227424e226c: no such file or directory 
INFO[0001] Downloading base image prom/alertmanager:v0.15.3 
2019/04/05 20:05:51 No matching credentials were found, falling back on anonymous
INFO[0001] Executing 0 build triggers                   
INFO[0001] Skipping unpacking as no commands require it. 
INFO[0001] Taking snapshot of full filesystem...        
INFO[0001] Storing source image from stage 0 at path /kaniko/stages/0 
INFO[0003] Deleting filesystem...                       
INFO[0003] No base image, nothing to extract            
INFO[0003] Unpacking rootfs as cmd USER 4191718 requires it. 
INFO[0003] Taking snapshot of full filesystem...        
INFO[0004] LABEL project=<my-project>                       
INFO[0004] Applying label project=<my-project>              
INFO[0004] COPY --from=upstream /bin/alertmanager /bin/alertmanager 
INFO[0004] Taking snapshot of files...                  
INFO[0004] COPY --from=upstream /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt 
INFO[0004] Taking snapshot of files...                  
INFO[0004] COPY --from=upstream /etc/nsswitch.conf /etc/nsswitch.conf 
INFO[0004] Taking snapshot of files...                  
INFO[0004] Using files from context: [/builds/wmd/hopcloud-thirdparty/hopcloud-alertmanager/passwd] 
INFO[0004] COPY --chown=0:0 passwd /etc/passwd          
INFO[0004] Taking snapshot of files...                  
INFO[0004] EXPOSE 9093                                  
INFO[0004] cmd: EXPOSE                                  
INFO[0004] Adding exposed port: 9093/tcp                
INFO[0004] USER 4191718                                 
INFO[0004] cmd: USER                                    
INFO[0004] ENTRYPOINT ["/bin/alertmanager"]             
INFO[0004] CMD        [ "--config.file=/etc/alertmanager/config.yml",              "--storage.path=/alertmanager" ] 
2019/04/05 20:05:55 pushed blob sha256:755f71954f5678ea5d7abaa9324959e536067a51332c7fcb9e1832d47b3d0fec
2019/04/05 20:05:55 pushed blob sha256:f74df0e0e91fe2c111555688756efeb59cfb29d4d75ab90b7fbcf6bed6bbdae6
2019/04/05 20:05:55 pushed blob sha256:3da111e28cadb8bb46f10bed9de53f0de7cbb74dc4fd22db1ded8ab81f52c206
2019/04/05 20:05:55 pushed blob sha256:9e57d8d1fbc22639da8d7019ea323f99fd639a1520312fc65f065f524959375f
2019/04/05 20:05:57 pushed blob sha256:83ebe99b26904b90173246b81fa7a9ea6f65f021f6350d2eb5303dc75e553508
2019/04/05 20:05:58 <my-registry>/<my-repo>:<my-tag>: digest: sha256:5c1b3eab5f386f9722c50d7b795dc89220184a35751861ad6e73aa148699296c size: 912

(This Dockerfile is taking Prometheus' alertmanager, copying the binary and some other important stuff into a scratch container, and forcing it to run as a non-root user. Stripping down third-party images like this is part of the security process where I work.)

This is a clunky workaround and I'd still appreciate this bug being fixed.

@rnsv

This comment has been minimized.

Copy link

commented Apr 28, 2019

Using gcr.io/kaniko-project/executor:debug allowed me to workaround this issue

When using executor:latest

INFO[0000] USER 10400:10400
INFO[0000] cmd: USER
error building image: error building stage: open /etc/passwd: no such file or directory

When using executor:debug

INFO[0002] Taking snapshot of full filesystem...
INFO[0002] USER 10400:10400
INFO[0002] cmd: USER
2019/04/28 07:34:31 existing blob: sha256:67ddbfb20a22d7c0ea0df56
@djuarezg

This comment has been minimized.

Copy link

commented Apr 29, 2019

Suffering from this as well on gcr.io/kaniko-project/executor:debug-v0.9.0 :

INFO[0000] cmd: USER                                    
error building image: error building stage: open /etc/passwd: no such file or directory

With the latest version:

error building image: error building stage: user: invalid userid <username>
@gugahoi

This comment has been minimized.

Copy link

commented May 11, 2019

Using gcr.io/kaniko-project/executor:debug allowed me to get past this issue when running both locally and on Cloud Build. Shasum: 5a6b3193506212ea7703214bf3b21cd898e5e2ec14d6b988cc9d11a9b34f32d4

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.