Multistage build directory copy fails #1110

tarvitz · 2020-03-06T13:47:08Z

Actual behavior
Multistage build fails.

#: Dockerfile
FROM python:3.6-alpine as part-one
RUN set -x \
    && python -m venv /venv

FROM python:3.6-alpine
COPY --from=part-one /venv /venv

The latest kaniko result build and output:

docker run --rm -w /build-app -v `pwd`:/build-app -v ${HOME}/.docker:/kaniko/.docker \
    --entrypoint "" gcr.io/kaniko-project/executor:latest /kaniko/executor -c . -f Dockerfile \
    --destination "test" \
    --no-push

Output:

# previous stages are cut
INFO[0011] args: [-c set -x     && python -m venv /venv] 
+ python -m venv /venv
INFO[0013] Taking snapshot of full filesystem...        
INFO[0013] Resolving paths                              
INFO[0014] Saving file /venv for later use              
INFO[0014] Deleting filesystem...                       
INFO[0014] Retrieving image manifest python:3.6-alpine  
INFO[0015] Retrieving image manifest python:3.6-alpine  
INFO[0016] Unpacking rootfs as cmd COPY --from=part-one /venv /venv requires it. 
INFO[0019] Taking snapshot of full filesystem...        
INFO[0019] Resolving paths                              
INFO[0020] COPY --from=part-one /venv /venv             
error building image: error building stage: failed to execute command: read /kaniko/0/venv/lib: is a directory

Expected behavior
Multistage build runs well.

Output with kaniko: gcr.io/kaniko-project/executor:v0.17.1

INFO[0012] args: [-c set -x     && python -m venv /venv] 
+ python -m venv /venv
INFO[0014] Taking snapshot of full filesystem...        
INFO[0015] Saving file /venv for later use              
INFO[0015] Deleting filesystem...                       
INFO[0015] Retrieving image manifest python:3.6-alpine  
INFO[0016] Retrieving image manifest python:3.6-alpine  
INFO[0018] Unpacking rootfs as cmd COPY --from=part-one /venv /venv requires it. 
INFO[0021] Taking snapshot of full filesystem...        
INFO[0022] COPY --from=part-one /venv /venv             
INFO[0022] Taking snapshot of files...                  
INFO[0022] Skipping push to container registry due to --no-push flag

To Reproduce
Steps to reproduce the behavior:

Use provided Dockerfile
Try to build it by the latest kaniko

Additional Information
Dockerfile:

FROM python:3.6-alpine as part-one
RUN set -x \
    && python -m venv /venv

FROM python:3.6-alpine
COPY --from=part-one /venv /venv

Build context not required
Kaniko images:

ok: sha256:970d32fa1eb2827a1f216d0b1090c8a575f0a29eb0a786ebd6fa544a56946867 (gcr.io/kaniko-project/executor:v0.17.1)
failure: sha256:876c47402df7e62af4e12b907a872bd962d5d8d4c39372ace61e9931bd734af9 (gcr.io/kaniko-project/executor:latest)

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing	- [ no ]
Please check if the build works in docker but not in kaniko	- [ yes ]
Please check if this error is seen when you use `--cache` flag	- [ n/a ]
Please check if your dockerfile is a multistage dockerfile	- [ yes ]

The text was updated successfully, but these errors were encountered:

tejal29 · 2020-03-07T04:57:48Z

Thanks @tarvitz .

I have this PR #1114 and #1113 which ended up fixing this

FROM python:3.6-alpine as part-one
RUN set -x \
    && python -m venv /venv

FROM python:3.6-alpine

Here i used the gcr.io/kaniko-project/executor:edge from latest master.

docker run  -v /usr/local/google/home/tejaldesai/.config/gcloud:/root/.config/gcloud -v /usr/local/google/home/tejaldesai/workspace/kaniko/integration:/workspace gcr.io/kaniko-project/executor:edge -f dockerfiles/Dockerfile1 --context=dir://workspace --destination=gcr.io/tejal-test/test_1110 --no-push --tarPath image.tar
INFO[0000] Resolved base name python:3.6-alpine to python:3.6-alpine 
INFO[0000] Resolved base name python:3.6-alpine to python:3.6-alpine 
INFO[0000] Resolved base name python:3.6-alpine to python:3.6-alpine 
INFO[0000] Resolved base name python:3.6-alpine to python:3.6-alpine 
INFO[0000] Retrieving image manifest python:3.6-alpine  
INFO[0001] Retrieving image manifest python:3.6-alpine  
INFO[0001] Retrieving image manifest python:3.6-alpine  
INFO[0002] Retrieving image manifest python:3.6-alpine  
INFO[0003] Built cross stage deps: map[0:[/venv]]       
INFO[0003] Retrieving image manifest python:3.6-alpine  
INFO[0003] Retrieving image manifest python:3.6-alpine  
INFO[0004] Unpacking rootfs as cmd RUN set -x     && python -m venv /venv requires it. 
INFO[0006] Taking snapshot of full filesystem...        
INFO[0006] Resolving paths                              
INFO[0007] RUN set -x     && python -m venv /venv       
INFO[0007] cmd: /bin/sh                                 
INFO[0007] args: [-c set -x     && python -m venv /venv] 
+ python -m venv /venv
INFO[0009] Taking snapshot of full filesystem...        
INFO[0009] Resolving paths                              
INFO[0010] Saving file /venv for later use              
INFO[0010] Deleting filesystem...                       
INFO[0011] Retrieving image manifest python:3.6-alpine  
INFO[0011] Retrieving image manifest python:3.6-alpine  
INFO[0012] Unpacking rootfs as cmd COPY --from=part-one /venv /venv requires it. 
INFO[0014] Taking snapshot of full filesystem...        
INFO[0014] Resolving paths                              
INFO[0015] COPY --from=part-one /venv /venv             
INFO[0015] Resolving paths                              
INFO[0015] Taking snapshot of files...

tejal29 · 2020-03-07T05:10:09Z

Can you use the latest image?

gcr.io/kaniko-project/executor:edge
gcr.io/kaniko-project/executor:edge-debug
gcr.io/kaniko-project/warmer:edge

tarvitz · 2020-03-07T09:12:10Z

Checked with executor:edge, here's result:

Unable to find image 'gcr.io/kaniko-project/executor:edge' locally
edge: Pulling from kaniko-project/executor
0a808f4ece2c: Pull complete
2281caba317a: Pull complete
a9a25b68dfd8: Pull complete
83da15a9ffd7: Pull complete
b1d29b7bd0d6: Pull complete
1d33a19aafcc: Pull complete
115a4b33d86a: Pull complete
90d698384a66: Pull complete
Digest: sha256:df865f7143e8035da29687a53c16e5f4c2d1187dff688d7c9b57f4d4a389b64f
Status: Downloaded newer image for gcr.io/kaniko-project/executor:edge
INFO[0000] Resolved base name python:3.6-alpine to python:3.6-alpine
INFO[0000] Resolved base name python:3.6-alpine to python:3.6-alpine
INFO[0000] Resolved base name python:3.6-alpine to python:3.6-alpine
INFO[0000] Resolved base name python:3.6-alpine to python:3.6-alpine
INFO[0000] Retrieving image manifest python:3.6-alpine
INFO[0001] Retrieving image manifest python:3.6-alpine
INFO[0003] Retrieving image manifest python:3.6-alpine
INFO[0004] Retrieving image manifest python:3.6-alpine
INFO[0005] Built cross stage deps: map[0:[/venv]]
INFO[0005] Retrieving image manifest python:3.6-alpine
INFO[0006] Retrieving image manifest python:3.6-alpine
INFO[0007] Unpacking rootfs as cmd RUN set -x     && python -m venv /venv requires it.
INFO[0011] Taking snapshot of full filesystem...
INFO[0011] Resolving paths
INFO[0012] RUN set -x     && python -m venv /venv
INFO[0012] cmd: /bin/sh
INFO[0012] args: [-c set -x     && python -m venv /venv]
+ python -m venv /venv
INFO[0014] Taking snapshot of full filesystem...
INFO[0014] Resolving paths
INFO[0015] Saving file /venv for later use
INFO[0015] Deleting filesystem...
INFO[0015] Retrieving image manifest python:3.6-alpine
INFO[0016] Retrieving image manifest python:3.6-alpine
INFO[0017] Unpacking rootfs as cmd COPY --from=part-one /venv /venv requires it.
INFO[0021] Taking snapshot of full filesystem...
INFO[0021] Resolving paths
INFO[0022] COPY --from=part-one /venv /venv
INFO[0022] Resolving paths
INFO[0022] Taking snapshot of files...
INFO[0022] Skipping push to container registry due to --no-push flag

Dockerfile is the same.
Thanks @tejal29. It seems everything is fine now.

joscarras · 2020-03-10T13:26:48Z

@tejal29 I just tried the latest version you posted above (not including the warmer) and I am still getting a similar error

error building image: error building stage: failed to execute command: copying dir: mkdir /usr/local/zeek-3.1.0/lib/zeekctl: not a directory

I was able to resolve the issue by using gcr.io/kaniko-project/executor:debug-v0.17.1

Nainterceptor · 2020-03-11T20:08:01Z

Hello,

I'm adding some noise to this thread to find this issue via Google if somebody else have an issue in their CI/CD. So, sorry if my comment is not relevant for the issue resolution.

This issue cause a failure in many of our gitlab CD, we've switched to debug-v0.17.1 too, solving the issue. I guess that gitlab documentation should not recommand to configure on :debug tag :/

As a reference, our DockerFile, and our .gitlab-ci.yml in version 1.4.5 are affected by this issue, and this project is opensource.

Here, we are doing an npm install (yes, nodeJS project here, sorry), then we copy node_modules from the first stage to the second.
Copy is failing, but the CI step pass.
So, our issue is visible on runtime; while some files are not well copied, and some imports are failing. I guess that the copy partially works.

Best regards,
Gaël

sereinity · 2020-03-19T09:20:48Z

For me v0.19.0 has fixed the issue.

tarvitz · 2020-03-19T14:17:32Z

I've updated back to gcr.io/kaniko-project/executor:debug works fine with copying virtual environment directory of python (like in the example above).

dmilan77 · 2020-05-22T23:52:47Z

I just tested with gcr.io/kaniko-project/executor:debug

#: Dockerfile
FROM python:3.7.7-slim-buster as part-one
RUN set -x \
    && python -m venv /venv
RUN ls -l /venv

FROM python:3.7.7-slim-buster
COPY --from=part-one /venv /venv
RUN ls -l /venv

It's failing.

INFO[0016] args: [-c ls -l /venv]
ls: cannot access '/venv': No such file or directory
error building image: error building stage: failed to execute command: waiting for process to exit: exit status 2

mepunit · 2022-02-24T09:30:55Z

Is it resolved?

I am still facing this issue.

tarvitz · 2022-03-02T18:56:26Z

Yes, the latest version of Kaniko works fine against the problem I've reported (I've just taken a look if builds are being built and they're fine).

tejal29 added the regression label Mar 6, 2020

tejal29 mentioned this issue Mar 7, 2020

Add more tests for Copy and some fixes. #1114

Merged

tejal29 added the in progress label Mar 7, 2020

MatteoJoliveau mentioned this issue Mar 10, 2020

Possible bug in upstream Kaniko affecting latest version banzaicloud/drone-kaniko#33

Closed

tejal29 closed this as completed Mar 19, 2020

dmilan77 mentioned this issue May 23, 2020

Multistage COPY fails gcr.io/kaniko-project/executor:debug #1280

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Multistage build directory copy fails #1110

Multistage build directory copy fails #1110

tarvitz commented Mar 6, 2020

tejal29 commented Mar 7, 2020

tejal29 commented Mar 7, 2020

tarvitz commented Mar 7, 2020 •

edited

Loading

joscarras commented Mar 10, 2020

Nainterceptor commented Mar 11, 2020

sereinity commented Mar 19, 2020

tarvitz commented Mar 19, 2020

dmilan77 commented May 22, 2020 •

edited

Loading

mepunit commented Feb 24, 2022 •

edited

Loading

tarvitz commented Mar 2, 2022

Multistage build directory copy fails #1110

Multistage build directory copy fails #1110

Comments

tarvitz commented Mar 6, 2020

tejal29 commented Mar 7, 2020

tejal29 commented Mar 7, 2020

tarvitz commented Mar 7, 2020 • edited Loading

joscarras commented Mar 10, 2020

Nainterceptor commented Mar 11, 2020

sereinity commented Mar 19, 2020

tarvitz commented Mar 19, 2020

dmilan77 commented May 22, 2020 • edited Loading

mepunit commented Feb 24, 2022 • edited Loading

tarvitz commented Mar 2, 2022

tarvitz commented Mar 7, 2020 •

edited

Loading

dmilan77 commented May 22, 2020 •

edited

Loading

mepunit commented Feb 24, 2022 •

edited

Loading