Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Second multistage build in same container fails because cross stage deps are not cleaned up #1217

Closed
AndreKR opened this issue May 3, 2020 · 8 comments
Closed

Second multistage build in same container fails because cross stage deps are not cleaned up #1217

AndreKR opened this issue May 3, 2020 · 8 comments
Labels
needs-reproduction

Comments

@AndreKR
Copy link

AndreKR commented May 3, 2020

I'm using kaniko with GitLab CI to build several images in one go. While this might not be the recommended way to run kaniko, it is something that many people do. When using GitLab CI we use the kaniko debug image and use the builtin shell to run a script. The script contains two (in this example identical) executor runs:

/kaniko/executor --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/Dockerfile --destination $CI_REGISTRY_IMAGE:test --cleanup
/kaniko/executor --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/Dockerfile --destination $CI_REGISTRY_IMAGE:test --cleanup

The problem is that the second build fails because there are still files in /kaniko/0 from the first build.

Here is a Dockerfile that reproduces the issue. It's slightly less simple than I'd hoped for because the issue only surfaces when there are certain symlinks among the cross-stage dependencies.

FROM alpine:3.9 AS mozjpeg-builder
RUN apk --no-cache add git bash autoconf automake build-base libtool nasm pkgconfig
RUN rm /bin/sh && ln -s /bin/bash /bin/sh
RUN git clone git://github.com/mozilla/mozjpeg.git
WORKDIR mozjpeg
RUN git checkout v3.3.1
RUN autoreconf -fiv && ./configure --prefix=/opt/mozjpeg && make install

FROM alpine:3.9
COPY --from=mozjpeg-builder /opt/mozjpeg/bin/jpegtran /opt/mozjpeg/bin/cjpeg /usr/bin/
COPY --from=mozjpeg-builder /opt/mozjpeg/lib64 /usr/lib/

Here is the full log output of both builds as well as directory listings of /kaniko. I believe the relevant parts are:

The end of the first (working) build:

 INFO[0134] Taking snapshot of full filesystem...        
 INFO[0135] Resolving paths                              
 INFO[0136] Saving file /opt/mozjpeg/bin/jpegtran for later use 
 INFO[0136] Saving file /opt/mozjpeg/bin/cjpeg for later use 
 INFO[0136] Saving file /opt/mozjpeg/lib64 for later use 
 INFO[0136] Deleting filesystem...                       
 INFO[0136] Retrieving image manifest alpine:3.9         
 INFO[0136] Retrieving image manifest alpine:3.9         
 INFO[0137] Unpacking rootfs as cmd COPY --from=mozjpeg-builder /opt/mozjpeg/bin/jpegtran /opt/mozjpeg/bin/cjpeg /usr/bin/ requires it. 
 INFO[0137] Taking snapshot of full filesystem...        
 INFO[0137] Resolving paths                              
 INFO[0137] COPY --from=mozjpeg-builder /opt/mozjpeg/bin/jpegtran /opt/mozjpeg/bin/cjpeg /usr/bin/ 
 INFO[0137] Resolving paths                              
 INFO[0137] Taking snapshot of files...                  
 INFO[0137] COPY --from=mozjpeg-builder /opt/mozjpeg/lib64 /usr/lib/ 
 INFO[0137] Resolving paths                              
 INFO[0137] Taking snapshot of files...                  
 INFO[0137] Deleting filesystem...

The directory listing after this build:

 /kaniko
 /kaniko/executor
 /kaniko/warmer
 /kaniko/224861295
 /kaniko/.docker
 /kaniko/.docker/config.json
 /kaniko/706913616
 /kaniko/423980661
 /kaniko/117792945
 /kaniko/270267403
 /kaniko/Dockerfile
 /kaniko/598151260
 /kaniko/103397870
 /kaniko/0
 /kaniko/0/opt
 /kaniko/0/opt/mozjpeg
 /kaniko/0/opt/mozjpeg/bin
 /kaniko/0/opt/mozjpeg/bin/cjpeg
 /kaniko/0/opt/mozjpeg/bin/jpegtran
 /kaniko/0/opt/mozjpeg/lib64
 /kaniko/0/opt/mozjpeg/lib64/libturbojpeg.so
 /kaniko/0/opt/mozjpeg/lib64/libturbojpeg.so.0
 /kaniko/0/opt/mozjpeg/lib64/libjpeg.so
 /kaniko/0/opt/mozjpeg/lib64/pkgconfig
 /kaniko/0/opt/mozjpeg/lib64/pkgconfig/libjpeg.pc
 /kaniko/0/opt/mozjpeg/lib64/pkgconfig/libturbojpeg.pc
 /kaniko/0/opt/mozjpeg/lib64/libturbojpeg.so.0.1.0
 /kaniko/0/opt/mozjpeg/lib64/libturbojpeg.la
 /kaniko/0/opt/mozjpeg/lib64/libjpeg.so.62
 /kaniko/0/opt/mozjpeg/lib64/libturbojpeg.a
 /kaniko/0/opt/mozjpeg/lib64/libjpeg.a
 /kaniko/0/opt/mozjpeg/lib64/libjpeg.so.62.2.0
 /kaniko/0/opt/mozjpeg/lib64/libjpeg.la
 /kaniko/.config
 /kaniko/.config/gcloud
 /kaniko/.config/gcloud/docker_credential_gcr_config.json
 /kaniko/ssl
 /kaniko/ssl/certs
 /kaniko/ssl/certs/ca-certificates.crt
 /kaniko/docker-credential-acr
 /kaniko/docker-credential-ecr-login
 /kaniko/docker-credential-gcr

A closer look at the contents of /kaniko/0/opt/mozjpeg/lib64:

 drwxr-xr-x    3 0        0             4096 May  3 20:07 .
 drwxr-xr-x    4 0        0             4096 May  3 20:07 ..
 -rw-r--r--    1 0        0           814116 May  3 20:07 libjpeg.a
 -rwxr-xr-x    1 0        0              928 May  3 20:07 libjpeg.la
 lrwxrwxrwx    1 0        0               17 May  3 20:07 libjpeg.so -> libjpeg.so.62.2.0
 lrwxrwxrwx    1 0        0               17 May  3 20:07 libjpeg.so.62 -> libjpeg.so.62.2.0
 -rwxr-xr-x    1 0        0           665216 May  3 20:07 libjpeg.so.62.2.0
 -rw-r--r--    1 0        0           931278 May  3 20:07 libturbojpeg.a
 -rwxr-xr-x    1 0        0              959 May  3 20:07 libturbojpeg.la
 lrwxrwxrwx    1 0        0               21 May  3 20:07 libturbojpeg.so -> libturbojpeg.so.0.1.0
 lrwxrwxrwx    1 0        0               21 May  3 20:07 libturbojpeg.so.0 -> libturbojpeg.so.0.1.0
 -rwxr-xr-x    1 0        0           746416 May  3 20:07 libturbojpeg.so.0.1.0
 drwxr-xr-x    2 0        0             4096 May  3 20:07 pkgconfig

And then the end of the second (identical, but failing) build:

 INFO[0135] Taking snapshot of full filesystem...        
 INFO[0135] Resolving paths                              
 INFO[0136] Saving file /opt/mozjpeg/bin/jpegtran for later use 
 INFO[0136] Saving file /opt/mozjpeg/bin/cjpeg for later use 
 INFO[0136] Saving file /opt/mozjpeg/lib64 for later use 
 error building image: symlink libjpeg.so.62.2.0 /kaniko/0/opt/mozjpeg/lib64/libjpeg.so: file exists

Obviously the expected result would be that the second build finishes without errors. I would expect either --cleanup to clean up those saved dependencies or I would expect the "Saving file ... for later use" step to overwrite the files from the previous build.

Triage Notes for the Maintainers

Description Yes/No
Please check if this a new feature you are proposing
Please check if the build works in docker but not in kaniko
Please check if this error is seen when you use --cache flag
Please check if your dockerfile is a multistage dockerfile
@AndreKR
Copy link
Author

AndreKR commented May 3, 2020

With v0.17.1 it works. With v0.19.0 it's broken.

@tejal29
Copy link
Member

tejal29 commented May 4, 2020

I verified this on latest master and this is no longer an issue. Looks like #1214 fixed it.

gcr.io/kaniko-project/executor:debug-edge

Here is the full log

tejaldesai@@~$  docker run -it --entrypoint /busybox/sh -v /usr/local/google/home/tejaldesai/.config/gcloud:/root/.config/gcloud -v /usr/local/google/home/tejaldesai/workspace/example:/workspace gcr.io/kaniko-project/executor:debug-edge
/ # /kaniko/executor -f Dockerfile --context=dir://workspace --destination=gcr.io/tejal-test/test1
INFO[0000] Resolved base name alpine:3.9 to mozjpeg-builder 
INFO[0000] Retrieving image manifest alpine:3.9         
INFO[0001] Retrieving image manifest alpine:3.9         
INFO[0002] Retrieving image manifest alpine:3.9         
INFO[0002] Retrieving image manifest alpine:3.9         
INFO[0003] Built cross stage deps: map[0:[/opt/mozjpeg/bin/jpegtran /opt/mozjpeg/bin/cjpeg /opt/mozjpeg/lib64]] 
INFO[0003] Retrieving image manifest alpine:3.9         
INFO[0004] Retrieving image manifest alpine:3.9         
INFO[0004] Executing 0 build triggers                   
INFO[0004] Unpacking rootfs as cmd RUN apk --no-cache add git bash autoconf automake build-base libtool nasm pkgconfig requires it. 
INFO[0005] RUN apk --no-cache add git bash autoconf automake build-base libtool nasm pkgconfig 
INFO[0005] Taking initial snapshot 1                    
INFO[0005] Taking snapshot of full filesystem...        
INFO[0005] Resolving paths                              
INFO[0005] cmd: /bin/sh                                 
INFO[0005] args: [-c apk --no-cache add git bash autoconf automake build-base libtool nasm pkgconfig] 
fetch http://dl-cdn.alpinelinux.org/alpine/v3.9/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.9/community/x86_64/APKINDEX.tar.gz
(1/39) Installing m4 (1.4.18-r1)
(2/39) Installing libbz2 (1.0.6-r7)
(3/39) Installing perl (5.26.3-r0)
...
....
(38/39) Installing nasm (2.13.03-r0)
(39/39) Installing pkgconf (1.6.0-r0)
Executing busybox-1.29.3-r10.trigger
Executing ca-certificates-20191127-r0.trigger
OK: 233 MiB in 53 packages
INFO[0007] Taking snapshot of full filesystem...        
INFO[0007] Resolving paths                              
INFO[0014] RUN rm /bin/sh && ln -s /bin/bash /bin/sh    
INFO[0014] cmd: /bin/sh                                 
INFO[0014] args: [-c rm /bin/sh && ln -s /bin/bash /bin/sh] 
INFO[0014] Taking snapshot of full filesystem...        
INFO[0014] Resolving paths                              
INFO[0014] RUN git clone git://github.com/mozilla/mozjpeg.git 
INFO[0014] cmd: /bin/sh                                 
INFO[0014] args: [-c git clone git://github.com/mozilla/mozjpeg.git] 
Cloning into 'mozjpeg'...
remote: Enumerating objects: 289, done.        
remote: Counting objects: 100% (289/289), done.        
remote: Compressing objects: 100% (185/185), done.        
remote: Total 17062 (delta 185), reused 148 (delta 103), pack-reused 16773        
Receiving objects: 100% (17062/17062), 12.16 MiB | 29.66 MiB/s, done.
Resolving deltas: 100% (12108/12108), done.
INFO[0016] Taking snapshot of full filesystem...        
INFO[0016] Resolving paths                              
INFO[0017] WORKDIR mozjpeg                              
INFO[0017] cmd: workdir                                 
INFO[0017] Changed working directory to mozjpeg         
INFO[0017] RUN git checkout v3.3.1                      
INFO[0017] cmd: /bin/sh                                 
INFO[0017] args: [-c git checkout v3.3.1]               
Note: checking out 'v3.3.1'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at f154ccc0 Version bump
INFO[0017] Taking snapshot of full filesystem...        
INFO[0017] Resolving paths                              
INFO[0018] RUN autoreconf -fiv && ./configure --prefix=/opt/mozjpeg && make install 
INFO[0018] cmd: /bin/sh                                 
INFO[0018] args: [-c autoreconf -fiv && ./configure --prefix=/opt/mozjpeg && make install] 
autoreconf: Entering directory `.'
autoreconf: configure.ac: not using Gettext
....
  CCLD     libsimd.la
ar: `u' modifier ignored since `D' is the default (see `U')
make[3]: Entering directory '/mozjpeg/simd'
make[3]: Nothing to be done for 'install-exec-am'.
make[3]: Nothing to be done for 'install-data-am'.
make[3]: Leaving directory '/mozjpeg/simd'
make[2]: Leaving directory '/mozjpeg/simd'
make[1]: Leaving directory '/mozjpeg/simd'
Making install in md5
make[1]: Entering directory '/mozjpeg/md5'
  CC       md5cmp-md5cmp.o
  CC       md5cmp-md5.o
  CC       md5cmp-md5hl.o
  CCLD     md5cmp
make[2]: Entering directory '/mozjpeg/md5'
make[2]: Nothing to be done for 'install-exec-am'.
make[2]: Nothing to be done for 'install-data-am'.
make[2]: Leaving directory '/mozjpeg/md5'
make[1]: Leaving directory '/mozjpeg/md5'
make[1]: Entering directory '/mozjpeg'
  CC       cjpeg-cdjpeg.o
  CC       cjpeg-cjpeg.o
  CC       cjpeg-rdgif.o
  ...
  CCLD     libjpeg.la
ar: `u' modifier ignored since `D' is the default (see `U')
  CCLD     cjpeg
 ...
  CC       tjutil.o
  CCLD     tjunittest
make[2]: Entering directory '/mozjpeg'
 ./install-sh -c -d '/opt/mozjpeg/lib64'
 /bin/sh ./libtool   --mode=install /usr/bin/install -c   libjpeg.la libturbojpeg.la '/opt/mozjpeg/lib64'
libtool: install: /usr/bin/install -c .libs/libjpeg.so.62.2.0 /opt/mozjpeg/lib64/libjpeg.so.62.2.0
libtool: install: (cd /opt/mozjpeg/lib64 && { ln -s -f libjpeg.so.62.2.0 libjpeg.so.62 || { rm -f libjpeg.so.62 && ln -s libjpeg.so.62.2.0 libjpeg.so.62; }; })
libtool: install: (cd /opt/mozjpeg/lib64 && { ln -s -f libjpeg.so.62.2.0 libjpeg.so || { rm -f libjpeg.so && ln -s libjpeg.so.62.2.0 libjpeg.so; }; })
libtool: install: /usr/bin/install -c .libs/libjpeg.lai /opt/mozjpeg/lib64/libjpeg.la
libtool: install: /usr/bin/install -c .libs/libturbojpeg.so.0.1.0 /opt/mozjpeg/lib64/libturbojpeg.so.0.1.0
libtool: install: (cd /opt/mozjpeg/lib64 && { ln -s -f libturbojpeg.so.0.1.0 libturbojpeg.so.0 || { rm -f libturbojpeg.so.0 && ln -s libturbojpeg.so.0.1.0 libturbojpeg.so.0; }; })
libtool: install: (cd /opt/mozjpeg/lib64 && { ln -s -f libturbojpeg.so.0.1.0 libturbojpeg.so || { rm -f libturbojpeg.so && ln -s libturbojpeg.so.0.1.0 libturbojpeg.so; }; })
libtool: install: /usr/bin/install -c .libs/libturbojpeg.lai /opt/mozjpeg/lib64/libturbojpeg.la
libtool: install: /usr/bin/install -c .libs/libjpeg.a /opt/mozjpeg/lib64/libjpeg.a
libtool: install: chmod 644 /opt/mozjpeg/lib64/libjpeg.a
libtool: install: ranlib /opt/mozjpeg/lib64/libjpeg.a
libtool: install: /usr/bin/install -c .libs/libturbojpeg.a /opt/mozjpeg/lib64/libturbojpeg.a
libtool: install: chmod 644 /opt/mozjpeg/lib64/libturbojpeg.a
libtool: install: ranlib /opt/mozjpeg/lib64/libturbojpeg.a
libtool: finish: PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/sbin" ldconfig -n /opt/mozjpeg/lib64
----------------------------------------------------------------------
Libraries have been installed in:
   /opt/mozjpeg/lib64

...
See any operating system documentation about shared libraries for
more information, such as the ld(1) and ld.so(8) manual pages.
----------------------------------------------------------------------
 ./install-sh -c -d '/opt/mozjpeg/bin'
  /bin/sh ./libtool   --mode=install /usr/bin/install -c cjpeg djpeg jpegtran rdjpgcom wrjpgcom tjbench '/opt/mozjpeg/bin'
libtool: install: /usr/bin/install -c .libs/cjpeg /opt/mozjpeg/bin/cjpeg
libtool: install: /usr/bin/install -c .libs/djpeg /opt/mozjpeg/bin/djpeg
libtool: install: /usr/bin/install -c .libs/jpegtran /opt/mozjpeg/bin/jpegtran
libtool: install: /usr/bin/install -c .libs/rdjpgcom /opt/mozjpeg/bin/rdjpgcom
libtool: install: /usr/bin/install -c .libs/wrjpgcom /opt/mozjpeg/bin/wrjpgcom
libtool: install: /usr/bin/install -c .libs/tjbench /opt/mozjpeg/bin/tjbench
 ./install-sh -c -d '/opt/mozjpeg/share/doc/mozjpeg'
...
opts/libjpeg.pc pkgscripts/libturbojpeg.pc '/opt/mozjpeg/lib64/pkgconfig'
make[2]: Leaving directory '/mozjpeg'
make[1]: Leaving directory '/mozjpeg'
INFO[0098] Taking snapshot of full filesystem...        
INFO[0098] Resolving paths                              
INFO[0099] Saving file opt/mozjpeg/bin/jpegtran for later use 
INFO[0099] Saving file opt/mozjpeg/bin/cjpeg for later use 
INFO[0099] Saving file opt/mozjpeg/lib64 for later use  
INFO[0099] Deleting filesystem...                       
INFO[0099] Retrieving image manifest alpine:3.9         
INFO[0100] Retrieving image manifest alpine:3.9         
INFO[0101] Executing 0 build triggers                   
INFO[0101] Unpacking rootfs as cmd COPY --from=mozjpeg-builder /opt/mozjpeg/bin/jpegtran /opt/mozjpeg/bin/cjpeg /usr/bin/ requires it. 
INFO[0101] COPY --from=mozjpeg-builder /opt/mozjpeg/bin/jpegtran /opt/mozjpeg/bin/cjpeg /usr/bin/ 
INFO[0101] Resolving paths                              
INFO[0101] Taking snapshot of files...                  
INFO[0101] COPY --from=mozjpeg-builder /opt/mozjpeg/lib64 /usr/lib/ 
INFO[0101] Resolving paths                              
INFO[0101] Taking snapshot of files...

@tejal29
Copy link
Member

tejal29 commented May 4, 2020

Resolved by #1225

@tejal29 tejal29 closed this as completed May 4, 2020
@AndreKR
Copy link
Author

AndreKR commented May 4, 2020 •
edited

I can still reproduce this with v0.20.0.

I eventually managed to make the test case much simpler:

FROM alpine:3.9 AS builder
RUN mkdir -p /opt/foo/bar
RUN touch /opt/foo/bar/a
RUN ln -s a /opt/foo/bar/s

FROM alpine:3.9
COPY --from=builder /opt/foo/bar /opt/baz/

And here is the full log:
https://gitlab.com/AndreKR/kaniko-test/-/jobs/537737323

The error is:

 INFO[0002] Taking snapshot of full filesystem...        
 INFO[0002] Resolving 481 paths                          
 INFO[0002] Saving file opt/foo/bar for later use        
 error building image: could not save file: symlink a /kaniko/0/opt/foo/bar/s: file exists

@AndreKR
Copy link
Author

AndreKR commented Jul 24, 2021

I have updated the repro repo to kaniko v1.6.0 and the issue persists.

Command line
Dockerfile
Full log

@AndreKR AndreKR mentioned this issue Jul 24, 2021
Open
@stratusjerry stratusjerry mentioned this issue Nov 29, 2021
Open
@hazam
Copy link

hazam commented Dec 23, 2022

v1.9.1-debug, issue still there

@Mistic92
Copy link

I have the same issue with 1.9.1

@tjohnston-cd tjohnston-cd mentioned this issue Feb 16, 2023
Closed
@aaron-prindle
Copy link
Collaborator

Dupe of #1406, if you are encountering this issue or want to continue the discussion please do so there thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
needs-reproduction
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@hazam @AndreKR @tejal29 @aaron-prindle @Mistic92