Multistage builds > Error building image: could not save file: copying file: write /kaniko/0/dev/full: no space left on device

[indianscout]

I am encountering a strange issue when doing multistage build with kaniko.
When using COPY --from= in the second stage, kaniko seems to eat up all diskspace in /var/lib/docker (using docker-ce):

$ df -h
...
/disks/docker   2T  2T   53M 100% /var/lib/docker
...

The console output when doing a kaniko build looks like:

 $ docker run -v $(pwd):/workspace gcr.io/kaniko-project/executor:latest --dockerfile=./Dockerfile --context=/workspace --no-push
INFO[0000] Resolved base name registry.access.redhat.com/ubi8/ubi-minimal:8.6-902 to dependencies 
INFO[0000] Retrieving image manifest registry.access.redhat.com/ubi8/ubi-minimal:8.6-902 
INFO[0000] Retrieving image registry.access.redhat.com/ubi8/ubi-minimal:8.6-902 from registry registry.access.redhat.com 
INFO[0000] Retrieving image manifest registry.access.redhat.com/ubi8/ubi-minimal:8.6-902 
INFO[0000] Returning cached image manifest              
INFO[0000] Built cross stage deps: map[0:[.]]           
INFO[0000] Retrieving image manifest registry.access.redhat.com/ubi8/ubi-minimal:8.6-902 
INFO[0000] Returning cached image manifest              
INFO[0000] Executing 0 build triggers                   
INFO[0000] Building stage 'registry.access.redhat.com/ubi8/ubi-minimal:8.6-902' [idx: '0', base-idx: '-1'] 
INFO[0000] Unpacking rootfs as cmd RUN echo "Hello stage 1"   && touch A_FILE_PATH requires it. 
INFO[0003] ARG USER=nobody                              
INFO[0003] ARG A_FILE_PATH=/usr/bin/a_file              
INFO[0003] RUN echo "Hello stage 1"   && touch A_FILE_PATH 
INFO[0003] Initializing snapshotter ...                 
INFO[0003] Taking snapshot of full filesystem...        
INFO[0004] Cmd: /bin/sh                                 
INFO[0004] Args: [-c echo "Hello stage 1"   && touch A_FILE_PATH] 
INFO[0004] Running: [/bin/sh -c echo "Hello stage 1"   && touch A_FILE_PATH] 
Hello stage 1
INFO[0004] Taking snapshot of full filesystem...        
INFO[0004] Saving file . for later use                  
error building image: could not save file: copying file: write /kaniko/0/dev/full: no space left on device

This does not happen when I build the image with docker-ce:

$ docker build - < Dockerfile
Sending build context to Docker daemon  2.048kB
Step 1/9 : FROM registry.access.redhat.com/ubi8/ubi-minimal:8.6-902 AS dependencies
8.6-902: Pulling from ubi8/ubi-minimal
a96e4e55e78a: Pull complete 
67d8ef478732: Pull complete 
Digest: sha256:6e79406e33049907e875cb65a31ee2f0575f47afa0f06e3a2a9316b01ee379eb
Status: Downloaded newer image for registry.access.redhat.com/ubi8/ubi-minimal:8.6-902
 ---> c9882b8114e3
Step 2/9 : ARG USER=nobody
 ---> Running in 1aa898089bf3
Removing intermediate container 1aa898089bf3
 ---> da20079ed534
Step 3/9 : ARG A_FILE_PATH=/usr/bin/a_file
 ---> Running in fd2f43ef6a26
Removing intermediate container fd2f43ef6a26
 ---> b96ec1468dbd
Step 4/9 : RUN echo "Hello stage 1"   && touch A_FILE_PATH
 ---> Running in 0b98f322dda8
Hello stage 1
Removing intermediate container 0b98f322dda8
 ---> 2c74725b6be9
Step 5/9 : FROM registry.access.redhat.com/ubi8/ubi-minimal:8.6-902
 ---> c9882b8114e3
Step 6/9 : ARG USER=nobody
 ---> Using cache
 ---> da20079ed534
Step 7/9 : COPY --from=dependencies ${A_FILE_PATH} ${A_FILE_PATH}
 ---> 2220a4d5ab88
Step 8/9 : RUN echo "Hello stage 2"
 ---> Running in 0efdca439c1e
Hello stage 2
Removing intermediate container 0efdca439c1e
 ---> aabbf5dabd6c
Step 9/9 : USER nobody
 ---> Running in 67b47ba45a95
Removing intermediate container 67b47ba45a95
 ---> b4ca6fa04f00
Successfully built b4ca6fa04f00

The Dockerfile to reproduce this looks like:

FROM registry.access.redhat.com/ubi8/ubi-minimal:8.6-902 AS dependencies

ARG USER=nobody
ARG A_FILE_PATH=/usr/bin/a_file

RUN echo "Hello stage 1" \
  && touch A_FILE_PATH

FROM registry.access.redhat.com/ubi8/ubi-minimal:8.6-902

ARG USER=nobody

COPY --from=dependencies ${A_FILE_PATH} ${A_FILE_PATH}
RUN echo "Hello stage 2"

USER nobody

The fact that kaniko makes docker to eat up all disk-space is causing other docker builds or containers to fail - which is a pretty severe side effect.

I could imagine that this is related with:

• Image build process Freezes on Taking snapshot of full filesystem... #1333
• Snapshotting is taking > 10 minutes #970

[indianscout]

Any news on this, I just ran a test and the problem still persists:

$ docker run -v $(pwd):/workspace gcr.io/kaniko-project/executor:latest --dockerfile=./Dockerfile --context=/workspace --no-push
Unable to find image 'gcr.io/kaniko-project/executor:latest' locally
latest: Pulling from kaniko-project/executor
9d4299bbd943: Pull complete 
c1848f67e812: Pull complete 
8b2826afefe3: Pull complete 
21629cece012: Pull complete 
0fec4bb73568: Pull complete 
7ce3ceba4a72: Pull complete 
6d0fb28d82cd: Pull complete 
7f04a316205d: Pull complete 
a8dae3110e38: Pull complete 
Digest: sha256:1f982af0b54be748221d9a35dcfa608660ab3d51229aa56bde5416f75aff7561
Status: Downloaded newer image for gcr.io/kaniko-project/executor:latest
INFO[0000] Resolved base name registry.access.redhat.com/ubi8/ubi-minimal:8.6-902 to dependencies 
INFO[0000] Retrieving image manifest registry.access.redhat.com/ubi8/ubi-minimal:8.6-902 
INFO[0000] Retrieving image registry.access.redhat.com/ubi8/ubi-minimal:8.6-902 from registry registry.access.redhat.com 
INFO[0001] Retrieving image manifest registry.access.redhat.com/ubi8/ubi-minimal:8.6-902 
INFO[0001] Returning cached image manifest              
INFO[0001] Built cross stage deps: map[0:[.]]           
INFO[0001] Retrieving image manifest registry.access.redhat.com/ubi8/ubi-minimal:8.6-902 
INFO[0001] Returning cached image manifest              
INFO[0001] Executing 0 build triggers                   
INFO[0001] Building stage 'registry.access.redhat.com/ubi8/ubi-minimal:8.6-902' [idx: '0', base-idx: '-1'] 
INFO[0001] Unpacking rootfs as cmd RUN echo "Hello stage 1"   && touch A_FILE_PATH requires it. 
INFO[0004] ARG USER=nobody                              
INFO[0004] ARG A_FILE_PATH=/usr/bin/a_file              
INFO[0004] RUN echo "Hello stage 1"   && touch A_FILE_PATH 
INFO[0004] Initializing snapshotter ...                 
INFO[0004] Taking snapshot of full filesystem...        
INFO[0005] Cmd: /bin/sh                                 
INFO[0005] Args: [-c echo "Hello stage 1"   && touch A_FILE_PATH] 
INFO[0005] Running: [/bin/sh -c echo "Hello stage 1"   && touch A_FILE_PATH] 
Hello stage 1
INFO[0005] Taking snapshot of full filesystem...        
INFO[0005] Saving file . for later use                  
error building image: could not save file: copying file: write /kaniko/0/dev/full: no space left on device

[indianscout]

Opened a question on Stackoverflow, since there seems to be no activity to analyze this problem.
https://stackoverflow.com/questions/73828076/kaniko-multistage-builds-error-building-image-could-not-save-file-copying-fi

[wihanbooyse]

Can confirm we're also running into this

[eelke-at-bolt]

I can confirm the problem on a fresh install of ubuntu jammy with docker 20.10.12 and gcr.io/kaniko-project/executor:v1.9.1-debug. On older ubuntu focal with docker 20.10.12 it works (same kaniko).

[Elsayed91]

error building image: could not save file: copying file: stat /kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/0/kaniko/.docker: file name too long
ERROR
ERROR: build step 0 "gcr.io/kaniko-project/executor:latest" failed: step exited with non-zero status: 1

when attempting to use it in cloudbuild. using regular docker/cloud shell actually builds the image successfully. It only fails in cloudbuild

sample Dockerfile

FROM gcr.io/kaniko-project/executor:debug AS kaniko
FROM apache/spark-py
COPY --from=kaniko /kaniko//kaniko/ #fails here
WORKDIR /

[nekhebit]

I am facing the same problem in a gitlab CI pipeline. Even with the hint provided as answer to @indianscout in stack overflow, I still get the same error: "[...] file name too long". Locally (docker) it works, in the pipeline (kaniko - works for all other images) it does not.

[0x217]

I have the same problem.
Any updates? :)

[mennthor]

Getting the same problem running this in Gitlab CI with kaniko.
In the dockerfile I use the kaniko reference directly like

FROM PYTHON:3.10
COPY --from=gcr.io/kaniko-project/executor:debug /kaniko/ /kaniko/

and then it crashes with
"""
error building image: could not save file: copying file: open /kaniko/0[...many more of these]/kaniko/.docker/config.json: file name too long
"""
Does somebody know what's going on? Looks like some kind of recursive confusion.

[oallauddin]

# information found here:
# https://stackoverflow.com/questions/69182455/possible-to-add-kaniko-to-alpine-image-or-add-jq-to-kaniko-image

FROM gcr.io/kaniko-project/executor:debug AS kaniko
FROM python:3.10

# create /kaniko directory
RUN set -e \
    && mkdir -p /kaniko

# have to copy each file and directory seperately
# can't just copy the entire /kaniko directory causes error with repeating /kaniko/0
COPY --from=kaniko /kaniko/.docker/ /kaniko/.docker/
COPY --from=kaniko /kaniko/docker-credential-acr-env /kaniko/docker-credential-acr-env
COPY --from=kaniko /kaniko/docker-credential-ecr-login /kaniko/docker-credential-ecr-login
COPY --from=kaniko /kaniko/docker-credential-gcr /kaniko/docker-credential-gcr
COPY --from=kaniko /kaniko/executor /kaniko/executor
COPY --from=kaniko /kaniko/ssl/ /kaniko/ssl/
COPY --from=kaniko /kaniko/warmer /kaniko/warmer

# install kaniko cert
RUN set -e \
    && cp /kaniko/ssl/certs/ca-certificates.crt/usr/local/share/ca-certificates/kaniko-ca-certificates.crt \
    && update-ca-certificates
	
# kaniko environment variables
ENV DOCKER_CONFIG=/kaniko/.docker/
ENV PATH="${PATH}:/kaniko"

[iamkhalidbashir]

Same issue

[Lord-Y]

same here

[RajithRajan]

same here. Any solution found?

[JoA-MoS]

I am not sure if this might help with the issue but on gitlab I was building a new image with kaniko that was using v1.20.0 but in my Dockerfile that I was trying to build an image with 1.16.0. I specified FROM gcr.io/kaniko-project/executor:v1.16.0 as kaniko in the Dockerfile but for some reason after build of the docker image the kaniko version was 1.20.0. I am wondering if some how the container executor version of kaniko is being copied.

[fschlz]

Also just ran into this issue on v1.9.2-debug, but after upgrading to v1.22.0-debug builds seem stable again.