Multi-Stage Image builds are failing at Stage 0 as the filesystem is being deleted.

[Srinivask9000]

Kaniko fails to store the multistage image at stage 0 - as it works fine with traditional docker files. After setting "--verbosity debug"
I found that it is deleting the entire file system on the Jenkins-Agent.

Dockerfile
`
FROM ubuntu:20.04 AS base
ENV PYTHONUNBUFFERED 1

COPY Pipfile.lock /app/

RUN apt-get update &&
apt-get install --no-install-recommends -y python3 python-is-python3 pipenv gnupg &&
pip install --no-cache-dir --upgrade pip &&
apt-get autoremove -y &&
apt-get clean -y

COPY . /app
WORKDIR /app

FROM base AS dev
RUN apt-get install --no-install-recommends -y vim git &&
pipenv install --system --deploy --dev &&
python manage.py collectstatic --noinput -c

FROM base AS production
RUN pipenv install python-crontab &&
pipenv install --system --deploy &&
python manage.py collectstatic --noinput -c
`

To Reproduce
Steps to reproduce the behavior:

1. Running the kaniko with the below steps in Jenkins agent

/kaniko/executor --context /home/jenkins/workspace/Kaniko-Image-Build-DL/Test-Image-Build/dataloader --dockerfile /home/jenkins/workspace/Kaniko-Image-Build-DL/Test-Image-Build/dataloader/Dockerfile --destination ********/kanikobuild:latest '--target=dev' --skip-tls-verify --verbosity debug

Additional Information

• I'm using the latest gcr.io/kaniko-project/executor:debug in Jenkins-agent

Description	Yes/No
Please check if this a new feature you are proposing
Please check if the build works in docker but not in kaniko	- [YES]
Please check if this error is seen when you use --cache flag
Please check if your dockerfile is a multistage dockerfile	- [ YES ]

[Srinivask9000]

can someone pls help me with a workaround/fix for this.

[sumkincpp]

@Srinivask9000 Could you please provide a minimal failing example and specific error message? Deleting filesystem is a normal behavior and not a problem.

I was able to build a project using latest gcr.io/kaniko-project/executor:debug

run.sh

docker run --rm -v "$(pwd):/workspace/:ro" \
    gcr.io/kaniko-project/executor:debug \
    --context=/workspace --dockerfile=/workspace/Dockerfile \
    --no-push --skip-tls-verify #--verbosity debug

Pipfile + pipenv lock

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[packages]
requests = "*"


[dev-packages]
pytest = "*"

./run.sh | grep INFO
INFO[0000] Resolved base name ubuntu:20.04 to base      
INFO[0000] Resolved base name base to dev               
INFO[0000] Resolved base name base to production        
INFO[0000] Retrieving image manifest ubuntu:20.04       
INFO[0000] Retrieving image ubuntu:20.04 from registry index.docker.io 
INFO[0001] Built cross stage deps: map[]                
INFO[0001] Retrieving image manifest ubuntu:20.04       
INFO[0001] Returning cached image manifest              
INFO[0001] Executing 0 build triggers                   
INFO[0001] Building stage 'ubuntu:20.04' [idx: '0', base-idx: '-1'] 
INFO[0001] Unpacking rootfs as cmd COPY Pipfile.lock /app/ requires it. 
INFO[0002] ENV PYTHONUNBUFFERED 1                       
INFO[0002] COPY Pipfile.lock /app/                      
INFO[0002] Taking snapshot of files...                  
INFO[0002] RUN apt-get update &&     apt-get install --no-install-recommends -y python3 python-is-python3 pipenv gnupg &&     pip install --no-cache-dir --upgrade pip &&     apt-get autoremove -y &&     apt-get clean -y 
INFO[0002] Initializing snapshotter ...                 
INFO[0002] Taking snapshot of full filesystem...        
INFO[0003] Cmd: /bin/sh                                 
INFO[0003] Args: [-c apt-get update &&     apt-get install --no-install-recommends -y python3 python-is-python3 pipenv gnupg &&     pip install --no-cache-dir --upgrade pip &&     apt-get autoremove -y &&     apt-get clean -y] 
INFO[0003] Running: [/bin/sh -c apt-get update &&     apt-get install --no-install-recommends -y python3 python-is-python3 pipenv gnupg &&     pip install --no-cache-dir --upgrade pip &&     apt-get autoremove -y &&     apt-get clean -y] 
debconf: delaying package configuration, since apt-utils is not installed
INFO[0050] Taking snapshot of full filesystem...        
INFO[0054] COPY . /app                                  
INFO[0054] Taking snapshot of files...                  
INFO[0054] WORKDIR /app                                 
INFO[0054] Cmd: workdir                                 
INFO[0054] Changed working directory to /app            
INFO[0054] No files changed in this command, skipping snapshotting. 
INFO[0054] Storing source image from stage 0 at path /kaniko/stages/0 
INFO[0055] Deleting filesystem...                       
INFO[0055] Base image from previous stage 0 found, using saved tar at path /kaniko/stages/0 
INFO[0055] Executing 0 build triggers                   
INFO[0056] Building stage 'base' [idx: '1', base-idx: '0'] 
INFO[0056] Unpacking rootfs as cmd RUN pipenv install --system --deploy --dev requires it. 
INFO[0059] RUN pipenv install --system --deploy --dev   
INFO[0059] Initializing snapshotter ...                 
INFO[0059] Taking snapshot of full filesystem...        
INFO[0061] Cmd: /bin/sh                                 
INFO[0061] Args: [-c pipenv install --system --deploy --dev] 
INFO[0061] Running: [/bin/sh -c pipenv install --system --deploy --dev] 
INFO[0067] Taking snapshot of full filesystem...        
INFO[0067] Deleting filesystem...                       
INFO[0067] Base image from previous stage 0 found, using saved tar at path /kaniko/stages/0 
INFO[0067] Executing 0 build triggers                   
INFO[0068] Building stage 'base' [idx: '2', base-idx: '0'] 
INFO[0068] Unpacking rootfs as cmd RUN pipenv install python-crontab &&     pipenv install --system --deploy requires it. 
INFO[0071] RUN pipenv install python-crontab &&     pipenv install --system --deploy 
INFO[0071] Initializing snapshotter ...                 
INFO[0071] Taking snapshot of full filesystem...        
INFO[0073] Cmd: /bin/sh                                 
INFO[0073] Args: [-c pipenv install python-crontab &&     pipenv install --system --deploy] 
INFO[0073] Running: [/bin/sh -c pipenv install python-crontab &&     pipenv install --system --deploy] 
Warning: the environment variable LANG is not set!
We recommend setting this in ~/.profile (or equivalent) for proper expected behavior.
Creating a virtualenv for this project…
Using /usr/bin/python3 (3.8.10) to create virtualenv…
created virtual environment CPython3.8.10.final.0-64 in 196ms
  creator CPython3Posix(dest=/root/.local/share/virtualenvs/app-4PlAip0Q, clear=False, global=False)
  seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, pkg_resources=latest, via=copy, app_data_dir=/root/.local/share/virtualenv/seed-app-data/v1.0.1.debian.1)
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator

Virtualenv location: /root/.local/share/virtualenvs/app-4PlAip0Q
Pipfile.lock (b6f97a) out of date, updating to (bbabdd)…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Updated Pipfile.lock (bbabdd)!
INFO[0094] Taking snapshot of full filesystem...        
INFO[0095] Skipping push to container registry due to --no-push flag

[Srinivask9000]

@sumkincpp Thanks for looking into this.

As I mentioned in the initial bug report, I'm facing this issue only while building the Image in Jenkins Agent not on local machine.

As you mentioned in your comment - it is working in my local to with docker run:

docker run -ti --rm -v /mnt/c/wsl/db/db:/workspace -v /mnt/c/wsl/config.json:/kaniko/.docker/config.json:ro gcr.io/kaniko-project/executor:debug --context /workspace --dockerfile /workspace/Dockerfile --target dev --no-push

But this kaniko build is failing only in Jenkins Agent, as it is deleting the core workspace for Jenkins which is needed to continue the multistage build(while using multistage dockerfile).

I'm initiating the build in jenkins using below script:

kaniko_build_uploadImage() {
    echo "Creating Kaniko Docker image with name, ${branchName}"
    sed -i -e "s/apt-get install --no-install-recommends -y python3/apt-get -o Dpkg::Options::="--force-confold" install --no-install-recommends -y python3/g" db/Dockerfile
    /kaniko/executor --context $(pwd)/db --dockerfile $(pwd)/db/Dockerfile --target=dev --destination $aws_id.dkr.ecr.$region.amazonaws.com/project/db:${branchName}
}

I am attaching the console log screenshot for your reference.

Pls let me know if you need any other details from my side.

[sumkincpp]

Kaniko does not work with plain call of /kaniko/executor on a VM - a whole empty root filesystem layout should be available for it at start, then for every stage kaniko will try reusing it, then when necessary emptying it, unpacking files, preparing them for the next stage, etc.

You should probably try using docker build agents or kubernetes in jenkins, both providing empty sandboxed environment (Container), but not directly running it in your jenkins node (VM)

PS. In Kubernetes Jenkins plugin you should probably change default workdir to /tmp/jenkins https://docs.cloudbees.com/docs/cloudbees-ci-kb/latest/cloudbees-ci-on-modern-cloud-platforms/what-you-need-to-know-when-using-kaniko-from-kubernetes-jenkins-agents

[Srinivask9000]

@sumkincpp I'm doing the same plain call of /kaniko/executor for building a few other images, the build & deployment
is working as expected without any filesystem deletion. Whereas this current image is totally different from others- this one is a multistage docker file while others are traditional ones.

The reason why I'm using Jenkins-Agent is: I'm setting up Jenkins on ECS Fargate & building Images on ECS Fargate Jenkins Agent containers.

And I finally got it working after using the flag "--ignore-path=" - which moved the jenkins workspace into out of the snapshot scope & proceeded to the next step.

I think It is mandatory to use --ignore-path for building multistage images on the Fargate Jenkins agent.

[aaron-prindle]

I believe this is resolved now from the above. As mentioned, when kaniko "cleans up" either via the --cleanup=true or when proceeding from one stage to another in a multi-stage Dockerfile kaniko deletes the filesystem EXCEPT FOR what is is it's set of "ignore-paths". To force kaniko to keep the jenkins workspace across stages/kaniko-executions using --ignore-path is required. Closing, feel free to comment and I can re-open if there is more here. Thanks

[qyqcswill]

Cool, it has bored me for almost a year.

[sachasmart]

Brutal.

[pablogrs]

Some useful info here.
https://docs.cloudbees.com/docs/cloudbees-ci-kb/latest/cloudbees-ci-on-modern-cloud-platforms/what-you-need-to-know-when-using-kaniko-from-kubernetes-jenkins-agents

[LeBakii]

@sumkincpp I'm doing the same plain call of /kaniko/executor for building a few other images, the build & deployment is working as expected without any filesystem deletion. Whereas this current image is totally different from others- this one is a multistage docker file while others are traditional ones.

The reason why I'm using Jenkins-Agent is: I'm setting up Jenkins on ECS Fargate & building Images on ECS Fargate Jenkins Agent containers.

And I finally got it working after using the flag "--ignore-path=" - which moved the jenkins workspace into out of the snapshot scope & proceeded to the next step.

I think It is mandatory to use --ignore-path for building multistage images on the Fargate Jenkins agent.

@Srinivask9000 hi, i have same setup and same issue with you. But I coulnd't understand the use of --ignore-path. Can you explain a little bit more use of --ignore-path in the project ? Or which path ?