docker pull is failing with 'filesystem layer verification failed for digest' for the image built by Kaniko

[vmamidala-conga]

Actual behavior
Image pull is failing with the image built by kaniko.

Expected behavior
Image pull should work

To Reproduce
Steps to reproduce the behavior:

1. Build the image with kaniko builder
2. Push the image to Jfrog
3. Pull the image from Jfrog using docker pull

Kaniko builder Dockerfile which is used to build the Image where the issue is observed

FROM gcr.io/kaniko-project/executor:latest AS kaniko
FROM centos:7
COPY --from=kaniko /kaniko /kaniko
RUN mkdir /workspace /cache -p
RUN yum install curl make -y
RUN yum install epel-release -y
RUN yum install jq -y
ENV PATH=$PATH:/kaniko \
    SSL_CERT_DIR=/kaniko/ssl/certs \
    DOCKER_CONFIG=/kaniko/.docker/

Dockerfile

FROM mcr.microsoft.com/dotnet/aspnet:2.1
ARG CSPROJ_PATH
EXPOSE 80
RUN apt-get update
WORKDIR /app
COPY ./${CSPROJ_PATH}/app .

Docker pull logs -- Failure logs

b039c7b9ef55: Download complete
88da81787506: Download complete
df1c457cc6ce: Download complete
cde52ecc2150: Verifying Checksum
02660c91a47e: Download complete
a4b8e53b8487: Download complete
9742a8248880: Download complete
2c5f6ac62125: Download complete
fabd12b83225: Download complete
fec1b8d68c7a: Download complete
filesystem layer verification failed for digest sha256:cde52ecc21505bf895acc188fe746b11ab0db500252e79e2d1703ddb70869319

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing
Please check if the build works in docker but not in kaniko
Please check if this error is seen when you use --cache flag
Please check if your dockerfile is a multistage dockerfile

[imjasonh]

Thanks for filing this bug. A few questions:

1. Are you reliably able to successfully build an image using your Kaniko image that produces pushable but un-pullable image layers?

• specifically, pulling on different machines, to eliminate cached bad layers?
• and/or, can you docker system prune to clear your layer cache?

2. Are you able to reproduce this at all with the "standard" Kaniko image, without your centos base?

If you have a public image you could share that might help narrow down the problem.

[vmamidala-conga]

Thank your for quick response @imjasonh ,

1. Yes, successfully built the image but is un-pullable.
   yes, I did docker prune and tried but same issue.
2. Yes, I could able to reproduce the same issue with 'standard' kaniko image. (standalone debug image I used)

I don't have the public image. Any more suggestion for testing?

[vmamidala-conga]

Update from my findings,

This issue is observed for large size images,
Image size~ 1GB

I could able to overcome the issue with Image (1GB) by using --cache=true flag along with kaniko executor.

may be this could helpful if anyone facing similar issue.

[vmamidala-conga]

Seems issue is still persist with --cache=true flag, this needs to be consider as bug.

[JanSchutteAdyen]

I've had a similar issue, also seemed to occur on large image layers. I was able to fix it by adding --use-new-run flag, can you give this a try? I am using kaniko v1.7.0

EDIT: Seems that this does not resolve this issue after all, it came back after a while

[kumprj]

Seems issue is still persist with --cache=true flag, this needs to be consider as bug.

We disabled caching for all runs due to this, unfortunately.

[roopaax]

Issue still exists with 1.7.0 version. Its occurs randomly

[roopaax]

  Warning  Failed     38s                kubelet            Error: ImagePullBackOff
  Warning  Failed     38s                kubelet            Failed to pull image "lfklekfe.docker.jfrog.io/test/test/base/frontend:latest": rpc error: code = Unknown desc = filesystem layer verification failed for digest sha256:f0f0a774c3043bbaeb626812e218390fd7dac743180756aeac9329cfff439bae