-
Notifications
You must be signed in to change notification settings - Fork 1.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Generated image is missing files generated via RUN #3123
Comments
FYI, I looked at other tickets with a similar problem (e.g., #2336), but either the root-cause described in those tickets is different or the proposed work-around did not work for me. I have tried many different work-arounds, and none worked for me (aside from touching every file in the file-system, but this is not an option for me) |
another observation: if I change this to a multi-stage build AND I do more than just use
Build command (note that I'm not pushing remotely just to safe roundtrip time, pushing to a remote registry has the same outcome):
Test command:
In some cases we get this result (installation worked):
In other cases we get this result (installated files were not committed to the snapshot/image):
As said, it's random and about 50% to have the one or other outcome. And even more weirdly, it seems to alternate if it works or if it fails. As if a cache would corrupt and then uncorrupt itself (note that in these experiments the cache is off). I have captured the stdout (build command outputs) + stderr (kamiko debug-verbosity logs) from a successful and failing build. |
Hi @clemenskol , did you find any workaround? |
unfortunately no. We had to move away from kaniko - it was the only "solution" that worked |
Same issue here and I'm pretty sure that we are not the only one having it..... @anoop142 , anybody able to reproduce on your side ? |
For me the basic case that fails is
grep: /home/foo: No such file or directory While this works
Seems like kaniko is skipping layers when EOF is used for RUN. |
Ok. At least seems not to be the case of @clemenskol. As far as I'm concern I do not use EOF too but, if this could be a problem, I'm doing it inside a Gitlab CI job. Best, |
@jrevillard you are right, skipping EOF command is indeed a different issue #1713. |
Actual behavior
Files generated via a
RUN
command should be included in the final image (e.g., regardless of file generation timestamp). This seems not to be the case.I have generated a minimal
Dockerfile
to demonstrate this:When building the image, the
python3.11
command is not property installed in the generated image, although it's clearly present while building.My build command:
The output of the final 2 commands can be seen in the build output:
When the generated image is then run, the file is not found (
python3.11
) simply does not exist.To test if this has to do with file timestamps, I have done the following modification:
In this case, the
python3.11
binary is in the generated image, but since it's not just the binary itself that is missing (but essentially most files installed viaapt
), the image is completely non-functional:Note that I have tried various alternatives using or not using
--cache
and using different--snapshot-mode
Expected behavior
All files are stored in the generated image.
If I build the image using the
Dockerfile
above viadocker buildx build
the image works as expected:To Reproduce
Steps to reproduce the behavior:
Dockerfile
aboveAdditional Information
--cache
flagThe text was updated successfully, but these errors were encountered: