Layer digest mismatch bug #37
Comments
|
@nktpro Hm, i'm not able to reproduce: Are you on |
|
@nlewo I was testing this in an The use case, which I believe is quite common among devs, is developing using Macs (darwin) with remote building Nix Linux servers. Essentially one would
For some reasons the calculated digests based on the local /nix store on Darwin differ from Linux, very much like that other issue that had a workaround. Most packages I tested so far worked fine though, so there's something odd about that |
@nktpro if it's not too late, i would be interested in. You can ping me on matrix (lewo on the nixos chan) or by email with details. Another approach would be to generate archive with |
|
@nktpro first of all, thanks for the remote mac access ;) When we build the image ( When we copy the image with Skopeo as user, Skopeo see this file instead: Since filename are differents, the tar checksum differ (maybe there are some other differences). Because I know nothing about macos, macos filesystem, nix hacks for macos, i don't know how we could fix this issue yet... |
|
The digest mismatch is due to the fact the MacOS filesystem ( When the However, when Nix copies the built closure on the MacOS FS, which is case-insensitive, it needs to encode filenames to avoid collisions ( So, that's a weird issue... and i see two solutions.
|
|
Thanks @nlewo for looking into this! That's indeed one heck of an issue...
I quickly tested this approach on a few existing real-world image builds, and the rabbit hole is quite deep. So far I found 3 very common transitive dependencies that share this same problem, which are
This definitely sounds viable. It's a bit of extra "magic" to reverse the prior "magic" from Nix daemon, but does not share any shortcomings of the 1st route.
As far as I have seen, generating the image.json file natively on macOS has never been practically viable vs. remote building the derivation on a Linux machine with the right arch. It wouldn't work anyway because there's almost always intermediate derivations that require being on the right arch to build, especially when building multi-arch images (targeting both amd64 and arm64) The only practical use case is being able to efficiently stream-push the downloaded nix store (whether from the remote builders or straight from a nix binary cache) to a local Docker daemon or local container registry for rapid testing. It's so efficient with nix2container because most of the time all the different store paths are already cached locally on the Macs once, which makes the pushes near instant. |
This build flag allows to publish an image on a case insensitive FS. Otherwise, hash can differs as detailled in #37 Fixes !37
This build flag allows to publish an image on a case insensitive FS. Otherwise, hash can differs as detailled in #37 Fixes !37
This build flag allows to publish an image on a case insensitive FS. Otherwise, hash can differs as detailled in #37 Fixes !37
This build flag allows to publish an image on a case insensitive FS. Otherwise, hash can differs as detailled in #37 Fixes !37
This build flag allows to publish an image on a case insensitive FS. Otherwise, hash can differs as detailled in #37 Fixes !37
|
So, #55 should improve the situation. This would allow us to avoid the FS file order issue ;) |
This build flag allows to publish an image on a case insensitive FS. Otherwise, hash can differs as detailled in #37 Fixes !37
This build flag allows to publish an image on a case insensitive FS. Otherwise, hash can differs as detailled in #37 Fixes !37
This build flag allows to publish an image on a case insensitive FS. Otherwise, hash can differs as detailled in #37 Fixes !37
|
Fixed by #48. |
|
Sorry to comment on an old issue, but I'm encountering a "digest mismatch" when pushing images to gitlab and to ECR on a x86_64-linux docker container running on an AWS EC2 instance. I'm not sure of the best way to diagnose the problem. Could one of you point me in the right direction? I would really appreciate it! |
|
@purefn It could be because of some differences between storepaths read at buildtime from storepaths read at runtime. At buildtime, nix2container generates the hash of the layer which is then also computed by the registry when the image is pushed (at runtime). So, the first thing to do is to compare buildtime and runtime layers. To do that, you could first start by generating the layer at runtime: you can run Let me know if i need to provide more detail. (Ideally, i should write a guide to do that...) |
|
Unfortunately I ended up needing to restructure somethings because I was using |
|
This is very strange. I'm not sure what is going on. Here's what I just discovered. I had a build which was able to successfully push the docker image. In the build, this command was run by the There were some comments on review, so I made some changes to a few scripts. These changes did not affect the docker image built. I see almost the same command as above being run - only the tag is different, but it fails Comparing the digests of the blobs that are being copied, they are the exact same between the two runs, as you would expect, just in different orders. To try and figure out what was going on, I tried adding Any suggestions on what to try next? |
Thank you for this wonderful tool!
I ran into a weird digest mismatch bug when a layer (transitively) includes the
ncursespackage. Here's a reproducer:In a fresh flake-based project that depends on
nix2container, given thisflake.nixfilenix build ...generates the image config with no issues:nix build -L -v '.#packages.x86_64-linux.reproducer'However,
skopeo copy ...will fail:with the following error:
In a more complex image with lots of other layers, the digests would match perfectly fine, as long as a layer doesn't include that
ncursespackage, for whatever reason :(The text was updated successfully, but these errors were encountered: