Current deployed WASM does not match the docker built version of the WASM

[icdev2dev]

The two methods of producing sha256sum for the CURRENT wasm do not seem to match.

FROM THE PAST: I was able to reproduce the build from the commit mentioned in the 5 month old article at https://www.joachim-breitner.de/blog/779-Verifying_the_code_of_the_Internet_Identity_service. i.e. the SHA256SUM was the same as mentioned in your article: d4af92…c04.

CURRENT WASM
I used the Update 20211115T1700Z.md (3 days ago) with the commit 1373a58 as it seemed to have the latest commit.

The dfx reported hash:
through dfx canister --no-wallet --network ic info rdmx6-jaaaa-aaaaa-aaadq-cai
0xfe00bbec790a6db930f6c826623d3831597227f71852e5538065099afad416b3

The docker build reported hash:
through sudo docker build -t internet-identity-service .
6b854cfffb919c1ec28c728e3ec9e594c86ed359343d8d200907f4ac2a163538

[nmattia]

That's odd indeed. The CI run does report the hash that's deployed:

Run sha256sum out/internet_identity.wasm
fe00bbec790a6db930f6c826623d3831597227f71852e5538065099afad416b3  out/internet_identity.wasm

and I remember the first few chars from that hash (i.e. I remember getting fe00 locally). However now I cannot reproduce it:

~/internet-identity$ docker build -t internet-identity-release . && docker run --rm internet-identity-release cat /internet_identity.wasm | shasum -a 256
Sending build context to Docker daemon  2.971MB
Step 1/27 : FROM ubuntu:20.10
...

6b854cfffb919c1ec28c728e3ec9e594c86ed359343d8d200907f4ac2a163538  -

What's odd is that it does seem to have changed at some point, I'm wondering if one dependency changed upstream in spite of the lockfiles. I will need to investigate, thanks for reporting!

[nomeata]

How good is Rust's package locking? Does it still allow maybe for variation in minor versions or in indirect dependencies? Or maybe rustup isn't pinning the rust release completely? Or Ubuntu packages like llvm have changed?

[nmattia]

In terms of crates/dependencies I think we're safe, the lockfiles record the shas as well. Indeed I'm suspecting an Ubuntu package, or even the rust toolchain that may have been upgraded.

[nmattia]

I've added some logs in #490 (and pinned the docker image), I don't know how reproducible we can get but hopefully this will help a bit.

[bitdivine]

Also, please clear the docker cache before reproducible builds, otherwise they seem to be reproducible only 96.4% of the time...

docker image prune -a

I would love to have a reproducible example of non-reproducible builds. Then we could fix this.

[nmattia]

For the past few months we've had daily tests running without docker to make sure the latest release can be reproduced bit for bit. Feel free to re-open if you run into the issue again.