Missing signature failures get cached permanently

[kirelagin]

Context:

• haskell.nix builds with a lot of IFD (which, I suppose, might be a factor here)
• our own private cache

At some point we accidentally pushed certain paths to our cache without including signatures. A number of people then tried to build the derivation and reported substitution failures due to this. We realised the mistake and pushed the missing signatures, however substitution failures persisted.

Here is what it looks like:

$ nix build /nix/store/ncj6d5wjlw2akq64rvdi5mmzfrb7x2qa-hpack-exe-hpack-0.34.1
warning: --- Invalid path signature --- nix-daemon
substituter 's3://serokell-private-cache' does not have a valid signature for path '/nix/store/ncj6d5wjlw2akq64rvdi5mmzfrb7x2qa-hpack-exe-hpack-0.34.1'
error: error: --- Error --- nix-daemon
build of '/nix/store/ncj6d5wjlw2akq64rvdi5mmzfrb7x2qa-hpack-exe-hpack-0.34.1' failed

# The path is signed
$ nix verify --store $STORE /nix/store/ncj6d5wjlw2akq64rvdi5mmzfrb7x2qa-hpack-exe-hpack-0.34.1
[1 paths verified]

$ nix path-info --store $STORE --sigs /nix/store/ncj6d5wjlw2akq64rvdi5mmzfrb7x2qa-hpack-exe-hpack-0.34.1
/nix/store/ncj6d5wjlw2akq64rvdi5mmzfrb7x2qa-hpack-exe-hpack-0.34.1	hydra.iohk.io:qSwuHykCYdFW70hVZ0syfCDBsE99aZA+/aTkAWuidNTt1iL4sOjDAa/9v+/V3gU2whuVV7TjsRY7F5ug0qd5Bw== serokell-1:zpGOTlODT0oWDL9vC6Vq+dnGx0uDXu7qspdr6JhYVItGOwtP1IKhEcyqYNpVj6KCUx4O00pJHehttObu/yAEAQ==

(serokell-1 is the trusted key here)

It is interesting that these paths ended up in a sort of weird state, but I don’t think this is relevant here, so I’ll create a separate issue.

Digging deeper:

$ sqlite3 ~/.cache/nix/binary-cache-v6.sqlite
sqlite> select * from nars where hashPart='ncj6d5wjlw2akq64rvdi5mmzfrb7x2qa';
1|ncj6d5wjlw2akq64rvdi5mmzfrb7x2qa|hpack-exe-hpack-0.34.1|nar/1abzshwj4g0d2qv3mnf4ygi6bgj30c876qlws2xwd89s4hysb0n9.nar.xz|xz|sha256:1abzshwj4g0d2qv3mnf4ygi6bgj30c876qlws2xwd89s4hysb0n9|1876564|sha256:1042xf1qb5zmj3rzl1p4j59a23q7z6cda9r490iiqg20h6bdp18p|11040888|6ihjyc8ylagb4s7mbg02zbazldwwwjzk-libffi-3.3 b3zsk4ihlpiimv3vff86bb5bxghgdzb9-gcc-9.2.0 danv012gh0aakh8xnk2b35vahklz72mk-gcc-9.2.0-lib fwpn2f7a4iqszyydw7ag61zlnp6xk5d3-glibc-2.30-dev h38pd7vcaqsaanv6z3qw1i1917kzppbb-numactl-2.0.13 m39igjlhfbkac44dxnyzmv40716ra632-libffi-3.3-dev msp4hm62a75pdidlc3s2ymma2g5hsjjk-zlib-1.2.11 ncj6d5wjlw2akq64rvdi5mmzfrb7x2qa-hpack-exe-hpack-0.34.1 vw909fhky273h9waklv0fm7m7bn0j4qy-gmp-6.2.0 xg6ilb9g9zhi2zg1dpi4zcp288rhnvns-glibc-2.30|r42mr0dhm3ayif5r1631nil5a7mkhiyg-hpack-exe-hpack-0.34.1.drv|hydra.iohk.io:qSwuHykCYdFW70hVZ0syfCDBsE99aZA+/aTkAWuidNTt1iL4sOjDAa/9v+/V3gU2whuVV7TjsRY7F5ug0qd5Bw== serokell-1:zpGOTlODT0oWDL9vC6Vq+dnGx0uDXu7qspdr6JhYVItGOwtP1IKhEcyqYNpVj6KCUx4O00pJHehttObu/yAEAQ==||1605286662|1

This cache is correct and nuking it has no effect. What matters is the root cache:

$ sudo sqlite3 ~root/.cache/nix/binary-cache-v6.sqlite
sqlite> select * from nars where hashPart='ncj6d5wjlw2akq64rvdi5mmzfrb7x2qa';
2|ncj6d5wjlw2akq64rvdi5mmzfrb7x2qa|hpack-exe-hpack-0.34.1|nar/1abzshwj4g0d2qv3mnf4ygi6bgj30c876qlws2xwd89s4hysb0n9.nar.xz|xz|sha256:1abzshwj4g0d2qv3mnf4ygi6bgj30c876qlws2xwd89s4hysb0n9|1876564|sha256:1042xf1qb5zmj3rzl1p4j59a23q7z6cda9r490iiqg20h6bdp18p|11040888|6ihjyc8ylagb4s7mbg02zbazldwwwjzk-libffi-3.3 b3zsk4ihlpiimv3vff86bb5bxghgdzb9-gcc-9.2.0 danv012gh0aakh8xnk2b35vahklz72mk-gcc-9.2.0-lib fwpn2f7a4iqszyydw7ag61zlnp6xk5d3-glibc-2.30-dev h38pd7vcaqsaanv6z3qw1i1917kzppbb-numactl-2.0.13 m39igjlhfbkac44dxnyzmv40716ra632-libffi-3.3-dev msp4hm62a75pdidlc3s2ymma2g5hsjjk-zlib-1.2.11 ncj6d5wjlw2akq64rvdi5mmzfrb7x2qa-hpack-exe-hpack-0.34.1 vw909fhky273h9waklv0fm7m7bn0j4qy-gmp-6.2.0 xg6ilb9g9zhi2zg1dpi4zcp288rhnvns-glibc-2.30|r42mr0dhm3ayif5r1631nil5a7mkhiyg-hpack-exe-hpack-0.34.1.drv|hydra.iohk.io:qSwuHykCYdFW70hVZ0syfCDBsE99aZA+/aTkAWuidNTt1iL4sOjDAa/9v+/V3gU2whuVV7TjsRY7F5ug0qd5Bw==||1604196422|1
1|ncj6d5wjlw2akq64rvdi5mmzfrb7x2qa||||||||||||1605290172|0

As you can see, this cache does not have the serokell-1 signature. Nuking the root cache fixed the issue.

I am not sure what would be the expected behaviour here, I would say, retry every time when a substituter has no valid signatures or, at least, let them expire.

[kirelagin]

Also note that the cached result had another signature, which was not trusted, – I am not sure if this was a factor, but could be.

[kirelagin]

Looks like #2127 is kind of related.

[DavHau]

I was running into the same problem. I first published my cache with either a missing or a wrong signing key.

Even after correcting the key, some host which has been accessing the cache before that, would refuse to accept any of the earlier accessed paths which it already had determined as untrusted.
nix-collect-garbage didn't help

Removing /root/.cache/nix on the rejecting host solved the issue.
According to #2127 (comment), this cache times out only after one month which seems way to long. Maybe this should be set to 1h, similar to the default tarball-ttl?

[stale]

I marked this as stale due to inactivity. → More info

[neosimsim]

I have the same issue when updating signatures. Here is what I did.

First I signed the derivation and filled the cache.

nix-build
nix sign-paths -r --key-file my-key.private -f .
nix copy --to 's3://my-cache' -f .

To test the cache I did

nix-collect-garbage -d
nix-build

Which then successfully fetches binaries from the cache.

But when I generate a new key

nix-store --generate-binary-cache-key my-cache-1 my-cache.private my-cache.public

and update the public key in my NixOS configuration followed by nixos-rebuild, the new signature is not accepted.

To reproduce this I had to clear my cache, because I realized that the narinfo of an already cached derivation does
not get updated. This might be another issue in fact:

aws s3 rm --recurive s3://my-cache

Then I repeated the process.

nix-build
nix sign-paths -r --key-file my-key.private -f .
nix copy --to 's3://my-cache' -f .

and again to test the cache

nix-collect-garbage -d
nix-build

which then gives me

warning: substituter 's3://sonnen-vpp-nix-cache' does not have a valid signature for path

and rebuilds the cached derivations.