Current ghc-8.8.3 has been corrupted in Hydra cache

[peti]

The downloaded store path for the current master version of ghc-8.8.3 is corrupt. I guess the build has worked fine on Hydra, but then something went wrong while uploading the file. Just run

nix-store --verify-path --repair /nix/store/dhnspmn5hc312lm46a98w4n1bz27bp40-ghc-8.8.3 

to see what I mean. I had to re-build the compiler locally with --option build-use-substitutes false to recover from this issue. Can we fix that archive in the cache somehow?

[vcunat]

@NixOS/infra

[grahamc]

👀

[grahamc]

I was able to fetch it okay:

[grahamc@Petunia:~]$ nix-store -r /nix/store/dhnspmn5hc312lm46a98w4n1bz27bp40-ghc-8.8.3
these paths will be fetched (134.19 MiB download, 1863.96 MiB unpacked):
  /nix/store/dhnspmn5hc312lm46a98w4n1bz27bp40-ghc-8.8.3
  /nix/store/jvkljrfxk8svpak297a0raqyrm10xq3h-ghc-8.8.3-doc
copying path '/nix/store/jvkljrfxk8svpak297a0raqyrm10xq3h-ghc-8.8.3-doc' from 'https://cache.nixos.org'...
copying path '/nix/store/dhnspmn5hc312lm46a98w4n1bz27bp40-ghc-8.8.3' from 'https://cache.nixos.org'...
warning: you did not specify '--add-root'; the result might be removed by the garbage collector
/nix/store/dhnspmn5hc312lm46a98w4n1bz27bp40-ghc-8.8.3

but the two arguments you provide don't seem to work for me?

[grahamc@Petunia:~]$ nix-store --verify-path --repair /nix/store/dhnspmn5hc312lm46a98w4n1bz27bp40-ghc-8.8.3
error: no flags expected
Try 'nix-store --help' for more information.

when I verify-path on its own, it seems okay:

[grahamc@Petunia:~]$ nix-store --verify-path  /nix/store/dhnspmn5hc312lm46a98w4n1bz27bp40-ghc-8.8.3  ; echo $?
0

Can you post the hash you get with:

[grahamc@Petunia:~]$ nix hash-path /nix/store/dhnspmn5hc312lm46a98w4n1bz27bp40-ghc-8.8.3
sha256-X1cmrtxXmar4P1L+Om7edBeo3I8lEx9dyX2Ti4Z6LQY=

[grahamc]

I've asked people in #nixos-dev to check this too with this script:

#!/bin/sh

set -x

nix-store -r /nix/store/dhnspmn5hc312lm46a98w4n1bz27bp40-ghc-8.8.3; echo $?

nix-store --verify-path  /nix/store/dhnspmn5hc312lm46a98w4n1bz27bp40-ghc-8.8.3; echo $?

nix hash-path /nix/store/dhnspmn5hc312lm46a98w4n1bz27bp40-ghc-8.8.3; echo $?

and have gotten sha256-X1cmrtxXmar4P1L+Om7edBeo3I8lEx9dyX2Ti4Z6LQY= from:

Germany (3x)
CZ
North Carolina, USA
London, UK
Sweden
France (2x)
Portland, USA
Germany / Bavaria / Landkreis Ebersberg Gemeinde Poing (near Markt Schwaben)
Paris (2x)
Frankfurt
London
OVH, Strasbourg
GCP, us-west-2
Oregon, US

[peti]

Oh, I mixed up my command line syntax. It is

nix-store --verify-path ...

and

nix-store --repair-path ...

..., of course.

Unfortunately, I don't have the broken store path any longer, because I re-built it locally. I didn't have the presence of mind to keep a copy of the broken one.

Anyhow, I am surprised that no-one else can observe the problem with --verify-path. Maybe my store got corrupted locally somehow? I just assumed that wouldn't be possible because it's mounted read-only.

[vcunat]

I had the path from a couple days ago and redownloaded it now, both the same hash as Graham posted. (By the way, nix-store -r won't force a redownload, contrary to --repair-path.)

[grahamc]

I think we have to assume based on what we're seeing this was local corruption somehow, is that wrong? I'm thinking an error in the CDN part would be caught by a signature check. Given the stability of the hashes across a wide number of locations, I'm thinking that leaves us with only local corruption as an option. Any other opinions?

[peti]

OK, I too believe the error must have been here. I just don't understand how a repeated re-download of the store path ended up with a verification error every time I ran it. Shouldn't my local error have disappeared the first time I ran --repair-path?

[edolstra]

It's possible that ~/.cache/nix/binary-cache-*.sqlite contained a wrong/corrupted hash for some reason. Maybe it got confused by the hash for the same store path from a different binary cache, or something.