Binary cache substituter does not handle server errors gracefully

[edolstra]

E.g. if a NAR has disappeared from the server (as often happens with Hydra), it says:

*** Downloading ‘http://hydra.nixos.org/nar/0spf0rb3hp2nxx75nigzgxz1m9f5524b-nixos-manpages’ to ‘/nix/store/0spf0rb3hp2nxx75nigzgxz1m9f5524b-nixos-manpages’...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (22) The requested URL returned error: 410 Gone

bzip2: Compressed file ends unexpectedly;
        perhaps it is corrupted?  *Possible* reason follows.
bzip2: Inappropriate ioctl for device
        Input file = (stdin), output file = (stdout)

It is possible that the compressed file(s) have become corrupted.
You can use the -tvv option to test integrity of such files.

You can use the `bzip2recover' program to attempt to recover
data from undamaged sections of corrupted files.

error: unexpected end-of-file

The bzip2 error is because we're piping curl directly into bzip2.

The error message should be improved. Also, for a 410 Gone error, we should just proceed as if we never got a .narinfo for this path from the server.

[Profpatsch]

Still happens for 410 Gone errors.

[mbbx6spp]

Temporary workaround suggestion for those having issues after doing --upgrade recently:

nixos-rebuild switch --fallback --upgrade

[rimmington]

Ran into this today with an internal Hydra instance. I couldn't figure out how to get around it, so I just temporarily disabled the cache.

[domenkozar]

Interestingly --fallback doesn't work for me. --option build-fallback true did the trick.

[wizeman]

@domenkozar: I ran into the same issue, see #807.
Are you using import-from-derivation by any chance? It only happened to me for those type of derivations.

[domenkozar]

Yes, it was importing nixpkgs from nix store

[edolstra]

This works correctly in 2.0.

[copumpkin]

It sure does!

[deepfire]

@edolstra, would it be considered a bug in Nix 2.0 if a 410 Gone leads to an instantiation failure?

[edolstra]

Instantiation or build failure? I guess it could only happen at instantiation time if you're importing from a derivation.

At build time, it would be a bug if the build failed on a 410 without --fallback. (Probably we should always fall back on a 404 or 410, but that's not the current behaviour IIRC.)

[deepfire]

@edolstra, we have managed to reproduce the build failure -- the relevant details captured in https://gist.github.com/cleverca22/2038f9abd1f9778e66b63fdf02d67054

Michael clarified that Nix is /nix/store/r2mk4a50zddpi3f7h7m3pdyngp2saasr-nix-2.0/bin/nix, fetched from http://nixos.org/releases/nix/nix-2.0/nix-2.0.tar.xz, hashes as (7024d327314bf92c1d3e6cccd944929828a44b24093954036bfb0115a92f5a14, sha256, outputHashMode=flat)

[domenkozar]

Yeah temporary workaround would be to patch Hydra to return 404 instead of 410.

[deepfire]

Replacing 410 with 404 (as per https://github.com/input-output-hk/iohk-ops/pull/348/files) didn't help:

these paths will be fetched (0.02 MiB download, 0.02 MiB unpacked):
  /nix/store/adm406ixpvkcg3xvc2hz7vnqjzmgrr73-update-notifier-2.3.0.tgz
  /nix/store/cnlv1rncz2qv5qnl8xkwha93ydbn0c0y-ws-2.3.1.tgz
copying path '/nix/store/adm406ixpvkcg3xvc2hz7vnqjzmgrr73-update-notifier-2.3.0.tgz' from 'https://hydra.iohk.io'...
copying path '/nix/store/cnlv1rncz2qv5qnl8xkwha93ydbn0c0y-ws-2.3.1.tgz' from 'https://cache.nixos.org'...
building '/nix/store/xi54xwnqkyrlibc1mgbin7v01y7wx78c-animate.css-3.5.2.tgz.drv'...
building '/nix/store/9v16cc6qgmg06a609s5mcglv5qcnxivj-array-equal-1.0.0.tgz.drv'...
file 'nar/adm406ixpvkcg3xvc2hz7vnqjzmgrr73-update-notifier-2.3.0.tgz' missing from binary cache

[domenkozar]

Ah that's a missing nar, not narinfo. I think hydra should return 404 for narinfo if nar is missing.

(although Nix should handle such case gracefully)

[edolstra]

I added automatic fallback for 404/410 errors: 4ac4f67, 691b758. (That is, you don't have to specify --fallback anymore.)

[deepfire]

@edolstra, do you find it appropriate to have a corresponding nix revision bump in Nixpkgs?