504 was not retried while fetching nar

[roberth]

Current behavior:

[...]
unable to download 'https://[redacted].cachix.org/nar/[redacted].nar.xz': HTTP error 504
[...]

Expected behavior:

• perform retries
• log the retries as warnings as usual

nix show-config

version: nix (Nix) 2.3

nix show-config 
allow-import-from-derivation = true
allow-new-privileges = false
allow-unsafe-native-code-during-evaluation = false
allowed-impure-host-deps = 
allowed-uris = 
allowed-users = *
auto-optimise-store = false
build-hook = /nix/store/...-nix-2.3/libexec/nix/build-remote
build-poll-interval = 5
build-users-group = nixbld
builders = 
builders-use-substitutes = false
compress-build-log = true
connect-timeout = 0
cores = 0
diff-hook = 
download-attempts = 5
enforce-determinism = true
extra-platforms = i686-linux
extra-sandbox-paths = 
extra-substituters = 
fallback = false
filter-syscalls = true
fsync-metadata = true
gc-check-reachability = false
gc-reserved-space = 8388608
hashed-mirrors = http://tarballs.nixos.org/
http-connections = 25
http2 = true
impersonate-linux-26 = false
keep-build-log = true
keep-derivations = true
keep-env-derivations = false
keep-failed = false
keep-going = false
keep-outputs = false
log-lines = 10
max-build-log-size = 0
max-free = 18446744073709551615
max-jobs = 4
max-silent-time = 0
min-free = 0
min-free-check-interval = 5
narinfo-cache-negative-ttl = 3600
narinfo-cache-positive-ttl = 2592000
netrc-file = /etc/nix/netrc
plugin-files = 
post-build-hook = 
pre-build-hook = 
print-missing = true
pure-eval = false
repeat = 0
require-sigs = true
restrict-eval = false
run-diff-hook = false
sandbox = true
sandbox-build-dir = /build
sandbox-dev-shm-size = 50%
sandbox-fallback = false
sandbox-paths = /bin/sh=/nix/store...-busybox-1.30.1/bin/busybox
secret-key-files = 
show-trace = false
stalled-download-timeout = 300
store = auto
substitute = true
substituters = https://cache.nixos.org/
sync-before-registering = false
system = x86_64-linux
system-features = benchmark big-parallel kvm nixos-test
tarball-ttl = 3600
timeout = 0
trace-function-calls = false
trusted-public-keys = cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
trusted-substituters = 
trusted-users = hercules-ci-agent
use-case-hack = false
use-sqlite-wal = true
user-agent-suffix = 

[edolstra]

Strange, looking at download.cc all 5xx errors except 501, 505 and 511 should be treated as retriable.

[domenkozar]

Looking at the code, it seems like this happens with Nix already consumes some of the body.

[domenkozar]

I think this is a bug since 53247d6, Nix will retry only if there is accept-ranges header in the response, but that might not be the case on server errors. In that case Nix should fallback and retry downloading the whole body again (as was the case before this commit).

nix/src/libstore/download.cc

Lines 420 to 422 in f800d45

	&& (!this->request.dataCallback
	|| writtenToSink == 0
	|| (acceptRanges && encoding.empty())))

[domenkozar]

@edolstra how feasible would it be to wait for status to be 2XX in writeCallback before body is consumed? It should really be set then as headers would be parsed before the body.

I think that would solve this and a bunch of other errors: I heard today from @michaelpj he has still seen error 7 while decompressing xz file which could be coming from feeding 500 errors to decompressor.

And I'd really like to backport this to 2.3.x since it's giving headaches to Nix users.

[hvdijk]

This fix broke the downloading of files over FTP with nix-prefetch-url, which for obvious reasons do not result one of the approved HTTP response codes.