warning: unable to download: HTTP error 200 (curl error: Timeout was reached);

[feijoas]

Issue description

Since a few days I am getting a curl timeout when installing the dependencies mono and a few others:

nix-shell -p mono
these paths will be fetched (195.30 MiB download, 771.40 MiB unpacked):
  /nix/store/0qqk6rlh0nrxvwhnxjqij0f71hwg0p3h-mesa-19.1.1-dev
  /nix/store/1bsdnvj90a6akngbi0lqqqdlv3h1pkqh-libgdiplus-5.6.1
  /nix/store/2acc44zvbm4j9dbw2l302sqp8mramda1-mono-5.20.1.27
  /nix/store/dadshmwrrxqsirphbry7n456hhxd8ras-ncurses-6.1-20190112-dev
  /nix/store/jkvs9bvqpg5gfb7xg48l4adx6ylvbnnv-mesa-19.1.1-drivers
  /nix/store/s4gx744wnfripgai2vn463lbqzbrbjwa-libGL-1.0.0-dev
  /nix/store/y0yh686dqiwq4m32vfc495ikr64ja48y-cairo-1.16.0-dev
copying path '/nix/store/dadshmwrrxqsirphbry7n456hhxd8ras-ncurses-6.1-20190112-dev' from 'https://cache.nix
os.org'...
copying path '/nix/store/jkvs9bvqpg5gfb7xg48l4adx6ylvbnnv-mesa-19.1.1-drivers' from 'https://cache.nixos.or
g'...
ncopying path '/nix/store/0qqk6rlh0nrxvwhnxjqij0f71hwg0p3h-mesa-19.1.1-dev' from 'https://cache.nixos.org'.
..
copying path '/nix/store/s4gx744wnfripgai2vn463lbqzbrbjwa-libGL-1.0.0-dev' from 'https://cache.nixos.org'..
.
copying path '/nix/store/y0yh686dqiwq4m32vfc495ikr64ja48y-cairo-1.16.0-dev' from 'https://cache.nixos.org'.
..
copying path '/nix/store/1bsdnvj90a6akngbi0lqqqdlv3h1pkqh-libgdiplus-5.6.1' from 'https://cache.nixos.org'.
..
copying path '/nix/store/2acc44zvbm4j9dbw2l302sqp8mramda1-mono-5.20.1.27' from 'https://cache.nixos.org'...
warning: unable to download 'https://cache.nixos.org/nar/0n3l6bnhv6r2r452szsv6xgqrmfyrf87yj0ma3a79bb37mf49cj6.nar.xz': HTTP error 200 (curl error: Timeout was reached); retrying in 308 ms
error 9 while decompressing xz file
error: build of '/nix/store/rnnlz1rgvqaq5bavcaix6gphf1dkxz4h-mono-5.20.1.27.drv' failed

I tried switching the cache from 151.101.194.217 to 151.101.66.217, but without any effect.

Steps to reproduce

nix-shell -p mono

Technical details

 - system: `"x86_64-linux"`
 - host os: `Linux 4.19.57, NixOS, 19.09pre184803.d567c486ca5 (Loris)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.2.2`
 - channels(user): `"home-manager"`
 - channels(root): `"nixos-19.09pre185176.1036dc66416, nixpkgs-19.09pre185562.00ef72610c8"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

sudo nix-channel --list
nixos https://nixos.org/channels/nixos-unstable
nixpkgs https://nixos.org/channels/nixpkgs-unstable

Here the output of the cloudfront.sh script from https://cache.nixos.org:
http://ix.io/1OOI

[davidak]

I can not reproduce it from Germany/Europe. Maybe a temporary problem of the CDN.

[davidak@ethmoid:~]$ nix-shell -p mono -I nixpkgs=https://releases.nixos.org/nixos/unstable/nixos-19.09pre185259.362be9608c3/nixexprs.tar.xz
these paths will be fetched (132.17 MiB download, 503.11 MiB unpacked):
  /nix/store/1bsdnvj90a6akngbi0lqqqdlv3h1pkqh-libgdiplus-5.6.1
  /nix/store/2acc44zvbm4j9dbw2l302sqp8mramda1-mono-5.20.1.27
  /nix/store/dadshmwrrxqsirphbry7n456hhxd8ras-ncurses-6.1-20190112-dev
  /nix/store/qqav3b3hibzs8gr7shbcgjd310wk9xv8-ncurses-6.1-20190112-man
  /nix/store/z0liwi5pish02m3j5vasi3imsqnixs2h-libexif-0.6.21
copying path '/nix/store/qqav3b3hibzs8gr7shbcgjd310wk9xv8-ncurses-6.1-20190112-man' from 'https://cache.nixos.org'...
copying path '/nix/store/z0liwi5pish02m3j5vasi3imsqnixs2h-libexif-0.6.21' from 'https://cache.nixos.org'...
copying path '/nix/store/1bsdnvj90a6akngbi0lqqqdlv3h1pkqh-libgdiplus-5.6.1' from 'https://cache.nixos.org'...
copying path '/nix/store/dadshmwrrxqsirphbry7n456hhxd8ras-ncurses-6.1-20190112-dev' from 'https://cache.nixos.org'...
copying path '/nix/store/2acc44zvbm4j9dbw2l302sqp8mramda1-mono-5.20.1.27' from 'https://cache.nixos.org'...

[nix-shell:~]$ 

[duhirsch]

Europe/Germany
NixOS 19.03

Running "nixos-rebuild switch" gives me the same error.
Sometimes it's also 'error 9 while decompressing xz file', "error: unexpected end-of-file", or hash mismatches.
The "error 9 ..." seems to have been fixed by
NixOS/nix@53247d6
Unfortunately there has not been a new release of nix which includes this commit.

I was able to reproduce this the last few days.
At the end of this discussion several people are having similar problems
https://discourse.nixos.org/t/the-nixos-cache-is-now-hosted-by-fastly/1061

At the moment the cache seems to be rather slow, sometimes it manages to use my full bandwidth but often it drops to ~500KB/s

[teto]

I've got lots of errors from Japan. Had to rerun ~7 times my nix-shell before it was able to fetch everything. On discourse, someone says to use -j1 : The HTTP error 200 (curl error: Timeout was reached); problem also occurs for me, though limiting to one job with -j1 fixes it.
I will try next time. The errors are kinda cryptic (error "9") so it's kinda depressing.

[lsix]

I have had this for a while in France. This seemed to be linked to a problem in my not fully ipv6 capable router. The simplest workaround I had was to only rely on ipv4 when needed.

Something like

$ sudo ip -6 route delete default

makes sure that you do not use IPv6 (non permanent), which is quite sad I have to admit.

[angristan]

I have had this problem for quite a while too (France)

[aexoxea]

I'm also having this issue with one of my computers, starting a bit over a month ago. I have two computers running NixOS at the same location (and sharing the same slow connection), one on channel nixos-unstable and one on channel nixos-19.03 (both were originally installed as 19.03).

The unstable computer has the issue, the 19.03 computer doesn't.

Further, the unstable computer only has the issue if it's using Nix from the unstable channel; after switching it to using Nix from the 19.03 channel as a workaround (i.e. setting nix.package = pkgs.stable.nix, using a nixpkgs.config.packageOverrides to define stable), this issue went away.

The issue doesn't happen if I download the failed files directly (e.g. curl -O https://cache.nixos.org/nar/<hash>.nar.xz). It also doesn't happen where the derivation fetches the files directly (e.g. NVIDIA drivers), only where Nix is using curl to get cached derivations from cache.nixos.org.

From observations, it appears that:

• The timeout is always 300 seconds. I haven't set a connect-timeout for Nix, and to my understanding (from the Nix and Nixpkgs manuals) there aren't any timeouts set by default. I do note that Nix has a hardcoded 300 second low-speed timeout (see libstore/download.cc#L247 and libstore/download.cc#L305-L306), but that might be coincidental.

• The timeout seems to be spurious. I've watched several cases where everything seems normal: no problem with the connection, Nix is happily extracting files under \nix\store... but the download suddenly 'times out', seemingly for no other reason than because 300 seconds have elapsed.

• Setting nix.maxJobs = 1 (the default) helps avoid other timeouts on my slow connection, but doesn't seem to help with these specific timeouts (except insofar as it lessens the number of them).

As @duhirsch noted, the next Nix release will support resumable NAR downloads (which is excellent 👍) in response to Nix#2952 (in which the same timeout messages can be seen). However, I feel that still leaves the question of why these specific timeouts happen to begin with.

Hope this helps 😃.

(p.s. I'm in Australia, as are the computers involved.)

(p.p.s. I did try disabling IPv6 as an alternative – networking.enableIPv6 = false;, removing it from the connection in NetworkManager and monitoring to confirm the cache download was happening over IPv4 – but that did not work for me.)

[feijoas]

@aexoxea thanks for the detailed report. Could you please post the relevant part of your configuration on howto use nix from stable?

[aexoxea]

Hi @feijoas,

You're welcome. The relevant part of my config is:

nix.package = pkgs.stable.nixStable;

nixpkgs.config.packageOverrides = pkgs: {
    stable = import <nixos-stable> {
        config = config.nixpkgs.config;
    };
};

This is based on an approach documented in NixOS Wiki:
https://nixos.wiki/wiki/Cheatsheet#Upgrading_individual_packages_to_a_different_channel

NB 1: This first requires that the 19.03 channel be added as "nixos-stable" and the channels refreshed, otherwise it won't evaluate.

NB 2: While this is working for me, I lack the knowledge to say whether this might cause other issues down the line, since I don't know to what extent Nix can be considered "like just another package" on a NixOS system.

[duhirsch]

@lsix's solution, which is disabling ipv6 works for me.
Instead of disabling it on the default interface one could also disable it on all interfaces via
echo 1 > /proc/sys/net/ipv6/conf/all/disable_ipv6
(must be executed under root permissions)

Nevertheless I consider this to be only a temporary fix, the real problem has to be somewhere else.

[vcunat]

I think the older thread did seem to indicate that IPv6 makes Fastly's issues more likely (or worse).

Killing all IPv6 seems an overkill... it's just a single name, so it should be perfectly enough e.g. to override one DNS record

cache.nixos.org.    CNAME    v2.shared.global.fastly.net.

EDIT: just for you, as people without these problems probably want to keep IPv6.

[feijoas]

Both approaches, disabling IPv6 and using nix from stable channel, do not work.

[feijoas]

I close this issue since it seems to be a nix error: NixOS/nix#3025

[zimbatm]

If you are still having issues, there is a new experimental cache with some improvements: https://discourse.nixos.org/t/improvements-to-cache-nixos-org-help-test-the-new-config/3620

[hwertz]

I have seen an error just like this; if I ran (I might have this backwards...) a "nix-shell" that was downloading larger NAR.xz items (copying ... into /nix/store/....) it'd show that xz exited with error 10; if I installed something with "nix-env -i" I got a HTTP error 200 (curl error: Timeout was reached).

Long story short, in ~/.config/nix/nix.conf I put "stalled-download-timeout = 99999999" (just an arbitrarily large number). I guess you can put this into /etc/nix/nix.conf if you prefer. PROBLEM FIXED!!!

Looks like default is 300, and it does sound fine to have a download abort if it's stalled for 5 minutes. But for whatever reason it's not... Anyway... I found the code and in essence curl "-Y" (--speed-limit) is set to "1" (byte per second), "-y" (--speed-time) is set to 300 (or whatever stalled-download-timeout is set to). Sounds fair enough. BUT, "nar.xz" are not downloaded then installed, they are downloaded into some kind of pipe, which (when it's extracting a bunch of tiny files) DOES cause the download to stall for a while. I don't think it's stalling for 5 minutes... but whatever, disabling this timeout fixes it. Cheers!

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/what-to-do-with-these-errors-or-warnings-when-upgrading-my-nixos-channel-from-22-05-to-23-05/34604/3