New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Users are reporting some issues with cache.nixos.org - gather information here #57

Open
rbvermaa opened this Issue Oct 15, 2018 · 22 comments

Comments

Projects
None yet
9 participants
@rbvermaa
Copy link
Member

rbvermaa commented Oct 15, 2018

Issue to centralize some of the issues some users are having with cache.nixos.org after the Fastly.com switch.

@matthewbauer

This comment has been minimized.

Copy link
Member

matthewbauer commented Oct 15, 2018

I'm getting it every time I try to use nix-index:

$ nix-index
+ querying available packages

error: fetching the references of store path '/nix/store/f03qiw1qz47qx685f2wfb76crxsp0ymg-node-grunt-cli-1.3.1' failed
caused by: request GET 'http://cache.nixos.org/f03qiw1qz47qx685f2wfb76crxsp0ymg.narinfo' failed with HTTP error 403 Forbidden

It looks like it's a different url failed each time.

@rbvermaa

This comment has been minimized.

Copy link
Member Author

rbvermaa commented Oct 16, 2018

@matthewbauer The 403 is expected, as that is what S3 returns when an object does not exist. However, it is weird that nix would fail on that. @edolstra any idea how that could happen?

Could you add some information about the nix version you are using when you get this error?

@rbvermaa

This comment has been minimized.

Copy link
Member Author

rbvermaa commented Oct 16, 2018

It might be that the error doesn't come from nix, but from nix-index, as well.

@rbvermaa

This comment has been minimized.

Copy link
Member Author

rbvermaa commented Oct 16, 2018

Looked at the nix-index code, and indeed error seems to come from nix-index, specifically https://github.com/bennofs/nix-index/blob/master/src/hydra.rs#L173 .

This is probably triggered by cache.nixos.org returning the 403 error that S3 returns on objects that do not exist. The Cloudfront setup returned a 404. Will see if we can change this to a 404.

@rbvermaa

This comment has been minimized.

Copy link
Member Author

rbvermaa commented Oct 16, 2018

@matthewbauer We have changed cache.nixos.org to return a 404. There might be some URL's that still have the 403 cached, but these should disappear in the next few hours, and return the 404. This should make nix-index work again.

@NinjaTrappeur

This comment has been minimized.

Copy link

NinjaTrappeur commented Oct 16, 2018

Hi! I'm the one reported the issue on the nixos discourse. I'm gonna answer @rbvermaa questions here.

Can you give some information about the host system, the nix version used and how you installed nix?

The host system is a nixos 18.09 with nix 2.1.1.

Also, how easy is it for you to reproduce?

Really easy, it fails ~80% of the times.

Once it downloaded one file, I have no problem downloading the following ones in the same CLI session (ie nix-channel, nix-build or nixos-rebuild call).

For instance, if I try to update my channels, I get

~ » nix --version                                                                         ninjatrappeur@thinkpad-nix
nix (Nix) 2.1.1
------------------------------------------------------------
~ » sudo nix-channel --update                                                             ninjatrappeur@thinkpad-nix
[sudo] Mot de passe de ninjatrappeur : 
unpacking channels...
warning: unable to download 'https://cache.nixos.org/hig48ji0b68r4d47gd83jpnmpi3hrfxi.narinfo': SSL connect error (35); retrying in 266 ms
warning: unable to download 'https://cache.nixos.org/hig48ji0b68r4d47gd83jpnmpi3hrfxi.narinfo': SSL connect error (35); retrying in 582 ms
warning: unable to download 'https://cache.nixos.org/hig48ji0b68r4d47gd83jpnmpi3hrfxi.narinfo': SSL connect error (35); retrying in 1189 ms
warning: unable to download 'https://cache.nixos.org/hig48ji0b68r4d47gd83jpnmpi3hrfxi.narinfo': SSL connect error (35); retrying in 2297 ms
warning: unable to download 'https://cache.nixos.org/hig48ji0b68r4d47gd83jpnmpi3hrfxi.narinfo': SSL connect error (35); retrying in 4008 ms
^Cerror: interrupted by the user
------------------------------------------------------------
~ » sudo nix-channel --update                                                             ninjatrappeur@thinkpad-nix
unpacking channels...
created 2 symlinks in user environment
----------------------------------------

(I interrupt the first call to reset the exponential retry delay).

My internet provider is OVH telecom if it's any help.

I can privately share with you my IP address if it's any help for the debug process.

@rbvermaa

This comment has been minimized.

Copy link
Member Author

rbvermaa commented Oct 16, 2018

@NinjaTrappeur Thanks for the info. Does a curl call work without issues? e.g. could you post output of:

curl -v https://cache.nixos.org/hig48ji0b68r4d47gd83jpnmpi3hrfxi.narinfo

@rbvermaa

This comment has been minimized.

Copy link
Member Author

rbvermaa commented Oct 16, 2018

Also, are you using IPv6? Just found similar issue here: https://community.fastly.com/t/i-often-cant-access-fastly-servers-using-https-ipv6-rst-packets-received/1317/4 . Perhaps it is related.

@NinjaTrappeur

This comment has been minimized.

Copy link

NinjaTrappeur commented Oct 16, 2018

~ » curl -v https://cache.nixos.org/hig48ji0b68r4d47gd83jpnmpi3hrfxi.narinfo              ninjatrappeur@thinkpad-nix
*   Trying 2a04:4e42:1d::729...
* TCP_NODELAY set
* Connected to cache.nixos.org (2a04:4e42:1d::729) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to cache.nixos.org:443 
* stopped the pause stream!
* Closing connection 0
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to cache.nixos.org:443 
------------------------------------------------------------
~ » curl -v -4 https://cache.nixos.org/hig48ji0b68r4d47gd83jpnmpi3hrfxi.narinfo           ninjatrappeur@thinkpad-nix
*   Trying 151.101.38.217...
* TCP_NODELAY set
* Connected to cache.nixos.org (151.101.38.217) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=Fastly, Inc.; CN=v2.shared.global.fastly.net
*  start date: Oct 15 11:17:24 2018 GMT
*  expire date: Mar 20 20:22:20 2019 GMT
*  subjectAltName: host "cache.nixos.org" matched cert's "cache.nixos.org"
*  issuer: C=BE; O=GlobalSign nv-sa; CN=GlobalSign CloudSSL CA - SHA256 - G3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x13bc2b0)
> GET /hig48ji0b68r4d47gd83jpnmpi3hrfxi.narinfo HTTP/2
> Host: cache.nixos.org
> User-Agent: curl/7.61.0
> Accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
< HTTP/2 404 
< server: Varnish
< retry-after: 0
< content-type: text/html
< accept-ranges: bytes
< accept-ranges: bytes
< date: Tue, 16 Oct 2018 10:11:44 GMT
< via: 1.1 varnish
< x-served-by: cache-ams4431-AMS
< x-cache: MISS
< x-cache-hits: 0
< x-timer: S1539684705.547020,VS0,VE88
< content-length: 3
< 
* Connection #0 to host cache.nixos.org left intact
404%                                               

Looks like an IPv6 error.

@NinjaTrappeur

This comment has been minimized.

Copy link

NinjaTrappeur commented Oct 16, 2018

Yup, looks related, I have the same router.

I'm gonna look for an option to force IPv4 on nix calls.

Thank you for your help!

@hamishmack

This comment has been minimized.

Copy link

hamishmack commented Oct 31, 2018

Any update on this. I'm not sure if this is the same issue, but we are seeing HTTP error 503 followed by error 7 while decompressing xz file here (Wellington, NZ):

copying path '/nix/store/vkfs0i8j9jk7b0y1n49ykraf49w0fqb4-python2.7-pycrypto-3.6.6' from 'https://cache.nixos.org'...
copying path '/nix/store/h54y4zm7pzckn67y1ixdbz6ga8v7gmbj-python2.7-libcloud-1.2.1' from 'https://cache.nixos.org'...
warning: unable to download 'https://cache.nixos.org/nar/1kigq2qc4d7pf9dpfna21p5r2shifkfbpdda0bzpw2p8hav6plfp.nar.xz': HTTP error 503; retrying in 264 ms
warning: unable to download 'https://cache.nixos.org/nar/1kigq2qc4d7pf9dpfna21p5r2shifkfbpdda0bzpw2p8hav6plfp.nar.xz': HTTP error 503; retrying in 593 ms
warning: unable to download 'https://cache.nixos.org/nar/1npnb3jcfqhyw816ncsscjl7wpwh06pygcw8cgv4jiix9q9bcrx4.nar.xz': HTTP error 503; retrying in 292 ms
error 7 while decompressing xz file
warning: unable to download 'https://cache.nixos.org/nar/1npnb3jcfqhyw816ncsscjl7wpwh06pygcw8cgv4jiix9q9bcrx4.nar.xz': HTTP error 503; retrying in 576 ms
error: build of '/nix/store/kczj7517hjs2l5j9kvy3s76lxn89la8l-nixops-1.6.drv' failed

Rerunning the nix command always seems to get a bit further and eventually works.

@rbvermaa

This comment has been minimized.

Copy link
Member Author

rbvermaa commented Nov 19, 2018

@hamishmack Sorry, I missed the notification for this issue. I have contacted Fastly support, to see if they can help diagnose this issue, will update here when I hear back.

@rbvermaa

This comment has been minimized.

Copy link
Member Author

rbvermaa commented Nov 19, 2018

We've changed some settings to be able to debug this issue better, based on suggestions by Fastly. Hopefully this gives us some more information about the 503 errors.

If you experience this again on your machine, can you let us know and go to https://www.fastly-debug.com/ and post the information here?

@terlar

This comment has been minimized.

Copy link

terlar commented Dec 10, 2018

I experience this issue and just disabled IPv6 to be able to upgrade my NixOS, going to that page it just infinitely spinns sayingCollecting data please wait..., I waited for 30 minutes, wasn't sure if it was supposed to return something by then.

@rbvermaa

This comment has been minimized.

Copy link
Member Author

rbvermaa commented Dec 19, 2018

@terlar Does the page spin after you switched to IPv4, or when you were still on IPv6?

@terlar

This comment has been minimized.

Copy link

terlar commented Dec 19, 2018

Both as far as I can remember. I am currently on vacation, but I will double check when I have access to my computer.

@bmillwood

This comment has been minimized.

Copy link

bmillwood commented Dec 21, 2018

I'm also having the 503 and then error 7 while decompressing xz file thing. And again, it seems to work if I rerun it enough times.

From fastly:

Please submit text block below with your ticket to Fastly
ewogICJnZW9pcCI6IHsKICAgICJjaSI6ICJob25nIGtvbmciLAogICAgInN0IjogIk5PIFJFR0lPTiIsCiAgICAiY3QiOiAiaG9uZyBrb25nIiwKICAgICJjbyI6ICJBUyIsCiAgICAiY19hc24iOiAiNDc2MCIsCiAgICAiY19hc25fbmFtZSI6ICJwY2N3IGxpbWl0ZWQiLAogICAgInJfaXAiOiAiMjE4LjEwMi4xMS4xMDYiLAogICAgInJfYXNuIjogIjQ3NjAiLAogICAgInJfYXNuX25hbWUiOiAicGNjdyBsaW1pdGVkIiwKICAgICJyX2NpIjogImhvbmcga29uZyIsCiAgICAicl9zdCI6ICJOTyBSRUdJT04iLAogICAgInJfY3QiOiAiaG9uZyBrb25nIiwKICAgICJyX2NvIjogIkFTIgogIH0sCiAgInBvcExhdGVuY3kiOiB7CiAgICAibnJ0IjogNDksCiAgICAiaXRtIjogNjAsCiAgICAidHlvIjogNTQsCiAgICAiaG5kIjogNTUsCiAgICAiaGtnIjogMiwKICAgICJzaW4iOiAzOCwKICAgICJmanIiOiAxOTYsCiAgICAibGF4IjogMjA1LAogICAgImZyYSI6IDE5NwogIH0sCiAgInBvcEFzc2lnbm1lbnRzIjogewogICAgImFjIjogImhrZyIsCiAgICAiYXMiOiAiaGtnIgogIH0sCiAgInJlcXVlc3QiOiB7CiAgICAicmVzb2x2ZXJfaXAiOiAiMjE4LjEwMi4xMS45NyIsCiAgICAicmVzb2x2ZXJfYXNfbmFtZSI6ICJIS1RJTVMtQVAgSEtUIExpbWl0ZWQsIEhLIiwKICAgICJyZXNvbHZlcl9hc19udW1iZXIiOiAiNDc2MCIsCiAgICAicmVzb2x2ZXJfY291bnRyeV9jb2RlIjogIkhLIiwKICAgICJjbGllbnRfaXAiOiAiMjE5Ljc5LjEzMC4xMzUiLAogICAgImNsaWVudF9hc19uYW1lIjogIkhLVElNUy1BUCBIS1QgTGltaXRlZCwgSEsiLAogICAgImNsaWVudF9hc19udW1iZXIiOiAiNDc2MCIsCiAgICAidGltZSI6ICIyMDE4LTEyLTIxVDE3OjE4OjIwLjAwMFoiLAogICAgImhvc3QiOiAid3d3LmZhc3RseS1kZWJ1Zy5jb20iLAogICAgImFjY2VwdCI6ICJ0ZXh0L2h0bWwsYXBwbGljYXRpb24veGh0bWwreG1sLGFwcGxpY2F0aW9uL3htbDtxPTAuOSwqLyo7cT0wLjgiLAogICAgInVzZXJhZ2VudCI6ICJNb3ppbGxhLzUuMCAoWDExOyBMaW51eCB4ODZfNjQ7IHJ2OjYzLjApIEdlY2tvLzIwMTAwMTAxIEZpcmVmb3gvNjMuMCIsCiAgICAiYWNjZXB0bGFuZ3VhZ2UiOiAiZW4tVVMsZW47cT0wLjUiLAogICAgImFjY2VwdGVuY29kaW5nIjogImd6aXAiLAogICAgImZhc3RseXNlcnZlcmlwIjogIjE1MS4xMDEuNzYuNjQiLAogICAgInhmZiI6ICIyMTkuNzkuMTMwLjEzNSIsCiAgICAiZGF0YWNlbnRlciI6ICJIS0ciLAogICAgImJhbmR3aWR0aF9tYnBzIjogIjY5Ljc1IiwKICAgICJjd25kIjogMTAwLAogICAgIm5leHRob3AiOiAiMTcyLjIwLjEwMC4xIiwKICAgICJydHQiOiAxMS42MzcsCiAgICAiZGVsdGFfcmV0cmFucyI6IDAsCiAgICAidG90YWxfcmV0cmFucyI6IDAKICB9Cn0=

Client IP Info
IP	219.79.130.135
AS Name	HKTIMS-AP HKT Limited, HK
AS Number	4760
City	hong kong
Continent	AS
Country	hong kong
State	NO REGION
Resolver IP Info
IP	218.102.11.97
AS Name	HKTIMS-AP HKT Limited, HK
AS Number	4760
Country Code	HK
Server Connection Info
IP	151.101.76.64
Datacenter	HKG
BW to server	69.75mbps
Congestion Window	100
Next Hop	172.20.100.1
RTT	11.637ms
Delta Retransmits	0
Total Retransmits	0
POP Latency (ms)
NRT	49
ITM	60
TYO	54
HND	55
HKG	2
SIN	38
FJR	196
LAX	205
FRA	197
Request Info
Time	Sat Dec 22 2018 01:18:20 GMT+0800 (HKT)
Host	www.fastly-debug.com
Accept	text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
User-Agent	Mozilla/5.0 (X11; Linux x86_64; rv:63.0) Gecko/20100101 Firefox/63.0
Accept-Language	en-US,en;q=0.5
Accept-Encoding	gzip
X-Forwarded-For	219.79.130.135
@splack

This comment has been minimized.

Copy link

splack commented Dec 28, 2018

I'm getting 499, 503, 504 and just hung connections with no response. Is there a way to see the upstream status documented somewhere?

@samueldr

This comment has been minimized.

@millettjon

This comment has been minimized.

Copy link

millettjon commented Feb 20, 2019

I started getting lots of these a few hours ago:
warning: unable to download 'https://cache.nixos.org/ax3igj2aglvv46vkcpmyklr6lcqlwz7z.narinfo': Couldn't connect to server (7); retrying in 253 ms
warning: unable to download 'https://cache.nixos.org/im74kvbg0swj3akq4gcbwnlw8pj6lz1a.narinfo': Couldn't connect to server (7); retrying in 267 ms
w
Using wget returns 404 for the same urls.

Here is the fastly report. Note that is doesn't seem to complete. I waited more than 5 minutes and it is still working. The partial information is below.
| Debug
Collecting data please wait.

Client IP Info
IP 181.226.182.157
AS Name
AS Number
City
Continent
Country
State
Resolver IP Info
IP
AS Name
AS Number
Country Code
Server Connection Info
IP 151.101.0.64
Datacenter SCL
BW to server
Congestion Window
Next Hop
RTT
Delta Retransmits
Total Retransmits
Request Info
Time Wed Feb 20 2019 15:42:24 GMT-0500 (EST)
Host www.fastly-debug.com
Accept text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,/;q=0.8
User-Agent Mozilla/5.0 (X11; Linux x86_64; rv:65.0) Gecko/20100101 Firefox/65.0
Accept-Language en-US,en;q=0.5
Accept-Encoding gzip
X-Forwarded-For 181.226.182.157

@millettjon

This comment has been minimized.

Copy link

millettjon commented Feb 21, 2019

Still having this issue. Curl now works with ipv4 (-4) option but not with (-6). Any way to force nix to use ipv4?

@terlar

This comment has been minimized.

Copy link

terlar commented Feb 21, 2019

You can disable ipv6. I am currently using networking.enableIPv6 = false; to avoid this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment