hetzner deploy fails with curl error 60

[manveru]

Let me know if you need any other info... this seems to be missing a certificate? It's my first attempt to deploy to hetzner.

building all machine configurations...
warning: dumping very large path (> 256 MiB); this may run out of memory
beta> copying closure...
beta> download-using-manifests.pl: perl: warning: Setting locale failed.
beta> download-using-manifests.pl: perl: warning: Please check that your locale settings:
beta> download-using-manifests.pl:      LANGUAGE = (unset),
beta> download-using-manifests.pl:      LC_ALL = (unset),
beta> download-using-manifests.pl:      LANG = "en_US.UTF-8"
beta> download-using-manifests.pl:     are supported and installed on your system.
beta> download-using-manifests.pl: perl: warning: Falling back to the standard locale ("C").
beta> download-from-binary-cache.pl: perl: warning: Setting locale failed.
beta> download-from-binary-cache.pl: perl: warning: Please check that your locale settings:
beta> download-from-binary-cache.pl:    LANGUAGE = (unset),
beta> download-from-binary-cache.pl:    LC_ALL = (unset),
beta> download-from-binary-cache.pl:    LANG = "en_US.UTF-8"
beta> download-from-binary-cache.pl:     are supported and installed on your system.
beta> download-from-binary-cache.pl: perl: warning: Falling back to the standard locale ("C").
beta> download-from-binary-cache.pl: could not download ‘https://cache.nixos.org/nix-cache-info’ (Curl error 60)

[rbvermaa]

@aszlig Is this perhaps related to the rescue image change you were talking about, a while back?

[aszlig]

@rbvermaa: Hm, nope... this should be completely unrelated.

@manveru: What is it printing after those lines? Because deploying to the target machine while in rescue mode could probably lead to these errors but they shouldn't be hard failures (as in: the closures then should have been copied over from the local machine).

[manveru]

There's nothing afterwards... I'm only on DSL and thought it was stuck after nothing happened after about 5 minutes. I just checked my iftop and it seems to be uploading a ton of stuff, so I'll try waiting around until it finishes this time.

[manveru]

Well, it only took 9 hours, but my deploy finished. Thanks for the hint.

It might be a good idea to give some feedback about this operation and not let users read an error message for hours and expecting them to think everything's alright :)

[fuzzy-id]

I ran into the same problem. I see what I can do to further analyze it.

I am currently stuck at getting access to the machine: nixops ssh asks for a password. Is it possible to query the nixops DB in order to obtain the password of the rescue system? Or even better: Can nixops ssh be fixed to also work on the rescue systems?

[manveru]

I think I fixed the ssh issue by setting services.openssh.permitRootLogin = "yes"

[fuzzy-id]

I interrupted the process when closures are copied and got access to the root server. The following might help:

root@rescue ~ # /nix/store/v5a69m1b823zm1yy8yvhrd2zi2w385b1-curl-7.44.0/bin/curl https://cache.nixos.org/nix-cache-info
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.
root@rescue ~ # /usr/bin/curl https://cache.nixos.org/nix-cache-info
StoreDir: /nix/store
WantMassQuery: 1
Priority: 40

So it all boils down to certificates.

[ip1981]

I saw those messages, but both times deployment was successful with no action from my side.

[NikolaMandic]

here is how to fix this
environment.variables={CURL_CA_BUNDLE = ["${pkgs.cacert}/etc/ssl/certs/ca-bundle.crt"];};

[grahamc]

The solution from @NikolaMandic didn't seem to fix it when starting with a fresh machine. Adding the following to /root/.bashrc on the rescue machine worked:

export CURL_CA_BUNDLE=<path-to-pkgs.cacert>/etc/ssl/certs/ca-bundle.crt

Note: After nixops gets past:

nix-dev...............> bind-mounting files in /etc... resolv.conf...passwd...group...done.
nix-dev...............> detecting hardware... done.
nix-dev...............> waiting for SSH...

you can use nixops ssh machinename to log in.

[NikolaMandic]

@grahamc I might have used old nixos I had this problem this weekend but was to lazy to work trought