Cargo can't download crates: SEC_E_ILLEGAL_MESSAGE

[AlexPublicProfile]

Problem

>cargo build
...
...
warning: spurious network error (1 tries remaining): [35] SSL connect error (schannel: next InitializeSecurityContext failed: SEC_E_ILLEGAL_MESSAGE (0x80090326) - This error usually occurs when a fatal SSL/TLS alert is received (e.g. handshake failed). More detail may be available in the Windows System event log.)
error: failed to download from `https://crates.io/api/v1/crates/argminmax/0.6.1/download`

Caused by:
  [35] SSL connect error (schannel: next InitializeSecurityContext failed: SEC_E_ILLEGAL_MESSAGE (0x80090326) - This error usually occurs when a fatal SSL/TLS alert is received (e.g. handshake failed). More detail may be available in the Windows System event log.)

But

>curl -L -o argminmax.crate https://crates.io/api/v1/crates/argminmax/0.6.1/download
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 55395  100 55395    0     0  84998      0 --:--:-- --:--:-- --:--:-- 84998

Steps

No response

Possible Solution(s)

No response

Notes

No response

Version

>cargo version
cargo 1.70.0 (ec8a8a0ca 2023-04-25)

[weihanglo]

Copied from this #11344 (comment):

…this is a duplicate of #8688 and #7104. Windows, unlike all other platforms, defaults to checking for TLS revocation certificates. We have considered disabling that on Windows, but have not followed through. This might also be resolved if crates.io leaves Heroku and switches to a service that supports OCSP stapling.

Also probably a duplicate of #12273, so I am going to close this. If you find anything interesting or different, please let us know. Thanks for the report anyway!

[AlexPublicProfile]

Yes, this issue is a duplicate of #12273. But they both have nothing to do with #11344 because they have different error messages.

#11344 is about "Unknown error (0x80092013) - The revocation function was unable to check revocation because the revocation server was offline."

This issue (and #12273) is about "SEC_E_ILLEGAL_MESSAGE (0x80090326) - This error usually occurs when a fatal SSL/TLS alert is received (e.g. handshake failed)."

So, you incorrectly closed two original issues due to your inattention.

[weihanglo]

My bad! It was too careless of me. Repoened.

This kind of bug happened a couple years ago (#9788). Was fixed with a libcurl version bump. I will try to look into its changelog a bit to see if we can find something suspicious.

[AlexPublicProfile]

Perhaps the root cause of the error is the incorrect configuration of the crates.io servers, which is why some versions of Windows schannel cannot work with them.

In the first post, I showed that Curl (the official distribution for Windows) normally downloads the required file. But it was with the default backend (openssl). Now I tested the curl with a non-standard backend (schannel) and got the same error:

>set CURL_SSL_BACKEND=schannel
>curl -L -o argminmax.crate https://crates.io/api/v1/crates/argminmax/0.6.1/download
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:06 --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:11 --:--:--     0
curl: (35) schannel: next InitializeSecurityContext failed: SEC_E_ILLEGAL_MESSAGE (0x80090326) - This error usually occurs when a fatal SSL/TLS alert is received (e.g. handshake failed). More detail may be available in the Windows System event log.

So, if you still try to solve the problem from the side of Cargo, then two questions arise:

• Why does Cargo for Windows use Curl with a non-default ssl backend?
• Is it possible to make a configurable ssl backend in Cargo (as in Curl) or even just call an external Curl?

[weihanglo]

I've got some time looking into this, but had a hard time reproducing it. Could you share the full output of cargo -vV so that we can know the environment you were in. My testing environment is like:

cargo 1.70.0 (ec8a8a0ca 2023-04-25)
release: 1.70.0
commit-hash: ec8a8a0cabb0e0cadef58902470f6c7ee7868bdc
commit-date: 2023-04-25
host: x86_64-pc-windows-msvc
libgit2: 1.6.3 (sys:0.17.0 vendored)
libcurl: 8.0.1-DEV (sys:0.4.61+curl-8.0.1 vendored ssl:Schannel)
os: Windows 10.0.20348 (Windows Server 2022 Datacenter) [64-bit]

Also, could you help bisect which version of cargo started suffering from this issue? That would help us narrow down the scope to investigate.

• Why does Cargo for Windows use Curl with a non-default ssl backend?

There were some discussions about it #8154 (comment). A native SSL backend could have access to Windows Credential Store, allowing for enterprise-wide certificates access. But yep, that could be a direction to fixing it.

• Is it possible to make a configurable ssl backend in Cargo (as in Curl) or even just call an external Curl?

I think it is possible. Someone needs to look into how to do that in https://github.com/alexcrichton/curl-rust.

[weihanglo]

BTW, I found theses issue but not sure if it is related to the issue here.

• Windows 11, version 21H2: KB5020387
• Windows Server 2022: KB5020436
• Windows 10, version 20H2; Windows 10, version 21H1; Windows 10, version 22H1; Windows 10 Enterprise LTSC 2021: KB5020435
• Windows 10 Enterprise LTSC 2019; Windows Server 2019: KB5020438
• Windows 10 2016 LTSB; Windows Server 2016: KB5020439
• Windows 10 2015 LTSB; KB5020440

[Shadlock0133]

i'm having the same issue. if someone has a workaround, please post it

[carols10cents]

Perhaps the root cause of the error is the incorrect configuration of the crates.io servers, which is why some versions of Windows schannel cannot work with them.

Do you know when this issue started to happen? That is, did downloading crates with Cargo on windows work at some point and then stop working? That could help us to narrow down configuration changes made to crates.io that could have caused this.

Do you have any thoughts on the kind of incorrect configuration of the servers that could cause this?

[overtube]

Have the same issue, on Archlinux

[overtube]

Probably some guy or group of guys spilled smoothies on the prod server
:)

[overtube]

Or maybe ssl certificate need prolongate

[AlexPublicProfile]

Do you know when this issue started to happen? That is, did downloading crates with Cargo on windows work at some point and then stop working? That could help us to narrow down configuration changes made to crates.io that could have caused this.

Еverything worked fine a couple of months ago.

Of course, the real problem is in the Windows schannel. But this is basic infrastructure software already installed by millions of users. So all sites should adapt to the features of its work. And most sites do this successfully (Curl even with a schannel backend can easily download files from most sites). But there is an error with crates.io...

Do you have any thoughts on the kind of incorrect configuration of the servers that could cause this?

I suspect that it may be in the list of allowed TLS algorithms or something like that.

[AlexPublicProfile]

I've got some time looking into this, but had a hard time reproducing it. Could you share the full output of cargo -vV so that we can know the environment you were in.

cargo 1.70.0 (ec8a8a0ca 2023-04-25)
release: 1.70.0
commit-hash: ec8a8a0cabb0e0cadef58902470f6c7ee7868bdc
commit-date: 2023-04-25
host: x86_64-pc-windows-gnu
libgit2: 1.6.3 (sys:0.17.0 vendored)
libcurl: 8.0.1-DEV (sys:0.4.61+curl-8.0.1 vendored ssl:Schannel)
os: Windows 6.1.7601 (Windows 7 Professional) [64-bit]

Also, could you help bisect which version of cargo started suffering from this issue? That would help us narrow down the scope to investigate.

This error occurred sometimes (perhaps on some servers?) with the previous version of Cargo. That's why I didn't start an issue about this error at the time.

And some more information:

• cargo downloads the file with the index from github without any problems even now.
• rastup update also works without problems. Does it use Curl or Reqwest backend? And on what servers are the files for it?

[carols10cents]

@overtube

Have the same issue, on Archlinux

You get the same error message on Archlinux, including the part that says More detail may be available in the Windows System event log? That would be incredibly surprising!

If your error message is different, and if you can reproduce the problem on Archlinux, could you please copy and paste the exact error message you're seeing please?

[nokotto]

Hi, I'm afraid that I have the same issue :_(

cargo -vV

cargo 1.70.0 (ec8a8a0ca 2023-04-25)
release: 1.70.0
commit-hash: ec8a8a0cabb0e0cadef58902470f6c7ee7868bdc
commit-date: 2023-04-25
host: x86_64-pc-windows-msvc
libgit2: 1.6.3 (sys:0.17.0 vendored)
libcurl: 8.0.1-DEV (sys:0.4.61+curl-8.0.1 vendored ssl:Schannel)
os: Windows 6.1.7601 (Windows 7 Professional) [64-bit]

[35] SSL connect error (schannel: next InitializeSecurityContext failed: SEC_E_ILLEGAL_MESSAGE (0x80090326) - This error usually occurs when a fatal SSL/TLS alert is received (e.g. handshake failed). More detail may be available in the Windows System event log.)

It's a random error, in the sense that if there are just a few libraries for to build, trying again, and again, the build can finish in success, but if there are many libraries involved (dependence of dependence, etc), most of the times will rise the error.

I tried also with previous versions, but happens the same:

rustup install 1.69 && rustup default 1.69
rustup install 1.68.2 && rustup default 1.68.2
rustup install 1.67.1 && rustup default 1.67.1
rustup install 1.65 && rustup default 1.65
rustup install 1.61 && rustup default 1.61

cargo -vV

cargo 1.61.0 (a028ae42f 2022-04-29)
release: 1.61.0
commit-hash: a028ae42fc1376571de836be702e840ca8e060c2
commit-date: 2022-04-29
host: x86_64-pc-windows-msvc
libgit2: 1.4.2 (sys:0.14.2 vendored)
libcurl: 7.80.0-DEV (sys:0.4.51+curl-7.80.0 vendored ssl:Schannel)
os: Windows 6.1.7601 (Windows 7 Professional) [64-bit]

Turned back to cargo 1.70.0, and from command line the trace of a test project:

set CARGO_LOG=trace
cargo build

Output:

... omitted a lot of trace logs ...

DEBUG cargo::core::package] handles remaining: 58
DEBUG cargo::core::package] 13 finished with Err(Error { description: "SSL connect error", code: 35, extra: Some("schannel: next InitializeSecurityContext failed: SEC_E_ILLEGAL_MESSAGE (0x80090326) - This error usually occurs when a fatal SSL/TLS alert is received (e.g. handshake failed). More detail may be available in the Windows System event log.") })

warning: spurious network error (1 tries remaining): [35] SSL connect error (schannel: next InitializeSecurityContext failed: SEC_E_ILLEGAL_MESSAGE (0x80090326)
 - This error usually occurs when a fatal SSL/TLS alert is received (e.g. handshake failed). More detail may be available in the Windows System event log.)
 
DEBUG cargo::core::package] download retry https://crates.io/api/v1/crates/encoding_rs/0.8.32/download for 6500ms
DEBUG cargo::util::network::sleep] ERIC: now=Instant { t: 1722.480046979s } next=Instant { t: 1722.497818906s }
DEBUG cargo::core::package] handles remaining: 58
DEBUG cargo::util::network::sleep] ERIC: now=Instant { t: 1722.513129849s } next=Instant { t: 1722.497818906s }
DEBUG cargo::util::network::sleep] ERIC: now=Instant { t: 1722.513129849s } next=Instant { t: 1722.631571247s }
DEBUG cargo::core::package] handles remaining: 59
DEBUG cargo::util::network::sleep] ERIC: now=Instant { t: 1722.517024415s } next=Instant { t: 1722.631571247s }
DEBUG cargo::core::package] handles remaining: 59
DEBUG cargo::util::network::sleep] ERIC: now=Instant { t: 1722.524171539s } next=Instant { t: 1722.631571247s }
DEBUG cargo::core::package] handles remaining: 59
DEBUG cargo::util::network::sleep] ERIC: now=Instant { t: 1722.534241372s } next=Instant { t: 1722.631571247s }

DEBUG cargo::core::package] handles remaining: 58
DEBUG cargo::core::package] 90 finished with Err(Error { description: "SSL connect error", code: 35, extra: Some("schannel: next InitializeSecurityContext failed: SEC_E_ILLEGAL_MESSAGE (0x80090326) - This error usually occurs when a fatal SSL/TLS alert is received (e.g. handshake failed). More detail may be available in the Windows System event log.") })

warning: spurious network error (1 tries remaining): [35] SSL connect error (schannel: next InitializeSecurityContext failed: SEC_E_ILLEGAL_MESSAGE (0x80090326)
 - This error usually occurs when a fatal SSL/TLS alert is received (e.g. handshake failed). More detail may be available in the Windows System event log.)
DEBUG cargo::core::package] download retry https://crates.io/api/v1/crates/siphasher/0.3.10/download for 6500ms

... omitted a lot of similar error logs ...

Then I downloaded curl for windows, with the same library version used by cargo (libcurl 8.0.1) for to try to get more information, but nothing is shown, just that curl for windows can download the files were cargo rise an error.

https://curl.se/windows/dl-8.0.1_1/curl-8.0.1_1-win64-mingw.zip

curl --version

curl 8.0.1 (x86_64-w64-mingw32) libcurl/8.0.1 OpenSSL/3.0.8 (Schannel) zlib/1.2.13 brotli/1.0.9 zstd/1.5.4 WinIDN libssh2/1.10.0 nghttp2/1.52.0 ngtcp2/0.13.1 nghttp3/0.9.0 libgsasl/2.2.0
Release-Date: 2023-03-20
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp ws wss
Features: alt-svc AsynchDNS brotli gsasl HSTS HTTP2 HTTP3 HTTPS-proxy IDN IPv6 Kerberos Largefile libz MultiSSL NTLM SPNEGO SSL SSPI threadsafe TLS-SRP UnixSock
ets zstd

-L Allow curl to follow any redirections
-v verbose
-o file_output

curl -L -v https://crates.io/api/v1/crates/hyper/0.14.27/download -o test_file_download.crate

Output (Yesterday )

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 108.157.98.90:443...
* Connected to crates.io (108.157.98.90) port 443 (#0)
* ALPN: offers h2,http/1.1
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
*  CAfile: D:\curl-8.0.1_1-win64-mingw\bin\curl-ca-bundle.crt
*  CApath: none
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [19 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [5007 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [264 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [36 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [36 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=crates.io
*  start date: Jan 25 00:00:00 2023 GMT
*  expire date: Feb 23 23:59:59 2024 GMT
*  subjectAltName: host "crates.io" matched cert's "crates.io"
*  issuer: C=US; O=Amazon; CN=Amazon RSA 2048 M02
*  SSL certificate verify ok.
} [5 bytes data]
* using HTTP/2
* h2h3 [:method: GET]
* h2h3 [:path: /api/v1/crates/hyper/0.14.27/download]
* h2h3 [:scheme: https]
* h2h3 [:authority: crates.io]
* h2h3 [user-agent: curl/8.0.1]
* h2h3 [accept: */*]
* Using Stream ID: 1 (easy handle 0x4d7f50)
} [5 bytes data]
> GET /api/v1/crates/hyper/0.14.27/download HTTP/2
> Host: crates.io
> user-agent: curl/8.0.1
> accept: */*
>
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [124 bytes data]
< HTTP/2 302
< content-length: 0
< location: https://static.crates.io/crates/hyper/hyper-0.14.27.crate
< server: nginx
< date: Thu, 06 Jul 2023 22:13:52 GMT
< x-content-type-options: nosniff
< x-frame-options: SAMEORIGIN
< x-xss-protection: 0
< content-security-policy: default-src 'self'; connect-src 'self' *.ingest.sentry.io https://docs.rs https://play.rust-lang.org https://static.crates.io; script
-src 'self' 'unsafe-eval' 'sha256-n1+BB7Ckjcal1Pr7QNBh/dKRTtBQsIytFodRiIosXdE='; style-src 'self' 'unsafe-inline' https://code.cdn.mozilla.net; font-src https:/
/code.cdn.mozilla.net; img-src *; object-src 'none'
< access-control-allow-origin: *
< strict-transport-security: max-age=31536000; includeSubDomains
< via: 1.1 vegur, 1.1 11dbcac5e035be02cf80e51d1841e116.cloudfront.net (CloudFront)
< vary: Accept,Accept-Encoding,Cookie
< x-cache: Miss from cloudfront
< x-amz-cf-pop: MAD56-P1
< x-amz-cf-id: vTlqHBZ8IHkuJR0GXdZOYChxZA_dwDZG_Wt5YJwVAojdta9KOGeR6g==
<
{ [0 bytes data]
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
* Connection #0 to host crates.io left intact
* Issue another request to this URL: 'https://static.crates.io/crates/hyper/hyper-0.14.27.crate'
*   Trying 151.101.1.91:443...
  0     0    0     0    0     0      0      0 --:--:--  0:00:21 --:--:--     0* connect to 151.101.1.91 port 443 failed: Timed out
*   Trying 151.101.65.91:443...
  0     0    0     0    0     0      0      0 --:--:--  0:00:22 --:--:--     0* Connected to static.crates.io (151.101.65.91) port 443 (#1)
* ALPN: offers h2,http/1.1
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
  0     0    0     0    0     0      0      0 --:--:--  0:00:22 --:--:--     0* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [19 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [2853 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [264 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=static.crates.io
*  start date: Mar 27 10:19:44 2023 GMT
*  expire date: Apr 27 10:19:43 2024 GMT
*  subjectAltName: host "static.crates.io" matched cert's "static.crates.io"
*  issuer: C=BE; O=GlobalSign nv-sa; CN=GlobalSign Atlas R3 DV TLS CA 2023 Q1
*  SSL certificate verify ok.
} [5 bytes data]
* using HTTP/2
* h2h3 [:method: GET]
* h2h3 [:path: /crates/hyper/hyper-0.14.27.crate]
* h2h3 [:scheme: https]
* h2h3 [:authority: static.crates.io]
* h2h3 [user-agent: curl/8.0.1]
* h2h3 [accept: */*]
* Using Stream ID: 1 (easy handle 0x4d7f50)
} [5 bytes data]
> GET /crates/hyper/hyper-0.14.27.crate HTTP/2
> Host: static.crates.io
> user-agent: curl/8.0.1
> accept: */*
>
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [249 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [249 bytes data]
* old SSL session ID is stale, removing
{ [5 bytes data]
< HTTP/2 200
< x-amz-id-2: 3QFUifxGNRNXxRynXQKDmklwi1ph8RlBWM/4P+WkzmrbHMEH5fbsJDbtb2TbDEJ0eSRbTCjV2/8=
< x-amz-request-id: FVXABEP5XV6TQWAN
< x-amz-replication-status: COMPLETED
< last-modified: Mon, 26 Jun 2023 16:10:07 GMT
< etag: "535a734549f7e537c6037ca04cdba01a"
< x-amz-server-side-encryption: AES256
< cache-control: public,max-age=31536000,immutable
< x-amz-version-id: PC_PU7AZI94WlYMKncwvHwa78C2_LfCG
< content-type: application/gzip
< server: AmazonS3
< x-timer: S1688681655.302900,VS0,VE6
< accept-ranges: bytes
< date: Thu, 06 Jul 2023 22:14:15 GMT
< via: 1.1 varnish
< age: 798784
< x-served-by: cache-mad22030-MAD
< x-cache: HIT
< x-cache-hits: 1
< content-length: 195700
<
{ [918 bytes data]
100  191k  100  191k    0     0   8483      0  0:00:23  0:00:23 --:--:-- 49431
* Connection #1 to host static.crates.io left intact

Can be seen a 21 seconds timed out when connecting to the download server, nevertheless curl does a retry to an alternative server what allowed the download. As a curiosity, observing the connections done by cargo with a network monitor, cargo did not tried to connect to such second server.

But I think this may be not significative about the issue we are talking about, because today I run again the test, there was not a time out connection and the downloads keep failing in cargo. What may be happening is both uses different configurations.

curl -L -v https://crates.io/api/v1/crates/hyper/0.14.27/download -o test_file.crate

Output (Today):

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:07 --:--:--     0*   Trying 108.157.98.90:443...
  0     0    0     0    0     0      0      0 --:--:--  0:00:09 --:--:--     0* Connected to crates.io (108.157.98.90) port 443 (#0)
* ALPN: offers h2,http/1.1
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
*  CAfile: D:\curl-8.0.1_1-win64-mingw\bin\curl-ca-bundle.crt
*  CApath: none
  0     0    0     0    0     0      0      0 --:--:--  0:00:09 --:--:--     0{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [19 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [5007 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [264 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [36 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [36 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=crates.io
*  start date: Jan 25 00:00:00 2023 GMT
*  expire date: Feb 23 23:59:59 2024 GMT
*  subjectAltName: host "crates.io" matched cert's "crates.io"
*  issuer: C=US; O=Amazon; CN=Amazon RSA 2048 M02
*  SSL certificate verify ok.
} [5 bytes data]
* using HTTP/2
* h2h3 [:method: GET]
* h2h3 [:path: /api/v1/crates/hyper/0.14.27/download]
* h2h3 [:scheme: https]
* h2h3 [:authority: crates.io]
* h2h3 [user-agent: curl/8.0.1]
* h2h3 [accept: */*]
* Using Stream ID: 1 (easy handle 0x3a7f50)
} [5 bytes data]
> GET /api/v1/crates/hyper/0.14.27/download HTTP/2
> Host: crates.io
> user-agent: curl/8.0.1
> accept: */*
>
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [124 bytes data]
< HTTP/2 302
< content-length: 0
< location: https://static.crates.io/crates/hyper/hyper-0.14.27.crate
< server: nginx
< date: Fri, 07 Jul 2023 10:16:16 GMT
< x-content-type-options: nosniff
< x-frame-options: SAMEORIGIN
< x-xss-protection: 0
< content-security-policy: default-src 'self'; connect-src 'self' *.ingest.sentry.io https://docs.rs https://play.rust-lang.org https://static.crates.io; scri
-src 'self' 'unsafe-eval' 'sha256-n1+BB7Ckjcal1Pr7QNBh/dKRTtBQsIytFodRiIosXdE='; style-src 'self' 'unsafe-inline' https://code.cdn.mozilla.net; font-src https
/code.cdn.mozilla.net; img-src *; object-src 'none'
< access-control-allow-origin: *
< strict-transport-security: max-age=31536000; includeSubDomains
< via: 1.1 vegur, 1.1 14ebdcf21d0310896336dbc3c885587c.cloudfront.net (CloudFront)
< vary: Accept,Accept-Encoding,Cookie
< x-cache: Miss from cloudfront
< x-amz-cf-pop: MAD56-P1
< x-amz-cf-id: gk9GH4U94lIQuQTGJhNlgMFRFegdunKpDV3vHlb6wjTXkGkmDZacTA==
<
{ [0 bytes data]
  0     0    0     0    0     0      0      0 --:--:--  0:00:10 --:--:--     0
* Connection #0 to host crates.io left intact
* Issue another request to this URL: 'https://static.crates.io/crates/hyper/hyper-0.14.27.crate'
  0     0    0     0    0     0      0      0 --:--:--  0:00:10 --:--:--     0*   Trying 151.101.1.91:443...
* Connected to static.crates.io (151.101.1.91) port 443 (#1)
* ALPN: offers h2,http/1.1
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [19 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [2853 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [264 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=static.crates.io
*  start date: Mar 27 10:19:44 2023 GMT
*  expire date: Apr 27 10:19:43 2024 GMT
*  subjectAltName: host "static.crates.io" matched cert's "static.crates.io"
*  issuer: C=BE; O=GlobalSign nv-sa; CN=GlobalSign Atlas R3 DV TLS CA 2023 Q1
*  SSL certificate verify ok.
} [5 bytes data]
* using HTTP/2
* h2h3 [:method: GET]
* h2h3 [:path: /crates/hyper/hyper-0.14.27.crate]
* h2h3 [:scheme: https]
* h2h3 [:authority: static.crates.io]
* h2h3 [user-agent: curl/8.0.1]
* h2h3 [accept: */*]
* Using Stream ID: 1 (easy handle 0x3a7f50)
} [5 bytes data]
> GET /crates/hyper/hyper-0.14.27.crate HTTP/2
> Host: static.crates.io
> user-agent: curl/8.0.1
> accept: */*
>
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [201 bytes data]
< HTTP/2 200
< x-amz-id-2: AwiV2hauFF8pAnACJszSfAZm0uuw8AIK52rlvP6WxcnNFGBaRkh66N+2HWzQm0l8R3/SSBbw5v0=
< x-amz-request-id: 6Y967XESJ41APCTW
< x-amz-replication-status: COMPLETED
< last-modified: Mon, 26 Jun 2023 16:10:07 GMT
< etag: "535a734549f7e537c6037ca04cdba01a"
< x-amz-server-side-encryption: AES256
< cache-control: public,max-age=31536000,immutable
< x-amz-version-id: PC_PU7AZI94WlYMKncwvHwa78C2_LfCG
< content-type: application/gzip
< server: AmazonS3
< x-timer: S1688724977.903215,VS0,VE1
< accept-ranges: bytes
< date: Fri, 07 Jul 2023 10:16:16 GMT
< via: 1.1 varnish
< age: 7183
< x-served-by: cache-mad2200104-MAD
< x-cache: HIT
< x-cache-hits: 1
< content-length: 195700
<
{ [930 bytes data]
100  191k  100  191k    0     0  18395      0  0:00:10  0:00:10 --:--:--  622k
* Connection #1 to host static.crates.io left intact

[nokotto]

This error occurred sometimes (perhaps on some servers?) with the previous version of Cargo. That's why I didn't start an issue about this error at the time.

And some more information:

* cargo downloads the file with the index from github without any problems even now.

* rastup update also works without problems. Does it use Curl or Reqwest backend? And on what servers are the files for it?

Happens me exactly the same. As I've tried installing previous versions where I didn't had so many issues and now fails, I'm more inclined to think something changed in the servers, or in the protocols, until reach the critical mass where now almost always fails.

That, in combination may be with some configuration in the library used by cargo, what passed more or less unperceived until now, due such configuration, or protocol, or number of connections, or something? I have not clue

[Mingun]

Also, could you help bisect which version of cargo started suffering from this issue? That would help us narrow down the scope to investigate.

Maybe the problem is due to switching manifest format, because I faced with this error on cargo that worked several weeks ago (when the old manifest format was still used) and I did not update it from those times (but updated now just to check if issue is fixed or not. The logs below from updated cargo).

My cargo -vV:

PS D:\Users\mingun> rustup update
info: syncing channel updates for 'stable-x86_64-pc-windows-gnu'
info: syncing channel updates for 'nightly-x86_64-pc-windows-gnu'
info: checking for self-update

   stable-x86_64-pc-windows-gnu unchanged - rustc 1.70.0 (90c541806 2023-05-31)
  nightly-x86_64-pc-windows-gnu unchanged - rustc 1.72.0-nightly (85bf07972 2023-07-06)

info: cleaning up downloads & tmp directories
PS D:\Users\mingun> cargo -vV
cargo 1.72.0-nightly (5b377cece 2023-06-30)
release: 1.72.0-nightly
commit-hash: 5b377cece0e0dd0af686cf53ce4637d5d85c2a10
commit-date: 2023-06-30
host: x86_64-pc-windows-gnu
libgit2: 1.6.4 (sys:0.17.2 vendored)
libcurl: 8.1.2-DEV (sys:0.4.63+curl-8.1.2 vendored ssl:Schannel)
os: Windows 6.1.7601 (Windows 7 Professional) [64-bit]

Anyway, there is any workarounds for this?

[Mingun]

I tried to set environment variable CURL_SSL_BACKEND=openssl as suggested here, but without success (I cannot install update, because I'm on Windows 7 and update for Windows 8 and above):

Without quotes seems to just run %CURL_SSL_BACKEND% command:

PS D:\Projects\serde\test_suite> $env:CURL_SSL_BACKEND=openssl; cargo +nightly-gnu -vv test --features unstable
help:

Standard commands
asn1parse         ca                ciphers           cmp
cms               crl               crl2pkcs7         dgst
dhparam           dsa               dsaparam          ec
ecparam           enc               engine            errstr
fipsinstall       gendsa            genpkey           genrsa
help              info              kdf               list
mac               nseq              ocsp              passwd
pkcs12            pkcs7             pkcs8             pkey
pkeyparam         pkeyutl           prime             rand
rehash            req               rsa               rsautl
s_client          s_server          s_time            sess_id
smime             speed             spkac             srp
storeutl          ts                verify            version
x509

Message Digest commands (see the `dgst' command for more details)
blake2b512        blake2s256        md4               md5
mdc2              rmd160            sha1              sha224
sha256            sha3-224          sha3-256          sha3-384
sha3-512          sha384            sha512            sha512-224
sha512-256        shake128          shake256          sm3

Cipher commands (see the `enc' command for more details)
aes-128-cbc       aes-128-ecb       aes-192-cbc       aes-192-ecb
aes-256-cbc       aes-256-ecb       aria-128-cbc      aria-128-cfb
aria-128-cfb1     aria-128-cfb8     aria-128-ctr      aria-128-ecb
aria-128-ofb      aria-192-cbc      aria-192-cfb      aria-192-cfb1
aria-192-cfb8     aria-192-ctr      aria-192-ecb      aria-192-ofb
aria-256-cbc      aria-256-cfb      aria-256-cfb1     aria-256-cfb8
aria-256-ctr      aria-256-ecb      aria-256-ofb      base64
bf                bf-cbc            bf-cfb            bf-ecb
bf-ofb            camellia-128-cbc  camellia-128-ecb  camellia-192-cbc
camellia-192-ecb  camellia-256-cbc  camellia-256-ecb  cast
cast-cbc          cast5-cbc         cast5-cfb         cast5-ecb
cast5-ofb         des               des-cbc           des-cfb
des-ecb           des-ede           des-ede-cbc       des-ede-cfb
des-ede-ofb       des-ede3          des-ede3-cbc      des-ede3-cfb
des-ede3-ofb      des-ofb           des3              desx
idea              idea-cbc          idea-cfb          idea-ecb
idea-ofb          rc2               rc2-40-cbc        rc2-64-cbc
rc2-cbc           rc2-cfb           rc2-ecb           rc2-ofb
rc4               rc4-40            seed              seed-cbc
seed-cfb          seed-ecb          seed-ofb          sm4-cbc
sm4-cfb           sm4-ctr           sm4-ecb           sm4-ofb

warning: spurious network error (3 tries remaining): [35] SSL connect error (schannel: next InitializeSecurityContext failed: SEC_E_ILLEGAL_MESSAGE (0x80090326) - This error usually occurs when a fatal SSL/TLS alert is received (e.g. handshake failed). More detail may be available in the Windows System event log.)
warning: spurious network error (3 tries remaining): [35] SSL connect error (schannel: next InitializeSecurityContext failed: SEC_E_ILLEGAL_MESSAGE (0x80090326) - This error usually occurs when a fatal SSL/TLS alert is received (e.g. handshake failed). More detail may be available in the Windows System event log.)
warning: spurious network error (3 tries remaining): [35] SSL connect error (schannel: next InitializeSecurityContext failed: SEC_E_ILLEGAL_MESSAGE (0x80090326) - This error usually occurs when a fatal SSL/TLS alert is received (e.g. handshake failed). More detail may be available in the Windows System event log.)
warning: spurious network error (2 tries remaining): [35] SSL connect error (schannel: next InitializeSecurityContext failed: SEC_E_ILLEGAL_MESSAGE (0x80090326) - This error usually occurs when a fatal SSL/TLS alert is received (e.g. handshake failed). More detail may be available in the Windows System event log.)
warning: spurious network error (2 tries remaining): [35] SSL connect error (schannel: next InitializeSecurityContext failed: SEC_E_ILLEGAL_MESSAGE (0x80090326) - This error usually occurs when a fatal SSL/TLS alert is received (e.g. handshake failed). More detail may be available in the Windows System event log.)
warning: spurious network error (2 tries remaining): [35] SSL connect error (schannel: next InitializeSecurityContext failed: SEC_E_ILLEGAL_MESSAGE (0x80090326) - This error usually occurs when a fatal SSL/TLS alert is received (e.g. handshake failed). More detail may be available in the Windows System event log.)
warning: spurious network error (1 tries remaining): [35] SSL connect error (schannel: next InitializeSecurityContext failed: SEC_E_ILLEGAL_MESSAGE (0x80090326) - This error usually occurs when a fatal SSL/TLS alert is received (e.g. handshake failed). More detail may be available in the Windows System event log.)
warning: spurious network error (1 tries remaining): [35] SSL connect error (schannel: next InitializeSecurityContext failed: SEC_E_ILLEGAL_MESSAGE (0x80090326) - This error usually occurs when a fatal SSL/TLS alert is received (e.g. handshake failed). More detail may be available in the Windows System event log.)
warning: spurious network error (1 tries remaining): [35] SSL connect error (schannel: next InitializeSecurityContext failed: SEC_E_ILLEGAL_MESSAGE (0x80090326) - This error usually occurs when a fatal SSL/TLS alert is received (e.g. handshake failed). More detail may be available in the Windows System event log.)
error: failed to download from `https://crates.io/api/v1/crates/syn/2.0.23/download`

Caused by:
  [35] SSL connect error (schannel: next InitializeSecurityContext failed: SEC_E_ILLEGAL_MESSAGE (0x80090326) - This error usually occurs when a fatal SSL/TLS alert is received (e.g. handshake failed). More detail may be available in the Windows System event log.)

With quotes (double or single) nothing happened:

PS D:\Projects\serde\test_suite> $env:CURL_SSL_BACKEND="openssl"; cargo +nightly-gnu -vv test --features unstable
warning: spurious network error (3 tries remaining): [35] SSL connect error (schannel: next InitializeSecurityContext failed: SEC_E_ILLEGAL_MESSAGE (0x80090326) - This error usually occurs when a fatal SSL/TLS alert is received (e.g. handshake failed). More detail may be available in the Windows System event log.)
warning: spurious network error (3 tries remaining): [35] SSL connect error (schannel: next InitializeSecurityContext failed: SEC_E_ILLEGAL_MESSAGE (0x80090326) - This error usually occurs when a fatal SSL/TLS alert is received (e.g. handshake failed). More detail may be available in the Windows System event log.)
warning: spurious network error (3 tries remaining): [35] SSL connect error (schannel: next InitializeSecurityContext failed: SEC_E_ILLEGAL_MESSAGE (0x80090326) - This error usually occurs when a fatal SSL/TLS alert is received (e.g. handshake failed). More detail may be available in the Windows System event log.)
warning: spurious network error (2 tries remaining): [35] SSL connect error (schannel: next InitializeSecurityContext failed: SEC_E_ILLEGAL_MESSAGE (0x80090326) - This error usually occurs when a fatal SSL/TLS alert is received (e.g. handshake failed). More detail may be available in the Windows System event log.)
warning: spurious network error (2 tries remaining): [35] SSL connect error (schannel: next InitializeSecurityContext failed: SEC_E_ILLEGAL_MESSAGE (0x80090326) - This error usually occurs when a fatal SSL/TLS alert is received (e.g. handshake failed). More detail may be available in the Windows System event log.)
warning: spurious network error (2 tries remaining): [35] SSL connect error (schannel: next InitializeSecurityContext failed: SEC_E_ILLEGAL_MESSAGE (0x80090326) - This error usually occurs when a fatal SSL/TLS alert is received (e.g. handshake failed). More detail may be available in the Windows System event log.)
warning: spurious network error (1 tries remaining): [35] SSL connect error (schannel: next InitializeSecurityContext failed: SEC_E_ILLEGAL_MESSAGE (0x80090326) - This error usually occurs when a fatal SSL/TLS alert is received (e.g. handshake failed). More detail may be available in the Windows System event log.)
warning: spurious network error (1 tries remaining): [35] SSL connect error (schannel: next InitializeSecurityContext failed: SEC_E_ILLEGAL_MESSAGE (0x80090326) - This error usually occurs when a fatal SSL/TLS alert is received (e.g. handshake failed). More detail may be available in the Windows System event log.)
warning: spurious network error (1 tries remaining): [35] SSL connect error (schannel: next InitializeSecurityContext failed: SEC_E_ILLEGAL_MESSAGE (0x80090326) - This error usually occurs when a fatal SSL/TLS alert is received (e.g. handshake failed). More detail may be available in the Windows System event log.)
error: failed to download from `https://crates.io/api/v1/crates/proc-macro2/1.0.63/download`

Caused by:
  [35] SSL connect error (schannel: next InitializeSecurityContext failed: SEC_E_ILLEGAL_MESSAGE (0x80090326) - This error usually occurs when a fatal SSL/TLS alert is received (e.g. handshake failed). More detail may be available in the Windows System event log.)
PS D:\Projects\serde\test_suite>

Is there a way to manually download all required crates and put them into cargo cache? Some flag in cargo that will show what it tries to download?

By the way, why -vv option does not work? I cannot believe that there is nothing to log when cargo tries to work with network! If I put -vv before toolchain selector, then cargo complain that toolchain is unknown command:

PS D:\Projects\serde\test_suite> cargo -vv +nightly-gnu test --features unstable
error: no such command: `+nightly-gnu`

        Cargo does not handle `+toolchain` directives.
        Did you mean to invoke `cargo` through `rustup` instead?

[nokotto]

Over this past saturday & sunday it has been working fine without any problems, but today the issue has come back again.

[Mingun]

A workaround until this is fixed: download crates manually. Unfortunately, you cannot know all packages at once (if you found the way, please share!), so you need repeat that steps for each missed package:

• run your build / test command, for example cargo build
• wait until cargo failed to download package. It will output a direct download link to it, like

  error: failed to download from `https://crates.io/api/v1/crates/syn/2.0.25/download`
  

• download *.crate by this link (syn-2.0.25.crate in this case) and save it to <user_dir>\.cargo\registry\cache\index.crates.io-6f17d22bba15001f (maybe you will have other hash at the end. If you don't known your user dir, that is one way to get it: %APPDATA%\..\.., worked for me)
• repeat those steps until successful build

[Shadlock0133]

another workaround i found is to use the proxy with custom certificate, e.g. mitmproxy. it's a bit of hassle to setup, but you only have to do it once

[jdno]

I'm almost certain that the SEC_E_ILLEGAL_MESSAGE error is caused by missing support for TLS 1.2 in the Secure Channel component in Windows 7. Sadly it only supports TLS 1.1 out-of-the-box, which was deprecated in June 2018.

The reason why the issues are appearing now is that we've started to serve traffic for crates (static.crates.io) and releases (static.rust-lang.org) through both CloudFront and Fastly, the latter of which only supports TLS 1.2 and higher. This is why it works sometimes (when traffic goes through CloudFront) and sometimes it doesn't (when traffic goes through Fastly).

As far as I can tell, this needs to be fixed on the user's machine. Microsoft provides a support article on how to Update to enable TLS 1.1 and TLS 1.2, which involves installing SP1 and adding a registry key. The article mentions an "easy fix" for the second step so that users don't need to edit the registry manually. A restart might be required.

[jdno]

A discussion in the Factorio forum links to the same knowledge base article as above about enabling TLS 1.1 and 1.2, but also includes a link to a Secure Channel update that apparently adds more cipher suites.

I can't confirm if this works, but if I have some time later I might try spinning up an old Windows 7 version to reproduce the issue locally.

[badrelmers]

@jdno I already have this update installed Secure Channel update, it does not solve this problem

[nokotto]

@jdno I already have this update installed Secure Channel update, it does not solve this problem

I also tried, same result (Thanks @jdno for having tried to find it). Lets identify it as kb2992611

Observing the client hello with wireshark, it adds 5 ciphers, but they are not within the ones that fastly servers accept:

Ciphers added by kb2992611

 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 (0xc028)
 TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 (0x009f)
 TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 (0x009e)
 TLS_RSA_WITH_AES_256_GCM_SHA384 (0x009d)
 TLS_RSA_WITH_AES_128_GCM_SHA256 (0x009c)

The 24 Cipher Suites offered to the server by cargo after kb2992611:

 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 (0xc028)
 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 (0xc027)
 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (0xc014)
 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (0xc013)
 TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 (0x009f)
 TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 (0x009e)
 TLS_RSA_WITH_AES_256_GCM_SHA384 (0x009d)
 TLS_RSA_WITH_AES_128_GCM_SHA256 (0x009c)
 TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 (0xc02c)
 TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 (0xc02b)
 TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384 (0xc024)
 TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256 (0xc023)
 TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA (0xc00a)
 TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA (0xc009)
 TLS_RSA_WITH_AES_256_CBC_SHA256 (0x003d)
 TLS_RSA_WITH_AES_128_CBC_SHA256 (0x003c)
 TLS_RSA_WITH_AES_256_CBC_SHA (0x0035)
 TLS_RSA_WITH_AES_128_CBC_SHA (0x002f)
 TLS_DHE_DSS_WITH_AES_256_CBC_SHA256 (0x006a)
 TLS_DHE_DSS_WITH_AES_128_CBC_SHA256 (0x0040)
 TLS_DHE_DSS_WITH_AES_256_CBC_SHA (0x0038)
 TLS_DHE_DSS_WITH_AES_128_CBC_SHA (0x0032)
 TLS_RSA_WITH_3DES_EDE_CBC_SHA (0x000a)
 TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA (0x0013)

Trying to find a method for to know the ciphers included without installing it, I've uncompressed with 7zip the windows6.1-kb********.msu, and again the .cab resultant, within a directory. With Notepad++ I launched a string "ECDHE" (and "TSL_") search in the directory. Among other strings, will find the NCRYPT_SCHANNEL_INTERFACE manifest xml file that contains the list of ciphers included in the update ( I guess we are talking about ncrypt.dll )

[AlexPublicProfile]

As far as I can tell, this needs to be fixed on the user's machine. Microsoft provides a support article on how to Update to enable TLS 1.1 and TLS 1.2, which involves installing SP1 and adding a registry key. The article mentions an "easy fix" for the second step so that users don't need to edit the registry manually. A restart might be required.

Yes, it works!

The necessary update kb3140245 was already installed for me. But the registry keys specified in the article were not added. And after adding them, everything began to work.

>set CURL_SSL_BACKEND=schannel
>curl -L -o argminmax.crate https://crates.io/api/v1/crates/argminmax/0.6.1/download
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:05 --:--:--     0
100 55395  100 55395    0     0   5319      0  0:00:10  0:00:10 --:--:-- 20380

[nokotto]

@AlexPublicProfile try launching that curl call with the -v parameter (verbose), for to know if that download was done through the cloudfront servers or through the fastly servers, and what cipher was used. The ip from were is downloaded the .crate after the redirection.

[nokotto]

let's summarize

Previously the crates downloads were done by the cloudfront servers, they accept the TSL 1.1,1.2,1.3 protocols, and a number of certain ciphers under each of them. Recently, fastly servers have been included in the load distribution, which support the TSL 1.2, 1.3 protocols, and only six specific ciphers.

When cargo is launched, firstly does a query to crates.io under cloudfront servers, gets crates data, and uses static.crates.io for to download. The IP resolution of the load balancer under static.crates.io can take you to either a cloudfront or a fastly server.

Cargo uses libcurl for the downloads. The curl library has two options available to handle the encryptions of the queries, on the one hand it can use the windows encryption libraries, called SCHANNEL, and on the other hand it can use the OpenSSL library.

Currently cargo only uses schannel, which makes it dependent on the encryption libraries available in windows (TSL protocols + ciphers ).

Windows 7 SP1 users

When cargo is addressed to cloudfront servers , the download occurs normally, without issues, since two conditions are met:

• If win7 does not have TSL 1.2 protocol enabled, the query is performed by TSL 1.1, using their respective ciphers.

• If win7 has TSL 1.2 enabled [1], the query is performed by this protocol, using their respective ciphers.

This happens in both cases because in addition to the TSL protocols, the encryption algorithms (Cipher Suits) available in windows 7 are also available (enabled) on the servers.

[1] Update to enable TLS 1.1 and TLS 1.2 + Regedit edit, #12296 (comment)

When cargo is addressed to fastly servers, the download fails:

• If win7 does not have TSL 1.2 protocol usage enabled, the operation terminates there.

• If win7 has TSL 1.2 enabled (by the method discussed above), the ciphers available in the windows library are not available on the servers.

The accepted ciphers in fastly servers at this moment are

TLS 1.3 (suites in server-preferred order)

 TLS_AES_256_GCM_SHA384                      (0x1302)  ECDH x25519 (eq. 3072 bits RSA)   FS 256
 TLS_CHACHA20_POLY1305_SHA256                (0x1303)  ECDH x25519 (eq. 3072 bits RSA)   FS 256
 TLS_AES_128_GCM_SHA256                      (0x1301)  ECDH x25519 (eq. 3072 bits RSA)   FS 128
		
TLS 1.2 (suites in server-preferred order)
 
 TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256       (0xc02f)  ECDH x25519 (eq. 3072 bits RSA)   FS 128
 TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384       (0xc030)  ECDH x25519 (eq. 3072 bits RSA)   FS 256
 TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 (0xcca8)  ECDH x25519 (eq. 3072 bits RSA)   FS 256

Mean while, workarounds

Someones temporally mounted local proxies, #12296 (comment) , #12296 (comment) or manually downloaded #12296 (comment)

In my case, I did a ping to cloudfront-static.crates.io , copied the IP, edited Windows\System32\drivers\etc\host as Admin, and added a line with such IP as resolver for static.crates.io ( so IP_NUMBER static.crates.io).

I have to warn my workaround have a problem, when the host behind such IP changes, it stop working. And also have the danger of such IP being assigned to other host-client. I should create a script for checking and updating such host, or to install a DNS server local for making such IP overriding and check.

Anyway, it is clear all this workarounds are a pain.

How would it be solved ?

If Cargo had the option for to enable OpenSSL within libcurl, it would make the encryption system independent. What is more, almost all -if not all- the servers use such OpenSSL libraries, that sounds even ironic. Anyway, the important thing is this would include the currently needed ciphers, and even allow the use of the TSL 1.3 protocol, and its used ciphers, in all the windows versions (win7 and win10 specifically), because is predictable that what is happening now will happen again with the time.

Meanwhile, if fastly-static.crates.io servers could enable one of the ciphers that cloudfront-static also has enabled under the protocol TSL1.2, life would be easier for many users.

[badrelmers]

Here is an easier TLS mitm proxy in one line and does not need installation or anything else:
download proxify_0.0.9_windows_386.zip
then run:

proxify.exe -http-addr 127.0.0.1:18080

then add the proxify cert to windows cert manager:

certutil -addstore -f -user "Root" "%USERPROFILE%\.config\proxify\cacert.pem"

now before using Cargo do:

set "CARGO_HTTP_CHECK_REVOKE=false"
set HTTPS_PROXY=127.0.0.1:18080
set HTTP_PROXY=127.0.0.1:18080

that's all

proxify is made in golang so it is faster than the python solutions, and have some very useful and interesting options... as @nokotto said: it is predictable that what is happening now will happen again with the time

[Moth-Tolias]

this is also affecting me on windows 8.1. rust is effectively unusable for me right now.

[AlexPublicProfile]

@AlexPublicProfile try launching that curl call with the -v parameter (verbose), for to know if that download was done through the cloudfront servers or through the fastly servers, and what cipher was used. The ip from were is downloaded the .crate after the redirection.

Yes, I was wrong because of an unlikely coincidence. I had the following sequence of steps:

• checked that Curl returns an error
• added keys to the registry (and restarted the computer)
• checked that Curl began to work normally.

And from this it was logical to suggest that the specified method solves the problem. But in fact, the load balancer changed the servers for me at that very moment.

Now the situation looks like this (and according to nokotto's analysis, no Windows patches will change it):

>curl -o argminmax.crate https://fastly-static.crates.io/crates/argminmax/argminmax-0.6.1.crate
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:04 --:--:--     0
curl: (35) schannel: next InitializeSecurityContext failed: SEC_E_ILLEGAL_MESSAGE (0x80090326) - This error usually occurs when a fatal SSL/TLS alert is received (e.g. handshake failed). More detail may be available in the Windows System event log.

>curl -o argminmax.crate https://cloudfront-static.crates.io/crates/argminmax/argminmax-0.6.1.crate
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 55395  100 55395    0     0  10398      0  0:00:05  0:00:05 --:--:-- 27382

[AlexPublicProfile]

As a very temporary solution: adding a cname record to the dns on the router, redirecting static.crates.io to cloudfront-static.crates.io.

[dbohdan]

The Proxify solution works for me. Thanks, @badrelmers!

I have implemented it on AppVeyor CI. Below are the changes to my appveyor.yml. I had to replace the certutil command with Import-Certificate in PowerShell, because the certutil command would hang for some reason.

index 7e630d3..7c25286 100644
--- a/appveyor.yml
+++ b/appveyor.yml
@@ -8,9 +8,16 @@ cache:
   - '$(CARGO_PATH) -> appveyor.yml, Cargo.lock, Cargo.toml'
 
 install:
+  - cmd: appveyor-retry appveyor DownloadFile https://github.com/projectdiscovery/proxify/releases/download/v0.0.9/proxify_0.0.9_windows_386.zip -FileName proxify.zip
+  - cmd: unzip -n proxify.zip
+  - ps: Start-Process -FilePath proxify.exe -ArgumentList "-config","proxify-config","-http-addr","127.0.0.1:18080"
+  - cmd: timeout /t 5
+  - ps: Import-Certificate -FilePath "proxify-config\cacert.pem" -CertStoreLocation Cert:\LocalMachine\Root
   - cmd: appveyor-retry appveyor DownloadFile https://win.rustup.rs -FileName rustup-init.exe
-  - cmd: rustup-init.exe --default-toolchain %RUST_VERSION% -y
+  - cmd: set HTTP_PROXY=127.0.0.1:18080
+  - cmd: set HTTPS_PROXY=127.0.0.1:18080
   - cmd: set PATH=%CARGO_PATH%\bin;%PATH%
+  - cmd: rustup-init.exe --default-toolchain %RUST_VERSION% -y
 
 build_script:
   - cmd: cargo build

Sorry about the mention noise above. I didn't remove the link to the issue from the commit message when debugging CI.

[jdno]

Hi folks,

Sorry for the long wait. I've been working with our contacts at Fastly and they've enabled a more permissive TLS configuration with more cipher suites for us. I've redeployed our services and the new suites are now live.

Let me know if this fixes the issue with cargo and crates.io for you.

[AlexPublicProfile]

Let me know if this fixes the issue with cargo and crates.io for you.

It's really all right now:

>set CURL_SSL_BACKEND=schannel
>curl -o argminmax.crate https://fastly-static.crates.io/crates/argminmax/argminmax-0.6.1.crate
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 55395  100 55395    0     0   8589      0  0:00:06  0:00:06 --:--:-- 20343

[nokotto]

Let me know if this fixes the issue with cargo and crates.io for you.

I also confirm it fixed it , thanks very much! ( I checked the Cargo's downloads went through Fastly )

[jdno]

Hi folks,

The new configuration has been live for the past 10 days and so far we've seen no issues. I'm going to close the GitHub issue, but feel free to report back if you get the above error on a patched Windows version again.

Thanks 👋

[hp8wvvvgnj6asjm7]

For anyone having problems behind a windows firewall: there is two cargo.exe.. one inside .cargo and one inside .rustup. Make sure to allow the other.

[solisadeqi]

facing same issue on mac
cargo -vV
cargo 1.76.0 (c84b367 2024-01-18)

any idea anyone?