(35) Insufficent Randomness (for Windows 7 with OpenSSL 3.0.3)

[DanTheMann15]

Since curl was updated with OpenSSL 3.0.3, i started getting error 35 when trying to get info on ipsw files hosted on apple's servers.
i'm not exactly sure why this issue is occuring, but here's some information that should help us track it down a bit.

OS: Windows 7 SP1 (x86_64)
Version(s): CURL 7.83.1 (7.83.0 with OpenSSL 3.0.3)
Downloaded From: https://curl.se/windows/

-V Output:

curl 7.83.1 (x86_64-pc-win32) libcurl/7.83.1 OpenSSL/3.0.3 (Schannel) zlib/1.2.12 brotli/1.0.9 libidn2/2.3.2 libssh2/1.10.0 nghttp2/1.47.0 libgsasl/1.10.0
Release-Date: 2022-05-11
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli gsasl HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz MultiSSL NTLM SPNEGO SSL SSPI TLS-SRP UnixSockets

How to Reproduce (example):
curl -i https://updates.cdn-apple.com/2022FCSWinter/fullrestores/002-81308/6BD01782-B7EA-4BF7-987A-C3D3F5755CC0/iPhone_4.0_64bit_15.4.1_19E258_Restore.ipsw

Expected (Something like this):

HTTP/1.1 200 OK
Content-Type: binary/octet-stream
Content-Length: 5216804428
Last-Modified: Mon, 28 Mar 2022 01:13:50 GMT
x-amz-meta-digest-sh1: bcb98612c204172a1b1a7c12577ffcdf38e0ead0
x-amz-meta-digest-sha256: 73a5eb08698c2ce1eb6499824800c09c855ad1cb53e7f8028c4410ee72599b93
x-amz-version-id: 2J8Pjdc3WOpauaC40VRcwcTm.9YLm1Cb
Accept-Ranges: bytes
Server: AmazonS3
Date: Sat, 14 May 2022 12:20:02 GMT
X-Cache: RefreshHit from cloudfront, hit-fresh, miss, none
Via: 1.1 b55214c298a6470ca4f296a9e493d5f2.cloudfront.net (CloudFront), http/1.1 uschi5-edge-lx-003.ts.apple.com (acdn/14
4.13255), http/1.1 uschi5-edge-bx-017.ts.apple.com (acdn/144.13255), http/1.1 uschi5-edge-bx-017.ts.apple.com (acdn/144.
13255)
X-Amz-Cf-Pop: ORD52-C1
X-Amz-Cf-Id: M5R-liZCWATc_FUuZIkMJX4_Uog2gHIFf4GaCkvSUb-nzaNNKyGV6g==
Cache-Control: max-age=21600, public
Etag: "b9ea458237a909061b66d2a7f7477b61-622"
Age: 14206
CDNUUID: aba81b1c-2462-4dba-9587-32816591578e-614671188
Connection: keep-alive

Warning: Binary output can mess up your terminal. Use "--output -" to tell
Warning: curl to output it to your terminal anyway, or consider "--output
Warning: " to save to a file.

Actual Result:

curl: (35) Insufficient randomness

[bagder]

My understanding of this is that OpenSSL 3 (actually since OpenSSL version >= 1.1.0) inits the random seed itself, which means that RAND_status() should return 1. When it doesn't, then something clearly is wrong and libcurl returns error for it:

curl/lib/vtls/openssl.c

Line 483 in 5227367

failf(data, "Insufficient randomness");

So... something seems to be wrong with either my assumptions or with the conditions in which the code runs in your system!

[alexetgus]

Hi,

I have the same problem with all the URLs. I think I'll do a rollback...

OS : Win7 Pro (x64)
Curl : 7.83.1
OpenSSL : 3.0.3 in Curl, 3.0.1 in Windows system

EDIT
I went back to version 7.83.0 and it works fine.

choco install curl --version 7.83.0 --allow-downgrade -y

[TheXman-TX]

I am also experiencing the exact same issue as the original poster. When trying to access ANY https site, using the 7.83.1 version of curl.exe on Windows 7 Pro x64, it immediately returns with:

curl: (35) Insufficient randomness.

This appears to be an issue with openssl 3.0.3, When I use SCHANNEL, with the 7.83.1 version of curl.exe, it works fine.

When I execute the exact same commands with cURL 7.83.0, I get the expected results.

For the record, when using the 7.83.1 version of curl.exe on Windows 10 Pro x64, everything works as expected.

cURL Version Info:

curl 7.83.1 (x86_64-pc-win32) libcurl/7.83.1 OpenSSL/3.0.3 (Schannel) zlib/1.2.12 brotli/1.0.9 libidn2/2.3.2 libssh2/1.10.0 nghttp2/1.47.0 libgsasl/1.10.0
Release-Date: 2022-05-11
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp 
Features: alt-svc AsynchDNS brotli gsasl HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz MultiSSL NTLM SPNEGO SSL SSPI TLS-SRP UnixSockets

[rsbeckerca]

Did you build OpenSSL 3.0.3 yourself? If so, what randomizer did you select?

[TheXman-TX]

To whom are you asking that question? If it is to me, the answer is no. I am using the openssl version that is embedded in the executable. The executable was downloaded from the curl.se site on the following page:

https://curl.se/windows/

[rsbeckerca]

To whom are you asking that question? If it is to me, the answer is no. I am using the openssl version that is embedded in the executable. The executable was downloaded from the curl.se site on the following page:

https://curl.se/windows/

Just checking whether a custom build might have chosen a bad randomizer. I have seen that happen when a poorly seeded PRNGD is set up.

[bagder]

This sounds like an OpenSSL 3.0.3 (on Windows 7?) bug to me. Do any earlier versions of OpenSSL 3 work better?

/cc @vszakats

[DanTheMann15]

this build of curl (for windows) was the last one to work correctly: https://curl.se/windows/dl-7.83.0/
starting with build 7.83.0_1 when the windows curl builds started being built with openssl 3.0.3.

3.0.2 worked fine, so i am not sure exactly what happened here. it could be a bug in openssl 3.0.3 but it's going to be some time and testing before we figure out the root cause.

EDIT: i'll add this in to help us figure this out more: openssl/openssl@openssl-3.0.2...openssl-3.0.3
the error does not occur in Windows 10 (strangely). i can't say for Windows 8/8.1.

[GaneJ]

I had the same problem too. Every https download. It didn't happen in http download.
Whether to use .netrc or not.
It didn't happen with
version 7.82.0 (libcurl/7.82.0 OpenSSL/3.0.2 (Schannel) zlib/1.2.11 brotli/1.0.9 libidn2/2.3.2 libssh2/1.10.0 nghttp2/1.47.0 libgsasl/1.10.0).
It only happen with
version 7.83.0 (libcurl/7.83.0 OpenSSL/3.0.3 (Schannel) zlib/1.2.12 brotli/1.0.9 libidn2/2.3.2 libssh2/1.10.0 nghttp2/1.47.0 libgsasl/1.10.0)
and
version 7.83.1 (libcurl/7.83.1 OpenSSL/3.0.3 (Schannel) zlib/1.2.12 brotli/1.0.9 libidn2/2.3.2 libssh2/1.10.0 nghttp2/1.47.0 libgsasl/1.10.0).

Btw, I'm using Windows 7. I got 8mb random file. I'm using default x64 build (https://curl.se/windows/).

this my config:
--show-error
--random-file "D:/_H87M-Pro/_cURL_Aria/_Default_Conf/Random.bin"
--cacert "C:/Application/Common/x64/cURL-CA-Bundle.crt"
--tcp-nodelay
--tlsv1
--no-ntlm
--no-ntlm-wb
--delegation none
--no-xattr
--ftp-pasv
--ftp-method singlecwd
--ftp-ssl-ccc-mode active
--keepalive-time 60
--retry-max-time 300
--retry-delay 20
--expect100-timeout 20
--connect-timeout 60
--retry 2
--parallel-max 3
--proto -all,+http,+https,+ftp,+ftps,+rtsp
--proto-redir -all,+http,+https,+ftp,+ftps,+rtsp
--proto-default http
--location
--location-trusted
--max-redirs 16
--remote-time
--create-dirs
--noproxy *
--referer ;auto
--netrc
--digest
--get

==================================================================

Why there is no random engine with version 7.83.0 and version 7.83.1?
Maybe this is the problem.

Random engine version 7.83.0 and version 7.83.1:
curl.exe --engine list
Build-time engines:
none

Random engine version 7.82.0:
curl.exe --engine list
Build-time engines:
rdrand
dynamic

[alexetgus]

Yes, that's certainly it. Version 7.83.1 no longer has a random number generator under Win7...

[rsbeckerca]

Yes, that's certainly it. Version 7.83.1 no longer has a random number generator under Win7...

In our build, we have this:

$ curl --version
curl 7.83.1 (nse-tandem-nskJ06) libcurl/7.83.1 OpenSSL/3.0.2 zlib/1.2.8
Release-Date: 2022-05-11
Protocols: dict file ftp ftps gopher gophers http https imap imaps mqtt pop3 pop3s rtsp smtp smtps telnet tftp
Features: alt-svc HSTS HTTPS-proxy IPv6 libz NTLM NTLM_WB SSL TLS-SRP UnixSockets
$ curl --engine list
Build-time engines:
  dynamic

This applies to both ia64, using PRNGD, and x86, using rdrand from OpenSSL.

[vszakats]

Thanks for the tests to all.

If I summarize correctly, both curl 7.83.0 and 7.83.1 built with OpenSSL 3.0.3, x64 flavour, when run on Win7, will fail with not enough randomness. 7.83.0 + 3.0.2 works fine. Buggy combinations will list no engines with curl --engine list, while good ones list both rdrand and dynamic.

The single constant component in the failing cases is OpenSSL 3.0.3. The way OpenSSL is built did not change between 3.0.2 and 3.0.3. This indicates it either to be a 3.0.3 bug, or an update that required some change in the way OpenSSL should be built. {NEWS,CHANGES}.md do not suggest anything like that, and nothing related to random generators or engines.

List of changes between 3.0.3 and 3.0.2. 90 commits in total.

One suspicious and low-level change is the full switch to OPENSSL_strcasecmp() from strcasecmp(). In curl-for-win builds of OpenSSL, the new function translates to stricmp_l() (in 3.0.2 it translated to stricmp()). This is locale-dependent comparison, that also requires explicit locale initialization, which is supposed to be done by OpenSSL automatically. It's possible (likely?) that the new, locale-dependent string comparator fails depending on the environment. String comparison is used in critical places when loading engines.

Probably the best would be to report this to OpenSSL.