`--ca-native`/`CURLSSLOPT_NATIVE_CA` break when cert store contains invalid certs

[MichalPetryka]

I did this

Enabling --ca-native/CURLSSLOPT_NATIVE_CA while using OpenSSL (backends other than OpenSSL and SChannel untested) and having invalid certs in Windows cert store makes CURL (both exe and libCurl) error out with:

curl: (35) TLS connect error: error:068000DD:asn1 encoding routines::illegal padding

This causes Serbian users to be unable to use OpenSSL based Curl since for whatever reason the Serbian government has their citizens add invalid certificates to their Windows cert store to access their services.

This has many other sources referencing the issue with either OpenSSL itself or software related to it:
openssl/openssl#16701
openssl/openssl#25023
https://www.dropboxforum.com/discussions/101001016/installation-fails-for-people-from-serbia-due-to-certificates---solution-windows/490440
https://bugs.python.org/issue45312
python/cpython#79846

Since the linked OpenSSL issues indicate they have no intention of allowing such invalid certs (which some APIs like SChannel seem to do) and suggest filtering off and ignoring such certs on application level, I believe CURL should do so either by default with --ca-native/CURLSSLOPT_NATIVE_CA or when provided with some new flags since Curl does the cert store loading by itself in:

curl/lib/vtls/openssl.c

Line 3214 in 41fe621

static CURLcode import_windows_cert_store(struct Curl_easy *data,

As far as I believe ignoring such certs brings no security implications since not adding them can only cause TLS rejections, not improper acceptances so I'd propose doing it automatically with the current options.

Reproduction:

1. Build Curl with OpenSSL backend
2. Add the cert from http://crl.mup.gov.rs/MUPCARoot.crt to Windows cert store
3. curl.exe --ca-native https://api.ipify.org

I expected the following

Curl is working and ignoring invalid certs that are not necessary to complete the request.

curl/libcurl version

curl 8.15.0-DEV (Windows) libcurl/8.15.0-DEV OpenSSL/3.5.1 (Schannel) zlib/1.3.1 brotli/1.1.0 zstd/1.5.7 nghttp2/1.66.0
Release-Date: [unreleased]
Protocols: dict file ftp ftps gopher gophers http https imap imaps ipfs ipns mqtt pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli HSTS HTTP2 HTTPS-proxy IPv6 Kerberos Largefile libz MultiSSL NTLM SPNEGO SSL SSPI threadsafe TLS-SRP Unicode UnixSockets zstd

operating system

Windows 10/11

[icing]

It would be useful to get a trace where the mentioned error really occurs. On first inspection, it seems that import_windows_cert_store() does ignore certificates with errors already. So, if you could run a command on your system like:

> curl -vvvv -ca-native url

it could help us understand what is going on. Thanks.

[ced777ric]

It seems that the issue does not occur when using verbose logging, only without logging

log.txt

[icing]

Which points to something very weird going on in your system. If there is some timing effect, you can try to reduce the tracing. For example trace only the SSL related things via:

> curl -v --trace-config ssl --ca-native https://api.ipify.org

and see if that makes a difference. Another thing I see in your log is that your connect times seem very long:

08:37:07.619000 [0-0] == Info: [DNS] resolve thread started for of api.ipify.org:443
...
08:37:07.744000 [0-0] == Info:   Trying 172.67.74.152:443...
...
08:37:07.869000 [0-0] == Info: [TCP] connected
...
08:37:08.290000 [0-0] == Info: [TCP] send(len=1564) -> 0, 1564
...
08:37:08.337000 [0-0] == Info: [SSL] ossl_populate_x509_store, path=none, blob=0
...
08:37:08.353000 [0-0] == Info: successfully imported Windows CA store
...
08:37:08.400000 [0-0] <= Recv SSL data, 1210 bytes (0x4ba)
...
08:37:08.759000 [0-0] <= Recv SSL data, 2524 bytes (0x9dc)
...
08:37:09.322000 [0-0] == Info: [TCP] send(len=80) -> 0, 80
08:37:09.338000 [0-0] == Info: SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / X25519MLKEM768 / id-ecPublicKey

Your curl takes ~120ms to resolve the hostname, ~120ms to connect the TCP socket, ~400ms(!) to send the TLS client hello, 110ms to get the first TLS server reply, 350ms to get the final TLS server reply and 560ms to send the TLS client answer. Either your client is very slow, or something is installed in your Windows machine/network that interferes here a lot.

To compare, on my macOS machine curl takes 30ms in total to establish the SSL connection to api.ipify.org, compared to the ~1600ms on your machine.

I suspect someone is messing with you, be it a local AntiVir/Security or middle boxes on your network path.

[MichalPetryka]

Which points to something very weird going on in your system. If there is some timing effect, you can try to reduce the tracing. For example trace only the SSL related things via:

curl -v --trace-config ssl --ca-native https://api.ipify.org
and see if that makes a difference. Another thing I see in your log is that your connect times seem very long:

08:37:07.619000 [0-0] == Info: [DNS] resolve thread started for of api.ipify.org:443
...
08:37:07.744000 [0-0] == Info:   Trying 172.67.74.152:443...
...
08:37:07.869000 [0-0] == Info: [TCP] connected
...
08:37:08.290000 [0-0] == Info: [TCP] send(len=1564) -> 0, 1564
...
08:37:08.337000 [0-0] == Info: [SSL] ossl_populate_x509_store, path=none, blob=0
...
08:37:08.353000 [0-0] == Info: successfully imported Windows CA store
...
08:37:08.400000 [0-0] <= Recv SSL data, 1210 bytes (0x4ba)
...
08:37:08.759000 [0-0] <= Recv SSL data, 2524 bytes (0x9dc)
...
08:37:09.322000 [0-0] == Info: [TCP] send(len=80) -> 0, 80
08:37:09.338000 [0-0] == Info: SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / X25519MLKEM768 / id-ecPublicKey

Your curl takes ~120ms to resolve the hostname, ~120ms to connect the TCP socket, ~400ms(!) to send the TLS client hello, 110ms to get the first TLS server reply, 350ms to get the final TLS server reply and 560ms to send the TLS client answer. Either your client is very slow, or something is installed in your Windows machine/network that interferes here a lot.

To compare, on my macOS machine curl takes 30ms in total to establish the SSL connection to api.ipify.org, compared to the ~1600ms on your machine.

I suspect someone is messing with you, be it a local AntiVir/Security or middle boxes on your network path.

1. This is a log from a reproduction test on a clean Windows 10 VM (with only the cert added), there's nothing that could be messing with it.
2. Only Serbian users report this and confirm the issue is gone when they remove the certs.
3. You have the linked OpenSSL issues that confirm it does reject those certs.
4. You have full repro steps provided in the issue which you could use to verify that the issue is not machine dependent.

In other words, the issue is verifiably and fully reproducibly proven to be caused by those certs, so I'd really prefer to avoid blaming other unrelated factors.
Even if it is a timing issue - which seems unlikely with it reproducing the same on every try on multiple machines - that'd still be a bug in the Curl code that needs to be fixed.

[bagder]

avoid blaming other unrelated factors.

Hey, no one is blaming anything or anyone. You come to us with a problem you have. It is perfectly legitimate for us to ask questions about what you see and report.

Do not assume that everyone can and is even willing to reproduce those problems (for example because it requires running and debugging on Windows) and you should probably instead be glad that some of us are still interested to help even if we can't - but that may mean that we ask more questions.

[icing]

@MichalPetryka I am not here to prove you wrong. I am saying that I cannot reproduce your problem and give hints where I see strange behaviour that I would use in my investigation of the root cause. In order to help you, as I cannot reproduce.

Do with that what you want. I do not see a bug in curl from this.

[MichalPetryka]

avoid blaming other unrelated factors.

Hey, no one is blaming anything or anyone. You come to us with a problem you have. It is perfectly legitimate for us to ask questions about what you see and report.

Do not assume that everyone can and is even willing to reproduce those problems (for example because it requires running and debugging on Windows) and you should probably instead be glad that some of us are still interested to help even if we can't - but that may mean that we ask more questions.

Sorry but it's a bit frustrating to receive a response that presumes the issue is caused by machine related factors after the issue is reported to be reproductible on any Windows machine with the provided steps. We are happy to assist with testing the issue but for that to be meaningful, I wanted to reiterate that the issue does not appear environment specific.

I am saying that I cannot reproduce your problem

I might be missing something but I did not see a mention of that in your original reply.
Are you sure you meet all of the required conditions:

1. A Windows Machine
2. A curl build using the OpenSSL API
3. Having the provided certificate added to an active Windows Certificate Store
4. Using --ca-native for the request

If yes then that'd be very surprising since the issue reproduces for multiple Serbian users and our VM reliably with just those conditions.

[vszakats]

I could reproduce it with the MSYS2 mingw-w64 build of curl (as one built with OpenSSL) on Windows 11 (VM):

> curl.exe -vvvv --ca-native https://curl.se/
[...]
14:57:01.691000 [0-0] == Info: [SSL] ossl_bio_cf_in_read(len=5) -> -1, err=81
14:57:01.691000 [0-0] == Info: [SSL] ossl_populate_x509_store, path=C:/msys64/clangarm64/etc/ssl/certs/ca-bundle.crt, blob=0
14:57:01.691000 [0-0] == Info: successfully imported Windows ROOT store
14:57:01.707000 [0-0] == Info: successfully imported Windows CA store
14:57:01.709000 [0-0] == Info:  CAfile: C:/msys64/clangarm64/etc/ssl/certs/ca-bundle.crt
14:57:01.709000 [0-0] == Info:  CApath: none
14:57:01.709000 [0-0] == Info: [SSL] SSL_connect() -> err=-1, detail=1
14:57:01.709000 [0-0] == Info: TLS connect error: error:068000DD:asn1 encoding routines::illegal padding
14:57:01.709000 [0-0] == Info: [SSL] cf_connect() -> 35, done=0
14:57:01.709000 [0-0] == Info: [HTTPS-CONNECT] connect, all attempts failed
14:57:01.709000 [0-0] == Info: [HTTPS-CONNECT] connect -> 35, done=0
[...]

> curl.exe -V
curl 8.13.0 (Windows) libcurl/8.13.0 OpenSSL/3.5.0 zlib/1.3.1 brotli/1.1.0 zstd/1.5.7 libidn2/2.3.8 libpsl/0.21.5 libssh2/1.11.1 nghttp2/1.65.0 nghttp3/1.9.0
Release-Date: 2025-04-02
Protocols: dict file ftp ftps gopher gophers http https imap imaps ipfs ipns ldap ldaps mqtt pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp ws wss
Features: alt-svc AsynchDNS brotli HSTS HTTP2 HTTP3 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM PSL SPNEGO SSL SSPI threadsafe TLS-SRP UnixSockets zstd

Official Windows binaries (LibreSSL) and curl bundled with Windows 11 (Schannel) did not reproduce it.

With the latest packages:

> curl.exe -vvvv --ca-native https://curl.se/
[...]
15:17:37.436000 [0-0] == Info: [SSL] ossl_bio_cf_in_read(len=5) -> 81, 0
15:17:37.436000 [0-0] == Info: [SSL] ossl_populate_x509_store, path=C:/msys64/clangarm64/etc/ssl/certs/ca-bundle.crt, blob=0
15:17:37.440000 [0-0] == Info: successfully imported Windows ROOT store
15:17:37.441000 [0-0] == Info: successfully imported Windows CA store
15:17:37.446000 [0-0] == Info:  CAfile: C:/msys64/clangarm64/etc/ssl/certs/ca-bundle.crt
15:17:37.446000 [0-0] == Info:  CApath: none
15:17:37.448000 [0-0] == Info: [SSL] SSL_connect() -> err=-1, detail=1
15:17:37.448000 [0-0] == Info: TLS connect error: error:068000DD:asn1 encoding routines::illegal padding
15:17:37.448000 [0-0] == Info: [SSL] cf_connect() -> 35, done=0
15:17:37.448000 [0-0] == Info: [HTTPS-CONNECT] connect, all attempts failed
15:17:37.448000 [0-0] == Info: [HTTPS-CONNECT] connect -> 35, done=0
[...]

> curl.exe -V
curl 8.15.0 (Windows) libcurl/8.15.0 OpenSSL/3.5.2 zlib/1.3.1 brotli/1.1.0 zstd/1.5.7 libidn2/2.3.8 libpsl/0.21.5 libssh2/1.11.1 nghttp2/1.66.0 ngtcp2/1.14.0 nghttp3/1.11.0
Release-Date: 2025-07-16
Protocols: dict file ftp ftps gopher gophers http https imap imaps ipfs ipns ldap ldaps mqtt pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp ws wss
Features: alt-svc AsynchDNS brotli HSTS HTTP2 HTTP3 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM PSL SPNEGO SSL SSPI threadsafe TLS-SRP UnixSockets zstd

[icing]

It seems like OpenSSL does not deeply inspect the certificate on loading, but will on use and then fail. This means we cannot in curl detect the offending certificate, I guess?

[icing]

Reading the openssl issue 25023, the trouble with http://crl.mup.gov.rs/MUPCARoot.crt is that it is indeed a broken certificate, OpenSSL freaks out on parsing it and there is no option in OpenSSL to change that behaviour, e.g. make it ignore such a cert in the root store.

If that all is correct, I do not see what we could do in curl to make this work. There is nothing wrong in curl how it loads these certificates or uses OpenSSL.