curl with nss doesn't send intermediate client certificates?

[tomfitzhenry]

I did this

curl --cert client.crt --key client.crt https://server.that.prompts.for.client.certs.example.com/

where client.crt contains a client certificate, an intermediate certificate, and the corresponding private key.

I expected the following

Successful connection!

I actually got

...
* SSL read: errno -12195 (SSL_ERROR_UNKNOWN_CA_ALERT)
* Peer does not recognize and trust the CA that issued your certificate.

curl/libcurl version

$ curl --version
curl 7.29.0 (x86_64-redhat-linux-gnu) libcurl/7.29.0 NSS/3.19.1 Basic ECC zlib/1.2.7 libidn/1.28 libssh2/1.4.3
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smtp smtps telnet tftp
Features: AsynchDNS GSS-Negotiate IDN IPv6 Largefile NTLM NTLM_WB SSL libz

operating system

$ cat /etc/redhat-release
CentOS Linux release 7.2.1511 (Core)

More details

When I execute the same command, with the same certificates, against a curl/openssl:

curl 7.48.0 (x86_64-apple-darwin13.4.0) libcurl/7.48.0 OpenSSL/1.0.2h zlib/1.2.5
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: IPv6 Largefile NTLM NTLM_WB SSL libz TLS-SRP UnixSockets

then it successfully connects.

When I execute the same command, with the same version of curl, against a host that has the full CA chain, then it successfully connects.

[tomfitzhenry]

If that's not enough to spot the problem, please say, and I can spend time building a proper reproducible bug case.

[bagder]

Are you even using the same CA cert store with NSS as with OpenSSL ?

And you say "curl with nss doesn't send intermediate client certificates" while the NSS error message say SSL_ERROR_UNKNOWN_CA_ALERT which to me sounds like NSS didn't like the server's certificate. Are you sure it isn't?

https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/SSL_functions/sslerr.html lists the error codes and for the above code it explains:

"Peer does not recognize and trust the CA that issued your certificate."

[kdudka]

On Wednesday, June 01, 2016 10:55:47 Tom Fitzhenry wrote:

If that's not enough to spot the problem, please say, and I can spend time
building a proper reproducible bug case.

That would definitely help! I have never used such a client certificate
myself. If you prepare a test-case that works with OpenSSL but not with NSS,
I will have a look at the implementation to see how we can improve it.

Kamil

[tomfitzhenry]

@bagder I just checked with -k, and I observed the same behaviour, suggesting it isn't that NSS doesn't like the server's certificate.

I think "Peer does not recognize and trust the CA that issued your certificate." means the server (NSS's peer) is the one having trust issues. That this error is under the section "All the error codes in the following block indicate that the local socket received an SSL3 or TLS alert record from the remote peer, reporting some issue that it had with an SSL record or handshake message it received." supports this too.

I will produce a reproducible test case, however.

[kdudka]

On Thursday, June 02, 2016 02:22:23 Tom Fitzhenry wrote:

I think "Peer does not recognize and trust the CA that issued your
certificate." means the server (NSS's peer) is the one having trust issues.

Exactly. That is also my understanding of the issue. I suspect the reason
that peer does not trust your certificate is that the intermediate certificate
is not presented to the peer at all.

[ghost]

Oh, and when I tried the same scenario, but with Firefox (which uses NSS), it was able to connect, suggesting the problem is not NSS, but how curl uses NSS.

But anyway, I need to talk less, and produce a test case. :)

[kdudka]

On Thursday, June 02, 2016 02:49:34 bbc-tomfitzhenry wrote:

Oh, and when I tried the same scenario, but with Firefox (which uses NSS),
it was able to connect, suggesting the problem is not NSS, but how curl
uses NSS.

Sort of. curl uses nss-pem to load certificates and keys from files, which
is a PKCS #11 module and definitely not something that Firefox would use.

Actually, you can try to use the SSL_DIR environment variable to make curl
use the certificate database from Firefox.

[bagder]

Out of date, this might be fixed if nss-pem is used. Feel free to (request a) reopen.

[thraidh]

I just ran into the same issue. Here is my setup:

• self-signed server, accepting a private root-CA
• client with client-certificate, signed by intermediate CA, signed by private root-CA
  Using this version of curl provided by CentOS 7:

curl 7.29.0 (x86_64-redhat-linux-gnu) libcurl/7.29.0 NSS/3.19.1 Basic ECC zlib/1.2.7 libidn/1.28 libssh2/1.4.3
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smtp smtps telnet tftp
Features: AsynchDNS GSS-Negotiate IDN IPv6 Largefile NTLM NTLM_WB SSL libz

It seems, that the intermediate CA is not sent in the SSL handshake, so the server cannot verify the client certificate. If the server knows about the intermediate certificate or I use openssl s_client with the same certificates, everything works fine. Neither using --cert ./full-chain.crt nor --CAfile ./root-and-intermediate-ca.crt works with curl.

[kdudka]

Yes, nss-pem currently does not support client certificates signed by intermediate certificates. Please open a pull request at https://github.com/kdudka/nss-pem or at least help us with setting up a testing environment needed for development of the feature.

As a workaround, you can import the client certificate to the native NSS database and make libcurl use it by setting the SSL_DIR environment variable and selecting the client certificate by its nickname.

[thraidh]

I'm working on a script setting up a test environment and perform a test case.

[thraidh]

Here is a script that produces a self-signed server certificate, a root CA, a client CA and a client certificate signed by the intermediate CA. After that it will run openssl s_server and openssl s_client using those certificates to show the good case and then openssl s_server and curl to show the difference.
It was tested on Fedora 22 with curl 7.40.0 using NSS/3.21 and openssl 1.0.1k to create the certificates.

setup-testcase.zip

[kdudka]

Thank you very much for providing the setup script!

It looks like the only problem is that curl completely ignores the --cacert option if the -k option is used. Could you please check whether the following patch makes it work as expected?

master...kdudka:insecure-cacert

[thraidh]

Thanks for your work. I'll check.

[thraidh]

Just added the server.crt to the ca-bundle.crt and removed -k. This confirms that intermediate certificates are handled correctly in general.
Now I'll have to build curl from source to check if your change works for me. May take a while.

[kdudka]

Perfect. I can prepare a Copr repository for testing in case it helped. Just let me know which version of curl for which Fedora/EPEL release you prefer. If you are still using Fedora 22, it is no longer available as a choice in Copr but I can at least prepare a SRPM that you can easily rebuild locally.

[thraidh]

Fedora is just running on my desktop (for which I somehow don't find the time to upgrade) . The actual problem occurred on a CentOS 7, curl 7.29.0. I tried to apply your patch to the source RPM, but apparently the directory layout changed and then I got distracted by real life. I should be able to fix that tomorrow.

[kdudka]

Please try the packages of RHEL-7.3 curl with the above patch included from the following Copr:
https://copr.fedorainfracloud.org/coprs/kdudka/curl-testing/

[thraidh]

I'll test as soon as possible. Unfortunately I got sick, so it may take some time.

[kdudka]

No worries. This is broken for 8+ years, so a few more days or weeks to get it fixed is fine. I wish you to get well soon!