SSL Error on macOS, when --insecure flag is given. #1184

Closed
fchandur opened this Issue Dec 30, 2016 · 9 comments

Projects

None yet

4 participants

@fchandur

I did this

./curl --insecure https://www.google.com

I expected the following

html document from google:
<!doctype html><html itemscope="" itemtype="...

Instead, I got this message

curl: (35) SSL: CA certificate set, but certificate verification is disabled

curl/libcurl version

$ ./curl --version
curl 7.52.1 (x86_64-apple-darwin16.3.0) libcurl/7.52.1 SecureTransport zlib/1.2.8
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp 
Features: AsynchDNS Debug TrackMemory IPv6 Largefile NTLM NTLM_WB SSL libz UnixSockets 

operating system

System Version: macOS 10.12.2 (16C67)
Kernel Version: Darwin 16.3.0

configuration

$ ./curl-config --configure
 '--with-darwinssl' '--enable-threaded-resolver' '--enable-debug' '--without-libssh2' '--prefix=/Users/user/Downloads/curl-7.52.1/build'

verbose

$ ./curl --insecure --verbose https://www.google.com
* STATE: INIT => CONNECT handle 0x7fd920801a08; line 1413 (connection #-5000)
* Rebuilt URL to: https://www.google.com/
* Added connection 0. The cache now contains 1 members
* STATE: CONNECT => WAITRESOLVE handle 0x7fd920801a08; line 1450 (connection #0)
*   Trying 172.217.6.100...
* TCP_NODELAY set
* STATE: WAITRESOLVE => WAITCONNECT handle 0x7fd920801a08; line 1531 (connection #0)
* Connected to www.google.com (172.217.6.100) port 443 (#0)
* STATE: WAITCONNECT => SENDPROTOCONNECT handle 0x7fd920801a08; line 1583 (connection #0)
* Marked for [keep alive]: HTTP default
* SSL: CA certificate set, but certificate verification is disabled
* Marked for [closure]: Failed HTTPS connection
* STATE: SENDPROTOCONNECT => PROTOCONNECT handle 0x7fd920801a08; line 1597 (connection #0)
* Expire cleared
* multi_done
* Curl_http_done: called premature == 0
* Closing connection 0
* The cache now contains 0 members
curl: (35) SSL: CA certificate set, but certificate verification is disabled

On an older osx, with the same version of libcurl as above, and same configuration, I do get the expected behavior. Following are some of the details:

curl/libcurl version

./curl --version
curl 7.52.1 (x86_64-apple-darwin15.6.0) libcurl/7.52.1 SecureTransport zlib/1.2.5
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp 
Features: AsynchDNS IPv6 Largefile NTLM NTLM_WB SSL libz UnixSockets 

configuration

$ ./curl-config --configure
'--without-libssh2' '--enable-threaded-resolver' '--with-darwinssl' '--prefix=/Users/user/Downloads/curl-7.52.1/build'

I did this

$ ./curl --insecure https://www.google.com

I got this

html document from google:
<!doctype html><html itemscope="" itemtype="...

@nickzman
Collaborator

Not an issue. You get that error if you put curl into insecure mode while simultaneously specifying a server certificate file or bundle.

In general, you shouldn't specify a server certificate file or bundle using curl on macOS unless you really know what you're doing. The option is there, and is intended for backward compatibility with scripts that also have to run on other operating systems where using the Keychain is not an option.

If that does not describe your use case, then just let curl use the Keychain for trust evaluation. And if you have any self-signed certificates, then use certtool or Keychain Access to add them to your Keychain.

@nickzman nickzman closed this Dec 30, 2016
@jay
Member
jay commented Dec 31, 2016 edited

If insecure mode is specified why don't we just ignore the bundle? I can see how it might be beneficial to put --cacert FILE into a curlrc and then let's say on the command line you want to go insecure, why does it matter if the bundle was specified?

@jay jay added the SSL/TLS label Dec 31, 2016
@nickzman
Collaborator

The --cacert/--capath and --insecure options contradict each other. That would be okay on back-ends that wholly rely on a third-party certificate database, e.g. OpenSSL. But Secure Transport is intended to be used with macOS' Keychain as a certificate database, with a third-party option only in place for backward compatibility. Since they're overriding the default behavior in contradictory ways, I think it makes sense to tell the user to pick one way or the other.

We do also say in the documentation that, when Secure Transport is in use, the --cacert option is only provided for backward compatibility reasons.

@jay
Member
jay commented Dec 31, 2016

I see your point about the contradiction but I disagree. I'm saying there are reasonable scenarios where it may happen there are contradictory options. Someone may have in their script or their program an option that adds the cacert, and then later on in some other function in their program or point in their script they decide to add insecure for that transfer. AFAICT with every other backend insecure will work regardless of cacert. Is there any reason why we can't do that for Secure Transport as well, and wouldn't it also be helpful for backwards compatibility and consistency?

@fchandur

Not an issue. You get that error if you put curl into insecure mode while simultaneously specifying a server certificate file or bundle.

I am not specifying a server certificate file or bundle, not explicitly, at least. Also, I am pointing out that same version of curl, built with same configuration options, behaves differently on different OSX versions.

Here is the code that was generated.

/********* Sample code generated by the curl command line tool **********
 * All curl_easy_setopt() options are documented at:
 * https://curl.haxx.se/libcurl/c/curl_easy_setopt.html
 ************************************************************************/
#include <curl/curl.h>

int main(int argc, char *argv[])
{
  CURLcode ret;
  CURL *hnd;

  hnd = curl_easy_init();
  curl_easy_setopt(hnd, CURLOPT_URL, "https://www.google.com");
  curl_easy_setopt(hnd, CURLOPT_NOPROGRESS, 1L);
  curl_easy_setopt(hnd, CURLOPT_USERAGENT, "curl/7.52.1");
  curl_easy_setopt(hnd, CURLOPT_MAXREDIRS, 50L);
  curl_easy_setopt(hnd, CURLOPT_SSL_VERIFYPEER, 0L);
  curl_easy_setopt(hnd, CURLOPT_SSL_VERIFYHOST, 0L);
  curl_easy_setopt(hnd, CURLOPT_VERBOSE, 1L);
  curl_easy_setopt(hnd, CURLOPT_TCP_KEEPALIVE, 1L);

  /* Here is a list of options the curl code used that cannot get generated
     as source easily. You may select to either not use them or implement
     them yourself.

  CURLOPT_WRITEDATA set to a objectpointer
  CURLOPT_INTERLEAVEDATA set to a objectpointer
  CURLOPT_WRITEFUNCTION set to a functionpointer
  CURLOPT_READDATA set to a objectpointer
  CURLOPT_READFUNCTION set to a functionpointer
  CURLOPT_SEEKDATA set to a objectpointer
  CURLOPT_SEEKFUNCTION set to a functionpointer
  CURLOPT_ERRORBUFFER set to a objectpointer
  CURLOPT_STDERR set to a objectpointer
  CURLOPT_DEBUGFUNCTION set to a functionpointer
  CURLOPT_DEBUGDATA set to a objectpointer
  CURLOPT_HEADERFUNCTION set to a functionpointer
  CURLOPT_HEADERDATA set to a objectpointer

  */

  ret = curl_easy_perform(hnd);

  curl_easy_cleanup(hnd);
  hnd = NULL;

  return (int)ret;
}
/**** End of sample code ****/

@bagder
Member
bagder commented Jan 2, 2017

I don't see how @fchandur specifed an additional cacert file and I agree with @jay that if --insecure is asked for, we should be able to totally ignore all ca cert files and options since the client asked to not verify the server at all.

@bagder bagder reopened this Jan 2, 2017
@nickzman
Collaborator
nickzman commented Jan 3, 2017

Then the attached patch ought to allow the contradictory commands. Does it solve the original report?

Also, it is possible to specify the certificate bundle in the curlrc, isn't it?
allow-cafile-and-verifypeer.txt

@fchandur
fchandur commented Jan 3, 2017

It does. Thank you.

Also, when invoking the configure script with the argument --with-darwinssl, by default, the the flag --with-ca-bundle=/etc/ssl/cert.pem is also set.

I think the default, especially when using SecureTransport, should be --without-ca-bundle, no?

configure:

./configure  --with-darwinssl

relevant output:

  ....
  SSL support:      enabled (Apple OS-native)
  ...
  ca cert bundle:   /etc/ssl/cert.pem
  ca cert path:     no
  ca fallback:      no
@nickzman nickzman closed this in ffbb0f0 Jan 3, 2017
@nickzman
Collaborator
nickzman commented Jan 3, 2017

Thanks. Daniel might know more, but I think that certificate bundle is always listed there but not actually used in the ST engine. I still suspect you have something in your curlrc file that is using a certificate bundle somewhere if you're not using the --cacert option.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment