Fix error message when trying to use a self-signed HTTPS-proxy #1331

nehaljwani · 2017-03-16T19:33:59Z

Use curl to fetch google.com via https-proxy

To setup https-proxy, I configured squid with the option:
https_port 3127 cert=/etc/squid/ssl_cert/myCA.pem

Now, I try to send a request, and it tells me that it is a self-signed certificate, which is expected. There is a hint in the output, which suggests that I should use --cacert to point to the CA certificate , or use -k to override this and go ahead with an insecure connection, all of this is expected.

/t/curl-curl-7_53_1 ❯❯❯ https_proxy=https://proxy.mydomain.com:3127 ./src/curl -vvvv https://google.com
* Rebuilt URL to: https://google.com/
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to proxy.mydomain.com (127.0.0.1) port 3127 (#0)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS alert, Server hello (2):
* SSL certificate problem: self signed certificate
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, Client hello (1):
curl: (60) SSL certificate problem: self signed certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.

I follow by the book, and provide the path to cacert using the flag and curl still throws the same error at me:

/t/curl-curl-7_53_1 ❯❯❯ https_proxy=https://proxy.mydomain.com:3127 ./src/curl --cacert /tmp/myCA.pem -vvvv https://google.com
* Rebuilt URL to: https://google.com/
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to proxy.mydomain.com (127.0.0.1) port 3127 (#0)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS alert, Server hello (2):
* SSL certificate problem: self signed certificate
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, Client hello (1):
curl: (60) SSL certificate problem: self signed certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.

Then, I try to pass the --insecure flag to override the self-signed warning, but curl is still not happy and throws the same message at me:

/t/curl-curl-7_53_1 ❯❯❯ https_proxy=https://proxy.mydomain.com:3127 ./src/curl --insecure -vvvv https://google.com
* Rebuilt URL to: https://google.com/
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to proxy.mydomain.com (127.0.0.1) port 3127 (#0)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS alert, Server hello (2):
* SSL certificate problem: self signed certificate
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, Client hello (1):
curl: (60) SSL certificate problem: self signed certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.

I give up, and add the self-signed certificate to the trust store:

/e/s/ssl_cert ❯❯❯ sudo cp /etc/squid/ssl_cert/myCA.pem /etc/pki/ca-trust/source/anchors/mySquidCA.pem
/e/s/ssl_cert ❯❯❯ sudo update-ca-trust

And now curl is happy:

/t/curl-curl-7_53_1 ❯❯❯ https_proxy=https://proxy.mydomain.com:3127 ./src/curl --insecure -vvvv https://google.com
* Rebuilt URL to: https://google.com/
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to proxy.mydomain.com (127.0.0.1) port 3127 (#0)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Proxy certificate:
*  subject: C=IN; ST=Haryana; L=Gurugram; O=Default Company Ltd; CN=proxy.mydomain.com; emailAddress=no-reply@gmail.com
*  start date: Mar 16 06:43:35 2017 GMT
*  expire date: Mar 16 06:43:35 2018 GMT
*  common name: proxy.mydomain.com (matched)
*  issuer: C=IN; ST=Haryana; L=Gurugram; O=Default Company Ltd; CN=proxy.mydomain.com; emailAddress=no-reply@gmail.com
*  SSL certificate verify ok.
* Establish HTTP proxy tunnel to google.com:443
> CONNECT google.com:443 HTTP/1.1
> Host: google.com:443
> User-Agent: curl/7.53.1-DEV
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 200 Connection established
< 
* Proxy replied OK to CONNECT request
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-ECDSA-AES128-GCM-SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: C=US; ST=California; L=Mountain View; O=Google Inc; CN=*.google.com
*  start date: Mar  9 02:43:31 2017 GMT
*  expire date: Jun  1 02:20:00 2017 GMT
*  issuer: C=US; O=Google Inc; CN=Google Internet Authority G2
*  SSL certificate verify ok.
> GET / HTTP/1.1
> Host: google.com
> User-Agent: curl/7.53.1-DEV
> Accept: */*
> 
< HTTP/1.1 302 Found
< Cache-Control: private
< Content-Type: text/html; charset=UTF-8
< Location: https://www.google.co.in/?gfe_rd=cr&ei=mefKWKORJ-3s8Af2l5HQDw
< Content-Length: 262
< Date: Thu, 16 Mar 2017 19:29:29 GMT
< Alt-Svc: quic=":443"; ma=2592000; v="36,35,34"
< 
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>302 Moved</TITLE></HEAD><BODY>
<H1>302 Moved</H1>
The document has moved
<A HREF="https://www.google.co.in/?gfe_rd=cr&amp;ei=mefKWKORJ-3s8Af2l5HQDw">here</A>.
</BODY></HTML>
* Connection #0 to host proxy.mydomain.com left intact

curl/libcurl version

/t/curl-curl-7_53_1 ❯❯❯ ./src/curl --version
curl 7.53.1-DEV (x86_64-unknown-linux-gnu) libcurl/7.53.1-DEV OpenSSL/1.0.2k zlib/1.2.8 libssh2/1.8.0
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp 
Features: IPv6 Largefile NTLM NTLM_WB SSL libz UnixSockets HTTPS-proxy

operating system

Fedora 25

The text was updated successfully, but these errors were encountered:

bagder · 2017-03-16T20:04:12Z

We've explicitly worked on phrasing the man page for --insecure to make it say that it concerns server connections. You want --proxy-insecure for proxy connections. The man page section for --insecure does also say "See also --proxy-insecure" these days.

nehaljwani · 2017-03-16T20:05:52Z

Agreed, but if that's the case, shouldn't the error message be corrected to state that, instead of the misleading option?

bagder · 2017-03-16T20:06:23Z

It should!

jay · 2017-03-16T20:11:30Z

Just to clear this up so we're all on the same page, when you use https_proxy environment variable then libcurl will use that as a proxy for https transfers only. In other words

https_proxy=https://foo:443 curl https://google.com # curl uses https_proxy env var
https_proxy=https://foo:443 curl http://google.com # curl doesn't use https_proxy env var

So is that what you are trying to do, use https_proxy only for https URLs?

Also we added an error message for this to help, see #1258 and e1187c4. I did not realize though it doesn't handle https proxies set via environment variables. We could show that error all the time, I guess.

nehaljwani · 2017-03-16T20:13:56Z

@jay We are indeed on the same page. I wanted to raise this issue only to correct the output message. That's all. Rest works as expected.

nehaljwani · 2017-03-16T20:17:28Z

To answer your other question, no, I am not trying to use https_proxy only for https URLs. That was just an example. The error message is still misleading if I do:

http_proxy=https://proxy.mydomain.com:3127 ./src/curl -vvvv http://google.com

jay · 2017-03-16T20:30:43Z

Well as I said it's supposed to show "HTTPS-proxy has similar options --proxy-cacert and --proxy-insecure" it's just not doing that because I didn't think of this scenario.

In order to detect this I would have to add a CURLINFO to get the proxy that was used for the connection, which may be useful in its own right. A simpler alternative would be just make the message appear all the time.

For example the extra line could show always:

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.
HTTPS-proxy has similar options --proxy-cacert and --proxy-insecure.

diff --git a/src/tool_operate.c b/src/tool_operate.c
index 8f76715..ff533ea 100644
--- a/src/tool_operate.c
+++ b/src/tool_operate.c
@@ -109,6 +109,9 @@ CURLcode curl_easy_perform_ev(CURL *easy);
   "If you'd like to turn off curl's verification of the certificate, use\n" \
   " the -k (or --insecure) option.\n"
 
+#define CURL_CA_CERT_ERRORMSG3                                              \
+  "HTTPS-proxy has similar options --proxy-cacert and --proxy-insecure.\n"
+
 static bool is_fatal_error(CURLcode code)
 {
   switch(code) {
@@ -1695,13 +1698,8 @@ static CURLcode operate_do(struct GlobalConfig *global,
           fprintf(global->errors, "curl: (%d) %s\n", result, (errorbuffer[0]) ?
                   errorbuffer : curl_easy_strerror(result));
           if(result == CURLE_SSL_CACERT)
-            fprintf(global->errors, "%s%s%s",
-                    CURL_CA_CERT_ERRORMSG1, CURL_CA_CERT_ERRORMSG2,
-                    ((config->proxy &&
-                      curl_strnequal(config->proxy, "https://", 8)) ?
-                     "HTTPS proxy has similar options --proxy-cacert "
-                     "and --proxy-insecure.\n" :
-                     ""));
+            fprintf(global->errors, "%s%s%s", CURL_CA_CERT_ERRORMSG1,
+                    CURL_CA_CERT_ERRORMSG2, CURL_CA_CERT_ERRORMSG3);
         }
 
         /* Fall through comment to 'quit_urls' label */

nehaljwani · 2017-03-16T20:35:21Z

IMO, from a developer's point of view, detection would be awesome! But from a user's point of view, having the additional message print always is better than having nothing or a misleading message.

jay · 2017-03-16T20:48:48Z

Here is a simpler change that will only show it if libcurl was built with HTTPS-proxy support:

diff --git a/src/tool_operate.c b/src/tool_operate.c
index 8f76715..572c8d0 100644
--- a/src/tool_operate.c
+++ b/src/tool_operate.c
@@ -1697,9 +1697,8 @@ static CURLcode operate_do(struct GlobalConfig *global,
           if(result == CURLE_SSL_CACERT)
             fprintf(global->errors, "%s%s%s",
                     CURL_CA_CERT_ERRORMSG1, CURL_CA_CERT_ERRORMSG2,
-                    ((config->proxy &&
-                      curl_strnequal(config->proxy, "https://", 8)) ?
-                     "HTTPS proxy has similar options --proxy-cacert "
+                    ((curlinfo->features & CURL_VERSION_HTTPS_PROXY) ?
+                     "HTTPS-proxy has similar options --proxy-cacert "
                      "and --proxy-insecure.\n" :
                      ""));
         }

If I can get a +1 I'll land it

bagder · 2017-03-16T21:35:27Z

I'm 👍 on that!

- Show the HTTPS-proxy options on CURLE_SSL_CACERT if libcurl was built with HTTPS-proxy support. Prior to this change those options were shown only if an HTTPS-proxy was specified by --proxy, but that did not take into account environment variables such as http_proxy, https_proxy, etc. Follow-up to e1187c4. Bug: #1331 Reported-by: Nehal J Wani

jay · 2017-03-16T22:27:42Z

Thanks guys, landed in 98afec0.

nehaljwani changed the title ~~The flags --insecure or --cacert do not work with HTTPS-proxy~~ Fix error message when trying to use a self-signed HTTPS-proxy Mar 16, 2017

jay added cmdline tool connecting & proxies TLS labels Mar 16, 2017

jay closed this as completed Mar 16, 2017

lock bot locked as resolved and limited conversation to collaborators May 6, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix error message when trying to use a self-signed HTTPS-proxy #1331

Fix error message when trying to use a self-signed HTTPS-proxy #1331

nehaljwani commented Mar 16, 2017

bagder commented Mar 16, 2017

nehaljwani commented Mar 16, 2017

bagder commented Mar 16, 2017

jay commented Mar 16, 2017

nehaljwani commented Mar 16, 2017

nehaljwani commented Mar 16, 2017

jay commented Mar 16, 2017

nehaljwani commented Mar 16, 2017 •

edited

Loading

jay commented Mar 16, 2017

bagder commented Mar 16, 2017

jay commented Mar 16, 2017

Fix error message when trying to use a self-signed HTTPS-proxy #1331

Fix error message when trying to use a self-signed HTTPS-proxy #1331

Comments

nehaljwani commented Mar 16, 2017

Use curl to fetch google.com via https-proxy

curl/libcurl version

operating system

bagder commented Mar 16, 2017

nehaljwani commented Mar 16, 2017

bagder commented Mar 16, 2017

jay commented Mar 16, 2017

nehaljwani commented Mar 16, 2017

nehaljwani commented Mar 16, 2017

jay commented Mar 16, 2017

nehaljwani commented Mar 16, 2017 • edited Loading

jay commented Mar 16, 2017

bagder commented Mar 16, 2017

jay commented Mar 16, 2017

nehaljwani commented Mar 16, 2017 •

edited

Loading