Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support for proxying HTTP to HTTPS #175

Closed
mmollaverdi opened this issue Apr 30, 2017 · 8 comments
Closed

Support for proxying HTTP to HTTPS #175

mmollaverdi opened this issue Apr 30, 2017 · 8 comments

Comments

@mmollaverdi
Copy link

Problem

I have some upstream APIs which run over https. Using toxyproxy to proxy https to https works fine, except that the local toxiproxy instances won't be responding with a valid SSL certificate, so over curl for example, I need to use the -k option:

$ toxiproxy-cli create https-to-https -l :9985 -u www.google.com:443
Created new proxy https-to-https

$ curl -H "Host: www.google.com" https://localhost:9985/
curl: (60) SSL certificate problem: Invalid certificate chain
More details here: https://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.

$ curl -H "Host: www.google.com" -k https://localhost:9985/
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>302 Moved</TITLE></HEAD><BODY>
<H1>302 Moved</H1>
The document has moved
<A HREF="https://www.google.com.au/?gfe_rd=cr&amp;ei=L1oFWaKgIZLp8we91LSwCw">here</A>.
</BODY></HTML>

This is hard to get around specially if you're using toxi-proxy to test the interactions between different APIs. The downstream APIs which talk to upstream dependencies through toxiproxy, need to disable SSL certificate verification. This is done differently depending on which programming language and which library that API uses.

Suggestion

My suggestion is to implement support for proxying https requests through http, i.e. a toxi-proxy proxy can listens over http, but communicates to upstream over https. One way to configure such proxies would be to include the desired protocol in both listen and upstream values, but there are probably other ways to achieve this too.

toxiproxy-cli create http-to-https -l http://localhost:9985 -u https://www.google.com:443

Note the inclusion of http in -l argument and https in -u above.
This doesn't seem to be supported currently.

Thoughts abut adding support for this or alternative solutions which would achieve this?
@jpittis
Copy link
Contributor

jpittis commented May 15, 2017

Hey there, thanks for opening an issue. Sorry for the delayed reply.

I believe the culprit in the situation described is the curl HTTPS client. When doing certificate verification, curl needs to compare the host it's querying against the host in the certificate received. I assume curl is provided with an IP address and does not know that it is querying google through a proxy. This is why adding -k fixes the error which demonstrates the error is with the client rather than the server.

Recreation / Example Solution

screen shot 2017-05-15 at 8 41 52 am

If I add www.google.com to my /etc/hosts file so that it resolves to 127.0.0.1, querying google from curl will now query the proxy instead.

screen shot 2017-05-15 at 8 41 00 am

screen shot 2017-05-15 at 8 40 30 am

Notice that this time, even though the HTTPS request is indeed going through toxiproxy, the certificate is being verified properly. I believe this demonstrates that with a correctly configured client, HTTPS works fine through toxiproxy.

I don't know if curl has an flag to set the host being used for certificate validation.

@jpittis jpittis mentioned this issue May 15, 2017
Closed
@dentarg
Copy link

dentarg commented May 23, 2017

I don't know if curl has an flag to set the host being used for certificate validation.

This seems to work for your setup above, but without any change to /etc/hosts

curl -s -v --resolve 'www.google.com:6677:127.0.0.1' https://www.google.com:6677

https://curl.haxx.se/docs/manpage.html#--resolve

@jpittis
Copy link
Contributor

jpittis commented May 23, 2017

Awesome thanks @dentarg. I'm closing this issue because I believe it's been resolved by the above comment. Please feel free to reopen it if you disagree @mmollaverdi.

@jpittis jpittis closed this as completed May 23, 2017
@sirupsen
Copy link
Contributor

@jpittis do you want to add a note to the README in the FAQ?

@noahmatisoff
Copy link

I'm trying to do something similarly, but with two important differences:

  1. The request URL has a path in it
  2. I'm not using cURL -- I'm using Faraday in Ruby.

Struggling to figure out how to do both here. Any insight?

@xthexder
Copy link
Contributor

xthexder commented May 11, 2018 •
edited
Loading

Toxiproxy doesn't know anything about the data it passes through the proxy. It is not protocol specific.

If you want to proxy https, you need to either have the request hostname match the proxy target somehow (like with /etc/hosts or --resolve), or disable certificate verification. This isn't secure, but for local proxies that shouldn't matter.
There's an example of how to do this in Faraday here

Another option would be to put something like nginx, or another http-specific proxy in front of the target server to handle ssl for you. That way all proxied connections can run on http instead.

@xthexder xthexder mentioned this issue Jun 27, 2018
Closed
@xthexder xthexder mentioned this issue Aug 10, 2018
Open
@ahmedtalhakhan ahmedtalhakhan mentioned this issue Dec 4, 2019
Open
@carpeliam
Copy link

carpeliam commented Jul 22, 2020 •
edited
Loading

screen shot 2017-05-15 at 8 41 52 am

If I add www.google.com to my /etc/hosts file so that it resolves to 127.0.0.1, querying google from curl will now query the proxy instead.

screen shot 2017-05-15 at 8 41 00 am

@jpittis thanks for this reply, we were able to duplicate the same success you showed for google.com and other popular websites. However, we're running into one issue with a website (not publicly accessible) where we do the exact same thing as you're showing, but we get an Unknown SSL protocol error from curl. Can you think of any reason why that might be the case, when all we've changed is the URL from your example? Here are the verbose curl logs:

$ curl --verbose -H "Host: our.special.website.com" https://our.special.website.com:6677
* Rebuilt URL to: https://our.special.website.com:6677/
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to our.special.website.com (127.0.0.1) port 6677 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* Unknown SSL protocol error in connection to our.special.website.com:6677
* Curl_http_done: called premature == 0
* Closing connection 0
curl: (35) Unknown SSL protocol error in connection to our.special.website.com:6677

@shashankjhs
Copy link

@carpeliam were you able find find solution to this https issue

@dliappis dliappis mentioned this issue Mar 24, 2022
Merged
@albcm-simetrik albcm-simetrik mentioned this issue Nov 22, 2023
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
8 participants
@carpeliam @dentarg @sirupsen @xthexder @mmollaverdi @jpittis @noahmatisoff @shashankjhs