Add support for proxying HTTPS server connections on the client side #192

Closed
ktoso opened this Issue Sep 8, 2016 · 38 comments

Comments

Projects
None yet
@ktoso
Member

ktoso commented Sep 8, 2016

Issue by sirthias
Monday Oct 27, 2014 at 15:52 GMT
Originally opened as akka/akka#16153


Required for one connection:

  1. Establish plain-text connection to proxy
  2. Send plain-text CONNECT target.host:443 HTTP/1.1 request to the proxy
  3. Wait for 2xx response (afterwards all communication on the connection will be directly tunneled through to the target host)
  4. Start SSL handshake, i.e. upgrade the connection to TLS/SSL
  5. Send the actual requests, read actual responses

This feature should likely be implemented on the level of host-level client-side API as in most cases you'll want a managed connection pool to the proxy.

Some pointers:

@ktoso ktoso added this to the http-backlog milestone Sep 8, 2016

@ktoso

This comment has been minimized.

Show comment
Hide comment
@ktoso

ktoso Sep 8, 2016

Member

Comment by sirthias
Monday Oct 27, 2014 at 15:55 GMT


/cc @jrudolph

Member

ktoso commented Sep 8, 2016

Comment by sirthias
Monday Oct 27, 2014 at 15:55 GMT


/cc @jrudolph

@ktoso

This comment has been minimized.

Show comment
Hide comment
@ktoso

ktoso Sep 8, 2016

Member

Comment by mackler
Tuesday Oct 28, 2014 at 19:41 GMT


+1

Member

ktoso commented Sep 8, 2016

Comment by mackler
Tuesday Oct 28, 2014 at 19:41 GMT


+1

@ktoso

This comment has been minimized.

Show comment
Hide comment
@ktoso

ktoso Sep 8, 2016

Member

Comment by mlenner
Monday Nov 03, 2014 at 15:03 GMT


+1

Member

ktoso commented Sep 8, 2016

Comment by mlenner
Monday Nov 03, 2014 at 15:03 GMT


+1

@ktoso

This comment has been minimized.

Show comment
Hide comment
@ktoso

ktoso Sep 8, 2016

Member

Comment by salanki
Tuesday Mar 31, 2015 at 21:59 GMT


+1

Member

ktoso commented Sep 8, 2016

Comment by salanki
Tuesday Mar 31, 2015 at 21:59 GMT


+1

@ktoso

This comment has been minimized.

Show comment
Hide comment
@ktoso

ktoso Sep 8, 2016

Member

Comment by jkew
Tuesday Apr 07, 2015 at 22:21 GMT


+1

Member

ktoso commented Sep 8, 2016

Comment by jkew
Tuesday Apr 07, 2015 at 22:21 GMT


+1

@ktoso

This comment has been minimized.

Show comment
Hide comment
@ktoso

ktoso Sep 8, 2016

Member

Comment by rikardNL
Tuesday May 26, 2015 at 05:33 GMT


+1

Member

ktoso commented Sep 8, 2016

Comment by rikardNL
Tuesday May 26, 2015 at 05:33 GMT


+1

@ktoso

This comment has been minimized.

Show comment
Hide comment
@ktoso

ktoso Sep 8, 2016

Member

Comment by gonstr
Thursday May 28, 2015 at 11:47 GMT


+1

Member

ktoso commented Sep 8, 2016

Comment by gonstr
Thursday May 28, 2015 at 11:47 GMT


+1

@ktoso

This comment has been minimized.

Show comment
Hide comment
@ktoso

ktoso Sep 8, 2016

Member

Comment by bdwashbu
Wednesday Jul 15, 2015 at 15:01 GMT


+1

Member

ktoso commented Sep 8, 2016

Comment by bdwashbu
Wednesday Jul 15, 2015 at 15:01 GMT


+1

@ktoso

This comment has been minimized.

Show comment
Hide comment
@ktoso

ktoso Sep 8, 2016

Member

Comment by ktoso
Wednesday Jul 15, 2015 at 21:10 GMT


Relates to akka/akka#17976

Member

ktoso commented Sep 8, 2016

Comment by ktoso
Wednesday Jul 15, 2015 at 21:10 GMT


Relates to akka/akka#17976

@ktoso

This comment has been minimized.

Show comment
Hide comment
@ktoso

ktoso Sep 8, 2016

Member

Comment by nilsga
Tuesday Oct 27, 2015 at 07:52 GMT


+1

Member

ktoso commented Sep 8, 2016

Comment by nilsga
Tuesday Oct 27, 2015 at 07:52 GMT


+1

@ktoso

This comment has been minimized.

Show comment
Hide comment
@ktoso

ktoso Sep 8, 2016

Member

Comment by carl297r
Thursday Nov 05, 2015 at 02:22 GMT


+1

Member

ktoso commented Sep 8, 2016

Comment by carl297r
Thursday Nov 05, 2015 at 02:22 GMT


+1

@ktoso

This comment has been minimized.

Show comment
Hide comment
@ktoso

ktoso Sep 8, 2016

Member

Comment by ktoso
Thursday Nov 12, 2015 at 00:41 GMT


Logging a "very big" +1 here, we may want to think how and when we could address this.

Member

ktoso commented Sep 8, 2016

Comment by ktoso
Thursday Nov 12, 2015 at 00:41 GMT


Logging a "very big" +1 here, we may want to think how and when we could address this.

@ktoso

This comment has been minimized.

Show comment
Hide comment
@ktoso

ktoso Sep 8, 2016

Member

Comment by ashugupt
Monday Dec 28, 2015 at 09:01 GMT


+1

Member

ktoso commented Sep 8, 2016

Comment by ashugupt
Monday Dec 28, 2015 at 09:01 GMT


+1

@ktoso

This comment has been minimized.

Show comment
Hide comment
@ktoso

ktoso Sep 8, 2016

Member

Comment by Madder
Monday Dec 28, 2015 at 13:56 GMT


+1

Member

ktoso commented Sep 8, 2016

Comment by Madder
Monday Dec 28, 2015 at 13:56 GMT


+1

@ktoso

This comment has been minimized.

Show comment
Hide comment
@ktoso

ktoso Sep 8, 2016

Member

Comment by mirelon
Thursday Feb 04, 2016 at 14:16 GMT


+1

Member

ktoso commented Sep 8, 2016

Comment by mirelon
Thursday Feb 04, 2016 at 14:16 GMT


+1

@ktoso

This comment has been minimized.

Show comment
Hide comment
@ktoso

ktoso Sep 8, 2016

Member

Comment by ktoso
Wednesday Feb 10, 2016 at 11:05 GMT


Logging another +1 (for my reference, sorry for the noise)

Member

ktoso commented Sep 8, 2016

Comment by ktoso
Wednesday Feb 10, 2016 at 11:05 GMT


Logging another +1 (for my reference, sorry for the noise)

@ktoso

This comment has been minimized.

Show comment
Hide comment
@ktoso

ktoso Sep 8, 2016

Member

Comment by andrasp3a
Wednesday Mar 30, 2016 at 09:43 GMT


+1

Member

ktoso commented Sep 8, 2016

Comment by andrasp3a
Wednesday Mar 30, 2016 at 09:43 GMT


+1

@ktoso

This comment has been minimized.

Show comment
Hide comment
@ktoso

ktoso Sep 8, 2016

Member

Comment by rahulsinghai
Thursday Mar 31, 2016 at 15:44 GMT


+1

Member

ktoso commented Sep 8, 2016

Comment by rahulsinghai
Thursday Mar 31, 2016 at 15:44 GMT


+1

@ktoso

This comment has been minimized.

Show comment
Hide comment
@ktoso

ktoso Sep 8, 2016

Member

Comment by mlangc
Wednesday Apr 20, 2016 at 07:27 GMT


+1

Member

ktoso commented Sep 8, 2016

Comment by mlangc
Wednesday Apr 20, 2016 at 07:27 GMT


+1

@ktoso

This comment has been minimized.

Show comment
Hide comment
@ktoso

ktoso Sep 8, 2016

Member

Comment by lregnier
Friday Apr 29, 2016 at 14:49 GMT


+1

Member

ktoso commented Sep 8, 2016

Comment by lregnier
Friday Apr 29, 2016 at 14:49 GMT


+1

@ktoso

This comment has been minimized.

Show comment
Hide comment
@ktoso

ktoso Sep 8, 2016

Member

Comment by douglaz
Wednesday Jun 29, 2016 at 14:43 GMT


+1000

Member

ktoso commented Sep 8, 2016

Comment by douglaz
Wednesday Jun 29, 2016 at 14:43 GMT


+1000

@ktoso

This comment has been minimized.

Show comment
Hide comment
@ktoso

ktoso Sep 8, 2016

Member

Comment by ktoso
Wednesday Jun 29, 2016 at 14:44 GMT


Akka team won't be able (time wise) to pick up this issue in the short-term, so we'd like to encourage you to try to contribute this feature, or contact us if you'd like sponsor its development.

Member

ktoso commented Sep 8, 2016

Comment by ktoso
Wednesday Jun 29, 2016 at 14:44 GMT


Akka team won't be able (time wise) to pick up this issue in the short-term, so we'd like to encourage you to try to contribute this feature, or contact us if you'd like sponsor its development.

@ktoso

This comment has been minimized.

Show comment
Hide comment
@ktoso

ktoso Sep 8, 2016

Member

Comment by nanothermite
Wednesday Aug 24, 2016 at 17:46 GMT


+1

Member

ktoso commented Sep 8, 2016

Comment by nanothermite
Wednesday Aug 24, 2016 at 17:46 GMT


+1

@ktoso

This comment has been minimized.

Show comment
Hide comment
@ktoso

ktoso Sep 8, 2016

Member

Comment by greenhost87
Thursday Sep 01, 2016 at 20:43 GMT


+1

Member

ktoso commented Sep 8, 2016

Comment by greenhost87
Thursday Sep 01, 2016 at 20:43 GMT


+1

@ktoso

This comment has been minimized.

Show comment
Hide comment
@ktoso

ktoso Sep 8, 2016

Member

Comment by nemccarthy
Thursday Sep 08, 2016 at 02:07 GMT


+1

Member

ktoso commented Sep 8, 2016

Comment by nemccarthy
Thursday Sep 08, 2016 at 02:07 GMT


+1

@ktoso

This comment has been minimized.

Show comment
Hide comment
@ktoso

ktoso Sep 8, 2016

Member

Comment by Yeitijem
Thursday Sep 08, 2016 at 12:50 GMT


+1

Member

ktoso commented Sep 8, 2016

Comment by Yeitijem
Thursday Sep 08, 2016 at 12:50 GMT


+1

@ktoso ktoso added the 1 - triaged label Sep 8, 2016

@hensg

This comment has been minimized.

Show comment
Hide comment

hensg commented Oct 31, 2016

+1

@jesinity

This comment has been minimized.

Show comment
Hide comment

+1

@kstokoz

This comment has been minimized.

Show comment
Hide comment
@kstokoz

kstokoz Dec 9, 2016

Contributor

+1

Contributor

kstokoz commented Dec 9, 2016

+1

@rklaehn

This comment has been minimized.

Show comment
Hide comment
@rklaehn

rklaehn Dec 10, 2016

Any update on this? Every single corporate firewall on the planet requires you to connect to the outside world via an https proxy, so not having this makes the akka-http client functionality pretty much useless.

rklaehn commented Dec 10, 2016

Any update on this? Every single corporate firewall on the planet requires you to connect to the outside world via an https proxy, so not having this makes the akka-http client functionality pretty much useless.

@mcamou

This comment has been minimized.

Show comment
Hide comment
@mcamou

mcamou Dec 10, 2016

Contributor

+1

Contributor

mcamou commented Dec 10, 2016

+1

@johanandren

This comment has been minimized.

Show comment
Hide comment
@johanandren

johanandren Dec 12, 2016

Member

Note that a PR is worth more than a million +1s. (Also please use the reaction button instead of spamming this ticket with +1 comments, thanks!)

Member

johanandren commented Dec 12, 2016

Note that a PR is worth more than a million +1s. (Also please use the reaction button instead of spamming this ticket with +1 comments, thanks!)

@alvarow

This comment has been minimized.

Show comment
Hide comment
@alvarow

alvarow Dec 12, 2016

+1, but if the +1 is not enough to express interest, I can always write: PLEASE FIX THIS, I AM INTERESTED IN SEEING THIS FIXED.

alvarow commented Dec 12, 2016

+1, but if the +1 is not enough to express interest, I can always write: PLEASE FIX THIS, I AM INTERESTED IN SEEING THIS FIXED.

@note

This comment has been minimized.

Show comment
Hide comment
@note

note Mar 2, 2017

Contributor

I was playing around this issue and managed to complete HTTPS request via proxy. Here is my code: https://github.com/akka/akka-http/compare/master...note:192-https-proxy?expand=1. It's not ready to be a PR, there is still a lot of work to do, I am mostly sharing it for early validation. So the solution boils down to having additional GraphStage (I called it ProxyGraphStage) between tlsStage and transportFlow. Its only goal is to send CONNECT, wait for OK answer and then simply forward all messages untouched. In future there should be correct error handling and so on. @jrudolph Does such approach makes sense? If it makes sense I will work further on this and prepare a PR.

Contributor

note commented Mar 2, 2017

I was playing around this issue and managed to complete HTTPS request via proxy. Here is my code: https://github.com/akka/akka-http/compare/master...note:192-https-proxy?expand=1. It's not ready to be a PR, there is still a lot of work to do, I am mostly sharing it for early validation. So the solution boils down to having additional GraphStage (I called it ProxyGraphStage) between tlsStage and transportFlow. Its only goal is to send CONNECT, wait for OK answer and then simply forward all messages untouched. In future there should be correct error handling and so on. @jrudolph Does such approach makes sense? If it makes sense I will work further on this and prepare a PR.

@jrudolph

This comment has been minimized.

Show comment
Hide comment
@jrudolph

jrudolph Mar 2, 2017

Member

Great work, @note for taking a stab at it. I just opened another PR yesterday which could be used as groundwork for it: it allows to redefine the transport the pool (or a single client connection) uses to access a host. The only predefined transport so far would be the existing TCP transport. Your HTTPS proxy support could be an HttpsProxyTransport, then we could have a SOCKS transport etc.

The basic change you would need to do is to include your underlying TCP connection to the HTTPS proxy into your code. Apart from that no changes to the existing http-core infrastructure would be necessary. WDYT?

It certainly looks promising, so please open a PR. I'll try to merge my PR this week so you can rebase on top of that.

Member

jrudolph commented Mar 2, 2017

Great work, @note for taking a stab at it. I just opened another PR yesterday which could be used as groundwork for it: it allows to redefine the transport the pool (or a single client connection) uses to access a host. The only predefined transport so far would be the existing TCP transport. Your HTTPS proxy support could be an HttpsProxyTransport, then we could have a SOCKS transport etc.

The basic change you would need to do is to include your underlying TCP connection to the HTTPS proxy into your code. Apart from that no changes to the existing http-core infrastructure would be necessary. WDYT?

It certainly looks promising, so please open a PR. I'll try to merge my PR this week so you can rebase on top of that.

@ktoso

This comment has been minimized.

Show comment
Hide comment
@ktoso

ktoso Mar 2, 2017

Member

For reference, the pluggable transport PR #917

Member

ktoso commented Mar 2, 2017

For reference, the pluggable transport PR #917

@note

This comment has been minimized.

Show comment
Hide comment
@note

note Mar 2, 2017

Contributor

@jrudolph Thanks for your feedback, your PR seems very helpful - with that I think there's a great chance I will be able to implement Https proxy without touching Http.scala at all. Will continue to work on this at latest on Monday.

Contributor

note commented Mar 2, 2017

@jrudolph Thanks for your feedback, your PR seems very helpful - with that I think there's a great chance I will be able to implement Https proxy without touching Http.scala at all. Will continue to work on this at latest on Monday.

@jrudolph

This comment has been minimized.

Show comment
Hide comment
@jrudolph

jrudolph Mar 2, 2017

Member

Great, thanks, @note. We (and lots of other people) are looking forward to it.

Member

jrudolph commented Mar 2, 2017

Great, thanks, @note. We (and lots of other people) are looking forward to it.

jrudolph added a commit to jrudolph/akka-http that referenced this issue Jun 14, 2017

jrudolph added a commit to jrudolph/akka-http that referenced this issue Jun 14, 2017

jrudolph added a commit to jrudolph/akka-http that referenced this issue Jun 14, 2017

jrudolph added a commit to jrudolph/akka-http that referenced this issue Jun 14, 2017

jrudolph added a commit to jrudolph/akka-http that referenced this issue Jun 14, 2017

ktoso added a commit that referenced this issue Jun 17, 2017

ktoso added a commit that referenced this issue Jun 17, 2017

@2m 2m modified the milestones: 10.0.8, backlog Jun 20, 2017

@2m 2m closed this Jun 20, 2017

@ktoso ktoso removed the 1 - triaged label Jun 20, 2017

tomrf1 added a commit to tomrf1/akka-http that referenced this issue Aug 13, 2017

tomrf1 added a commit to tomrf1/akka-http that referenced this issue Aug 13, 2017

tomrf1 added a commit to tomrf1/akka-http that referenced this issue Aug 13, 2017

tomrf1 added a commit to tomrf1/akka-http that referenced this issue Aug 13, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment