Add support for HTTP proxy servers on the client side #115

Open
akka-ci opened this Issue Sep 8, 2016 · 25 comments

Comments

Projects
None yet
@akka-ci
Collaborator

akka-ci commented Sep 8, 2016

Issue by jrudolph
Wednesday Feb 11, 2015 at 13:48 GMT
Originally opened as akka/akka#16853


In spray, there's support for connecting to HTTP servers through an HTTP proxy (spray/spray#102). This needs to be implemented for akka-http as well.

An important part of this is how to provide the configuration which requests to proxy and where to get the proxy configuration from. Spray allows configuration in the config file but also tries to pick up the well-known Java properties.

There's another ticket to enable proxying of https connections (#16153).

This is part of the bigger initiative to support a high-level HTTP client interface as tracked as #16856.

/cc @sirthias

@akka-ci akka-ci added this to the http-backlog milestone Sep 8, 2016

@akka-ci

This comment has been minimized.

Show comment
Hide comment
@akka-ci

akka-ci Sep 8, 2016

Collaborator

Comment by RichardBradley
Wednesday Sep 09, 2015 at 13:56 GMT


(+1 from me. This can be very helpful in development, as it lets you pass Akka HTTP traffic through debugging proxies like Fiddler which is easier to play with than WireShark.)

Collaborator

akka-ci commented Sep 8, 2016

Comment by RichardBradley
Wednesday Sep 09, 2015 at 13:56 GMT


(+1 from me. This can be very helpful in development, as it lets you pass Akka HTTP traffic through debugging proxies like Fiddler which is easier to play with than WireShark.)

@akka-ci

This comment has been minimized.

Show comment
Hide comment
@akka-ci

akka-ci Sep 8, 2016

Collaborator

Comment by tjdett
Friday Oct 09, 2015 at 05:40 GMT


👍. I wondered why, while porting my app from Spray to Akka-HTTP, my Betamax tests stopped working.

Respecting system properties http.proxyHost et al is rather important.

Collaborator

akka-ci commented Sep 8, 2016

Comment by tjdett
Friday Oct 09, 2015 at 05:40 GMT


👍. I wondered why, while porting my app from Spray to Akka-HTTP, my Betamax tests stopped working.

Respecting system properties http.proxyHost et al is rather important.

@akka-ci

This comment has been minimized.

Show comment
Hide comment
@akka-ci

akka-ci Sep 8, 2016

Collaborator

Comment by rkuhn
Tuesday Oct 20, 2015 at 08:44 GMT


What’s the status here, if this was in Spray then it should be in http:next, no?

Collaborator

akka-ci commented Sep 8, 2016

Comment by rkuhn
Tuesday Oct 20, 2015 at 08:44 GMT


What’s the status here, if this was in Spray then it should be in http:next, no?

@akka-ci

This comment has been minimized.

Show comment
Hide comment
@akka-ci

akka-ci Sep 8, 2016

Collaborator

Comment by jrudolph
Tuesday Oct 20, 2015 at 09:01 GMT


IIRC in February, we decided together to remove it from next.

Collaborator

akka-ci commented Sep 8, 2016

Comment by jrudolph
Tuesday Oct 20, 2015 at 09:01 GMT


IIRC in February, we decided together to remove it from next.

@akka-ci

This comment has been minimized.

Show comment
Hide comment
@akka-ci

akka-ci Sep 8, 2016

Collaborator

Comment by sirthias
Tuesday Oct 20, 2015 at 09:04 GMT


Also important to remember: spray only supports non-encrypted client-side proxying, which is loosing importance quickly. So this ticket really should be tackled together with #16153.

Collaborator

akka-ci commented Sep 8, 2016

Comment by sirthias
Tuesday Oct 20, 2015 at 09:04 GMT


Also important to remember: spray only supports non-encrypted client-side proxying, which is loosing importance quickly. So this ticket really should be tackled together with #16153.

@akka-ci

This comment has been minimized.

Show comment
Hide comment
@akka-ci

akka-ci Sep 8, 2016

Collaborator

Comment by rkuhn
Tuesday Oct 20, 2015 at 09:42 GMT


Ah, right. Now we have all the needed context on this ticket :-)

Collaborator

akka-ci commented Sep 8, 2016

Comment by rkuhn
Tuesday Oct 20, 2015 at 09:42 GMT


Ah, right. Now we have all the needed context on this ticket :-)

@akka-ci

This comment has been minimized.

Show comment
Hide comment
@akka-ci

akka-ci Sep 8, 2016

Collaborator

Comment by ktoso
Tuesday Jan 05, 2016 at 13:51 GMT


Another instance of need for it: ☝️ January 5, 2016 1:28 PM

Spray did have it so it's tricky to claim feature parity (for non-encrypted) hm...

Collaborator

akka-ci commented Sep 8, 2016

Comment by ktoso
Tuesday Jan 05, 2016 at 13:51 GMT


Another instance of need for it: ☝️ January 5, 2016 1:28 PM

Spray did have it so it's tricky to claim feature parity (for non-encrypted) hm...

@akka-ci

This comment has been minimized.

Show comment
Hide comment
@akka-ci

akka-ci Sep 8, 2016

Collaborator

Comment by weda
Wednesday Feb 10, 2016 at 19:00 GMT


Hi,

We try to make corporate world interested with akka http, but it really resides behind the proxy server. Is there a chance that this functionality will be released in next future .-)

Collaborator

akka-ci commented Sep 8, 2016

Comment by weda
Wednesday Feb 10, 2016 at 19:00 GMT


Hi,

We try to make corporate world interested with akka http, but it really resides behind the proxy server. Is there a chance that this functionality will be released in next future .-)

@akka-ci

This comment has been minimized.

Show comment
Hide comment
@akka-ci

akka-ci Sep 8, 2016

Collaborator

Comment by AlexGilleran
Thursday Aug 25, 2016 at 08:51 GMT


Could the config for this be removed or at least the lack of the feature noted in the docs? (http://doc.akka.io/docs/akka/2.4.9/scala/http/configuration.html). Just spent a very confused half-hour trying to set up a proxy only to come to github and find it isn't implemented at all :(.

Collaborator

akka-ci commented Sep 8, 2016

Comment by AlexGilleran
Thursday Aug 25, 2016 at 08:51 GMT


Could the config for this be removed or at least the lack of the feature noted in the docs? (http://doc.akka.io/docs/akka/2.4.9/scala/http/configuration.html). Just spent a very confused half-hour trying to set up a proxy only to come to github and find it isn't implemented at all :(.

@akka-ci

This comment has been minimized.

Show comment
Hide comment
@akka-ci

akka-ci Sep 8, 2016

Collaborator

Comment by ktoso
Thursday Aug 25, 2016 at 11:12 GMT


Good point. Done.

Again, any help on getting this done is very welcome.

Collaborator

akka-ci commented Sep 8, 2016

Comment by ktoso
Thursday Aug 25, 2016 at 11:12 GMT


Good point. Done.

Again, any help on getting this done is very welcome.

@akka-ci

This comment has been minimized.

Show comment
Hide comment
@akka-ci

akka-ci Sep 8, 2016

Collaborator

Comment by nemccarthy
Thursday Sep 08, 2016 at 02:08 GMT


+1 this is a real killer, we'd love to use it but without proxy support we are back to spray and even that doesnt support ssl :(

Collaborator

akka-ci commented Sep 8, 2016

Comment by nemccarthy
Thursday Sep 08, 2016 at 02:08 GMT


+1 this is a real killer, we'd love to use it but without proxy support we are back to spray and even that doesnt support ssl :(

@jypma

This comment has been minimized.

Show comment
Hide comment
@jypma

jypma Nov 8, 2016

Member

@ktoso proxy configuration values are still there: https://github.com/akka/akka-http/blob/master/akka-http-core/src/main/resources/reference.conf#L236 . Had me going for a while as well. Testing akka/alpakka#24 may actually come to depend on the need to do proxying in akka-http...

What else would need to be done besides sending all traffic to the proxy IP instead of the real one, and always using HTTP/1/1 with a Host: header?

Member

jypma commented Nov 8, 2016

@ktoso proxy configuration values are still there: https://github.com/akka/akka-http/blob/master/akka-http-core/src/main/resources/reference.conf#L236 . Had me going for a while as well. Testing akka/alpakka#24 may actually come to depend on the need to do proxying in akka-http...

What else would need to be done besides sending all traffic to the proxy IP instead of the real one, and always using HTTP/1/1 with a Host: header?

@jypma

This comment has been minimized.

Show comment
Hide comment
@jypma

jypma Nov 9, 2016

Member

Duplicate of #89 . @jrudolph merge tickets?

Member

jypma commented Nov 9, 2016

Duplicate of #89 . @jrudolph merge tickets?

@jypma jypma referenced this issue in akka/alpakka Nov 9, 2016

Merged

Initial import of https://github.com/bluelabsio/s3-stream #24

1 of 3 tasks complete
@synox

This comment has been minimized.

Show comment
Hide comment
@synox

synox Nov 14, 2016

Contributor

The proxy setting is still there: https://github.com/akka/akka-http/blob/master/akka-http-core/src/main/resources/reference.conf
Please remove it as it. I causes confusion and I tasted too much time for debugging.

#89 is not a duplicate, as it for server side, not for client side.

Contributor

synox commented Nov 14, 2016

The proxy setting is still there: https://github.com/akka/akka-http/blob/master/akka-http-core/src/main/resources/reference.conf
Please remove it as it. I causes confusion and I tasted too much time for debugging.

#89 is not a duplicate, as it for server side, not for client side.

@jypma

This comment has been minimized.

Show comment
Hide comment
@jypma

jypma Nov 24, 2016

Member

Thinking about PR'ing on this, but I have a hard time deciding what to do with maxConnections:

  1. We could apply it per target host, as it is now, simply routing them all through the same proxy if settings say so. This would then only indirectly limit the max connections per proxy
  2. We could apply it to the proxy, by routing all proxied requests for the same proxy to the same shared pool gateway. This would directly limit connections per proxy, but we'd lose ability to limit per target host.
  3. Somehow wrap PoolFlow (?) in something to make it proxy-aware, so it can limit both max connections per proxy AND per host

I'm opting for (1), for simplicity's sake. Anyone else?

Member

jypma commented Nov 24, 2016

Thinking about PR'ing on this, but I have a hard time deciding what to do with maxConnections:

  1. We could apply it per target host, as it is now, simply routing them all through the same proxy if settings say so. This would then only indirectly limit the max connections per proxy
  2. We could apply it to the proxy, by routing all proxied requests for the same proxy to the same shared pool gateway. This would directly limit connections per proxy, but we'd lose ability to limit per target host.
  3. Somehow wrap PoolFlow (?) in something to make it proxy-aware, so it can limit both max connections per proxy AND per host

I'm opting for (1), for simplicity's sake. Anyone else?

@michkhol

This comment has been minimized.

Show comment
Hide comment
@michkhol

michkhol Mar 7, 2017

+1 for this. Our internal services were switched to proxy-only access, and I really hate to do any hacking or forking to make our client software work again.

michkhol commented Mar 7, 2017

+1 for this. Our internal services were switched to proxy-only access, and I really hate to do any hacking or forking to make our client software work again.

@ktoso

This comment has been minimized.

Show comment
Hide comment
@ktoso

ktoso Mar 7, 2017

Member

Help in the ongoing PR is going to help get this delivered, +1's not really :-)
You're welcome to help with progress on it here:

  • #921
  • (old PR, rather focus on the above one IMO) #921
Member

ktoso commented Mar 7, 2017

Help in the ongoing PR is going to help get this delivered, +1's not really :-)
You're welcome to help with progress on it here:

  • #921
  • (old PR, rather focus on the above one IMO) #921
@synox

This comment has been minimized.

Show comment
Hide comment
@raboof

This comment has been minimized.

Show comment
Hide comment
@raboof

raboof Sep 5, 2017

Member

I'm not sure that covers all the aspects mentioned in this thread, such as honoring system properties such as http.proxyHost

Member

raboof commented Sep 5, 2017

I'm not sure that covers all the aspects mentioned in this thread, such as honoring system properties such as http.proxyHost

@jrudolph

This comment has been minimized.

Show comment
Hide comment
@jrudolph

jrudolph Sep 5, 2017

Member

@synox we now have https proxy support but not yet for generic http proxies and better configuration in general.

Member

jrudolph commented Sep 5, 2017

@synox we now have https proxy support but not yet for generic http proxies and better configuration in general.

@gilbertw1 gilbertw1 referenced this issue in gilbertw1/slack-scala-client Sep 13, 2017

Closed

Can't work behind proxy. #56

@2m 2m referenced this issue in playframework/play-ws Dec 18, 2017

Closed

MEGA ticket for Akka Http backend related tickets #207

1 of 12 tasks complete

@2m 2m referenced this issue in playframework/play-ws Jan 4, 2018

Open

Proxy support for akka-http backend #214

@usagiy

This comment has been minimized.

Show comment
Hide comment
@usagiy

usagiy Jul 2, 2018

Hi. This is really showstopper for us. Because akka sets Host parameter in request and we cant override it so there is no way to emulate behavior of http_proxy on curl or python requests for example.

usagiy commented Jul 2, 2018

Hi. This is really showstopper for us. Because akka sets Host parameter in request and we cant override it so there is no way to emulate behavior of http_proxy on curl or python requests for example.

@jlprat

This comment has been minimized.

Show comment
Hide comment
@jlprat

jlprat Jul 2, 2018

Member

Hi @usagiy, you are very welcome to help out with a PR if you have time to provide it. For the looks of it, it's not one of the tasks that will be pick immediately by the core team.

Member

jlprat commented Jul 2, 2018

Hi @usagiy, you are very welcome to help out with a PR if you have time to provide it. For the looks of it, it's not one of the tasks that will be pick immediately by the core team.

@adrianlyjak

This comment has been minimized.

Show comment
Hide comment
@adrianlyjak

adrianlyjak Jul 2, 2018

For those interested, This roughly works for an http proxy. It falls on its face if the HttpRequest's scheme is not http
https://gist.github.com/adrianlyjak/a05da7b7d1e2c3d6989c45bb14965898

For those interested, This roughly works for an http proxy. It falls on its face if the HttpRequest's scheme is not http
https://gist.github.com/adrianlyjak/a05da7b7d1e2c3d6989c45bb14965898

@dwalend

This comment has been minimized.

Show comment
Hide comment
@dwalend

dwalend Jul 2, 2018

@usagiy - I was able to get past a problem with similar buzzwords - "proxy" is poorly defined - too many different meanings - I had to work through someone else's ajp 2.4 's client-side https proxy. My solution is way outside : http4s' client, driving the traffic through a java AsyncHttpClient. It does work pretty robustly for a few customers. Hope it helps. https://gist.github.com/dwalend/3f8439e882cd4430b0e266675b3fba94

dwalend commented Jul 2, 2018

@usagiy - I was able to get past a problem with similar buzzwords - "proxy" is poorly defined - too many different meanings - I had to work through someone else's ajp 2.4 's client-side https proxy. My solution is way outside : http4s' client, driving the traffic through a java AsyncHttpClient. It does work pretty robustly for a few customers. Hope it helps. https://gist.github.com/dwalend/3f8439e882cd4430b0e266675b3fba94

@usagiy

This comment has been minimized.

Show comment
Hide comment
@usagiy

usagiy Jul 3, 2018

Thanks guys, thanks @adrianlyjak I have basically reused your code and it works

usagiy commented Jul 3, 2018

Thanks guys, thanks @adrianlyjak I have basically reused your code and it works

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment