New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support additional headers for TLS proxies #742
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@jjiang-stripe Thanks for your help! I like the separate key in the config, as I definitely think that seems cleaner/safer. I've added some areas for discussion and possible refinement, and don't think we are too far off from a great addition. Thanks!
lib/excon/ssl_socket.rb
Outdated
@@ -104,6 +104,12 @@ def initialize(data = {}) | |||
|
|||
request += "Proxy-Connection: Keep-Alive#{Excon::CR_NL}" | |||
|
|||
if @data[:proxy][:headers] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think the approach of using a separate key of ssl_proxy_headers
makes a lot of sense, especially as proxy is generally expected to be a string (though the code is tolerant of a hash, like you suggest, I doubt it gets used that way most of the time).
That being said, if we are already using a separate key, I wonder if it might be simpler/cleaner if we just leave that key/value as it is (instead of translating it to be inside the proxy hash) and just read directly from it here when we are setting up the ssl socket?
Does that make sense? I certainly might be missing something, but on review it occurred to me that sort of "cutting out the middle person" here and going directly to the hash could keep things simpler coding wise and avoid any potential issues of the mutated proxy hash being interpreted incorrectly elsewhere.
What do you think?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ah yes that makes total sense! I'll update in a bit
tests/proxy_tests.rb
Outdated
@@ -100,6 +100,10 @@ def env_proxy_tests(env) | |||
connection.data[:proxy][:scheme] | |||
end | |||
|
|||
tests('connection.data[:proxy][:headers]').returns({ 'x-proxy-id': 'abc123' }) do |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These tests are good for ensuring the data mutation into proxy happens as expected (unless we decide not to do that).
I think it would be great to also add a test toward the end of this file, where it uses the proxy.ru
rackup. These tests actually run a server, which receives proxy stuff and echos it back (so you can be more confident it was sent as expected). I think it would be great to also test the roundtrip on these proxy settings, though this would require updating proxy.ru
as well. I think maybe we could update it so the response passes back the value of x-proxy-id
and check the response headers to gain this confidence.
Does that make sense? What do you think?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yep! always happy to write more tests 😁 I'll dig into that
okay updated the I wasn't able to get the test set up properly working with the proxy CONNECT request though. it seems like Sinatra does not like CONNECT requests :( I pushed my (failing) wip tests in case you have any ideas on how I can fix them. |
@jjiang-stripe Looks great overall, and I appreciate you extracting the shared code to utils for the header writing. I'll have to research proxy testing more, I recall us having some challenges there before, and don't know the answer off hand. |
This reverts commit 3a9dba4.
So I've found that adding Then the test ends up failing with an SSL error |
oof just uncovered an unexpected consequence of using It was working when we were using there's a couple options here, but i'm not sure what's the best:
what do you think? |
@jjiang-stripe thanks for the ongoing work here. For the time being, I think having it with fewer tests is probably ok. I think I see some possibility for using the webrick proxy server class to actually test all of this, but I don't think we need to block on that, as I suspect it will be a little while before I get to it. I hadn't considered the data vs datum issue, so thanks for finding that and considering the options. I'm not sure we've had a case previously where middlewares impacted socket settings, so it may have just not come up before. From those options, I think adding an argument to socket which takes a datum sounds like a good approach to solve this issue and also make things a little more consistent in usage and expectations. Does that sound good to you? |
I was leaning towards the socket argument option too! I just updated the code and verified it's working with our middleware (setting |
I believe it is reusing sockets, but it's easy to miss. The socket method has an |
Also, fwiw, here is a quick issue I just created around improved proxy testing with some of the more promising candidates: #746 I hope to return to it at some point, but haven't had a lot of time for development lately, so we'll see. |
@jjiang-stripe Thanks! Just let me know if you have other questions or suggestions. I'll try to get a release out soon (maybe early in the week, as I prefer not to do it on the weekend when I'd be less likely to notice if something goes wrong with it). |
oh nice I see that bit for reusing sockets now. thanks for all your help here! 😁 |
Released in v0.81.0. |
Fixes #710
I don't write a whole lot of Ruby, so feel free to ask many questions/correct me if things look wonky!
This adds a new
ssl_proxy_headers
option to the connection parameters that can be used to specify headers to send on the proxy CONNECT request. I added it as a separate parameter, since@data[:proxy]
is typically used as a string value, and while users could build their own@data[:proxy]
hash, it seems a little cumbersome to use properly.