HTTPS requests fail through a proxy

[bhendo]

Requests to access a secure website (SSL/TLS) fail through a proxy.
Urllib3 does not properly implement the HTTP CONNECT method.

For example the following code should print 200.
Instead, with a burp proxy, it prints 502.

import urllib3

proxy = urllib3.proxy_from_url('http://localhost:8080/'
response = proxy.urlopen('GET', 'https://www.google.com/index.html')

print response.status

[bhendo]

import urllib3
import unittest

class TestProxyConnect(unittest.TestCase):
    def setUp(self):
        self.proxy_url='http://localhost:8080/'
        self.https_url='https://www.google.com/index.html'
    def test_connect(self):
        proxy=urllib3.proxy_from_url(self.proxy_url)
        res=proxy.urlopen('GET', self.https_url)
        self.assertEqual(res.status, 200)

if __name__ == '__main__':
    unittest.main()

[shazow]

Thank you for porting the report and adding the test, @khaosx. We'll have to port the test to one that doesn't depend on external services at some point.

I've been thinking about this bug and sadly I suspect it won't be trivial given the simplicity of the ProxyManager implementation right now. I think the best thing to do is to block this one on #44 which will require some added complexity for ProxyManager on its own and we can sneak this fix in there somehow.

If you'd like to take a crack at fixing this one yourself, you're more than welcome. :-)

[bhendo]

I came up with a fix that works for me. If you want to test it and add it please do.
I forked your code to post the fixes

https://github.com/khaosx/urllib3

[bhendo]

Following redirects through a connect tunnel is not thread safe at the moment.

After additional testing it probably makes sense to create an subclass of httpsconnectionpool and verifiedhttpsconnection and override a few methods instead of adding the fix right into connectionpool.py.

For now the new subclass of httpsconnectionpool should not follow redirects.

[shazow]

@khaosx Thanks for diving into this.

I took a quick look. Looks like you're trying to fix Issue #8 as well. That's a tricky one because there are cases where you do want to pass in the host. I'll post some feedback in comments.

I presume you're going to continue iterating on this, let me know when you think this is ready to be merged and I'll take a look and give more feedback.

[foxx]

@khaosx If you're interested, please see discussions on:
https://github.com/kennethreitz/requests/pull/478

We've implemented HTTP/HTTPS/SOCKS proxy support into Python Requests (which contains urllib3 modifications).

[sigmavirus24]

So for reference's sake, seems like #68 is work towards this?

[schlamar]

FYI, this was done in #170 :)

[shazow]

@schlamar Thanks! :)