SOCKS Proxy Support

[Lukasa]

This pull request finally adds SOCKS proxy support to the Requests module, pretty much four years to the day since we were first asked (in #324 for those who want to follow it).

This pull request relies on a pull request I have opened against urllib3, so we can't land it until urllib3 1.14.1 is released containing that code. However, once that's done this PR should be ready to rock.

I'm setting this up for the 2.10.0 milestone as I see no reason to wait for 3.0.0 for SOCKS support.

Resolves #324, #723, #1147, #1982, #2156, #2425, and #2562.

[jezdez]

🙏

[sigmavirus24]

While we're here, let's reformat this a bit:

if proxy in self.proxy_manager:
    manager = self.proxy_manager[proxy]
elif proxy.lower().startswith('socks'):
    username, password = get_auth_from_url(proxy)
    manager = self.proxy_manager[proxy] = SOCKSProxyManager(
        # ...
    )
else:
    proxy_headers = self.proxy_headers(proxy)
    manager = self.proxy_manager[proxy] = proxy_from_url(
        # ...
    )
return manager

I feel like this will be easier to read

[Lukasa]

Done

[sigmavirus24]

I see how you were being tricksy now. I wonder if we can be better about this.

We essentially have the same code path (url = request.path_url) if we're using a 'socks' proxy or if we're making an HTTPS request. What if we do

proxied_http_request = proxy and scheme != 'https'
using_socks_proxy = False
if proxy:
    using_socks_proxy == urlparse(proxy).scheme.lower().startswith('socks')

url = request.path_url
if proxied_http_request and not using_socks_proxy:
    url = urldefragauth(request.url)

I puzzled through my logic with

proxy	http_scheme && !socks_proxy_scheme	https_scheme	socks_proxy_scheme
truthy	urldefragauth(request.url)	request.path_url	request.path_url
falsey	request.path_url	request.path_url	request.path_url

where http_scheme is where (scheme != 'https') is True and https_scheme is where (scheme != 'https') is False. Hopefully the table makes sense.

[Lukasa]

Ok, I'm going with that. I degolfed this a little bit to aid clarity, but not much.

[sigmavirus24]

👍 on the code and 👍 on the milestone.

[sigmavirus24]

@Lukasa what do you think about having a socks extra for requests that installs PySocks and whatever else?

[Lukasa]

Done. =)

[kennethreitz]

Does PySocks require any compilation during installation? We should consider vendoring it if we want to consider this an actual feature of Requests.

[Lukasa]

PySocks requires no compilation. However, we'll have to rewrite import paths in urllib3 if you want to vendor PySocks.

[kennethreitz]

Well, if it's going to be used by a very small subset of users (likely the case), it's likely fine to leave it as an optional dependency.

We could pack it in without modifying urllib3's code, technically:

(packages/__init__.py)

try:
    from . import socks
    sys.modules['socks'] = socks
except ImportError:
    try:
        import socks
    except ImportError:
        pass

This is obviously not the best idea, but it does function properly.

[sigmavirus24]

Mucking with sys.modules in the past has caused us enough headaches that I'd rather not go too far down that path again

[kennethreitz]

I don't disagree :)

[loretoparisi]

@Lukasa is this was merged or not? Btw, how to configure the socks proxy?

[Lukasa]

This has not been merged.

When it is released, you can configure the socks proxy as you would configure other proxies: proxies={'http': 'socks5://localhost:8888/'}, for example.

[loretoparisi]

@Lukasa so it should work like

#proxy
        # SOCKS5 proxy for HTTP/HTTPS
        proxies = {
            'http' : "socks5://myproxy:9191",
            'https' : "socks5://myproxy:9191"
        }

        #headers
        headers = {

        }

        url='http://icanhazip.com/'
        res = requests.get(url, headers=headers, proxies=proxies)

[Lukasa]

Correct. Of course, that won't work unless you use a patched version of requests that contains both this patch and an updated urllib3, and you install the appropriate optional dependencies.

[loretoparisi]

@Lukasa uhm, so supposed to use on google appengine, I have to wait the devappserver2 python sdk update...

[Lukasa]

On GAE I suspect using a SOCKS proxy is going to take a while. I should also note that to use a SOCKS proxy on GAE will require that you enable socket support, which as I understand it is substantially more expensive than simply using the standard httplib replacement.

[loretoparisi]

btw, In the meanwhile I'm doing

>>> import requesocks
>>> import requests
>>> session = requesocks.session()
>>> session.proxies = {
...     'http' : "socks5://myproxy.net:9050",
...     'https' : "socks5://smyproxy.net:9050"
... }
>>> url='http://icanhazip.com/'
>>> resp = requests.get(url)
>>> resp
<Response [200]>
>>> resp.text
u'89.97.90.230\n'
>>> resp.text.strip()
u'89.97.90.230'
>>> resp = session.get(url)
>>> resp.text.strip()
u'171.25.193.78'
>>> 

that in most envs will work ( a part again appengine that lacks of pwd support 😡

[kennethreitz]

Is this done? Let's merge it if so.

[Lukasa]

We need an updated urllib3. I think @shazow is expecting to release this week.

[shazow]

Released!

[kennethreitz]

Sweet, let's do it.

[kennethreitz]

We'll include it in the next release, which can therefore happen this week (or even today).

[Lukasa]

@kennethreitz I'm away from my laptop, so if you're in a merging mood you can land the urllib3 update and then merge this.

Please note that it makes everyone's life easier if you merge in urllib3 directly from the brand new tag (1.15). Just a suggestion!

[darvid]

The anticipation is killing me.

[dequis]

What's [WIP] about this? Previous comments seem to suggest it's done, just waiting for the release.

(Also, referencing urllib3/urllib3#762 and #478)

[kennethreitz]

✨ 🍰 ✨

[darvid]

[kennethreitz]

Released, announced.