With requests >= 2.2.1, no_proxy environment variable contents are now handled correctly (see https://github.com/kennethreitz/requests/pull/945 ), but -- as far as I can tell -- because devpi_common/requests.py's new_requests_session function is using urllib.getproxies() to set requests's Session object's proxies member, the Session doesn't understand urllib's getproxies() returned-dict's "no" key's value as the list of hosts for which it should ignore proxies. The requests's Session object just checks the "http"/scheme key, (perhaps) finds a proxy, and uses it.
It's a little tricky to demonstrate the problem, so I "instrumented" requests/{utils,sessions}.py:
--- utils.py~ Tue Jan 21 18:34:38 2014
+++ utils.py Mon Feb 3 17:34:28 2014
@@ -492,6 +492,7 @@
if netloc.endswith(host) or netloc.split(':')[0].endswith(host):
# The URL does match something in no_proxy, so we don't want
# to apply the proxies on this URL.
+ print "no proxies for {}".format(host)
return {}
# If the system proxy settings indicate that this URL should be bypassed,
--- sessions.py~ Tue Jan 21 18:34:38 2014
+++ sessions.py Mon Feb 3 17:41:54 2014
@@ -366,11 +366,13 @@
verify = os.environ.get('CURL_CA_BUNDLE')
# Merge all the kwargs.
+ print "sending request with proxies 1: {} (self: {})".format(proxies, self.proxies)
proxies = merge_setting(proxies, self.proxies)
stream = merge_setting(stream, self.stream)
verify = merge_setting(verify, self.verify)
cert = merge_setting(cert, self.cert)
+ print "sending request with proxies 2: {}".format(proxies)
# Send the request.
send_kwargs = {
'stream': stream,
@@ -483,6 +485,7 @@
# Start time (approximately) of the request
start = datetime.utcnow()
# Send the request
+ print "sending {} to proxies: {} via adapter {}".format(request, proxies, adapter)
r = adapter.send(request, **kwargs)
# Total elapsed time of the request (approximately)
r.elapsed = datetime.utcnow() - start
You can see, given the below environment variables, the difference between what requests uses for proxies with the login POST request and what proxies devpi_common's use of requests causes to be used:
$ env | grep http_proxy
http_proxy=http://proxy.example.com:80
$ env | grep no_proxy
no_proxy=private_machine
This is the DESIRED behaviour:
$ python -c 'import requests ; requests.post("http://private_machine:3141/+login", data={"user": "uuu", "password": "xxx"}).text'
no proxies for private_machine
sending request with proxies 1: {} (self: {})
sending request with proxies 2: OrderedDict()
sending <PreparedRequest [POST]> to proxies: OrderedDict() via adapter <requests.adapters.HTTPAdapter object at 0x033D9A70>
File "<string>", line 1, in <module>
File "c:\python\lib\site-packages\requests\api.py", line 88, in post
return request('post', url, data=data, **kwargs)
File "c:\python\lib\site-packages\requests\api.py", line 44, in request
return session.request(method=method, url=url, **kwargs)
File "c:\python\lib\site-packages\requests\sessions.py", line 385, in request
resp = self.send(prep, **send_kwargs)
File "c:\python\lib\site-packages\requests\sessions.py", line 489, in send
r = adapter.send(request, **kwargs)
File "c:\python\lib\site-packages\requests\adapters.py", line 311, in send
conn = self.get_connection(request.url, proxies)
File "c:\python\lib\site-packages\requests\adapters.py", line 203, in get_connection
import traceback ; traceback.print_stack()
NOT using proxy for url http://private_machine:3141/+login
This is the UNDESIRED behaviour:
$ devpi login --password xxx uuu
no proxies for private_machine
sending request with proxies 1: {} (self: {'http': 'http://proxy.example.com:80', 'no': 'private_machine'})
sending request with proxies 2: OrderedDict([('http', 'http://proxy.example.com:80'), ('no', 'private_machine')])
sending <PreparedRequest [POST]> to proxies: OrderedDict([('http', 'http://proxy.example.com:80'), ('no', 'private_machine')]) via adapter
<requests.adapters.HTTPAdapter object at 0x031895F0>
File "c:\python\Scripts\devpi-script.py", line 9, in <module>
load_entry_point('devpi-common==1.2', 'console_scripts', 'devpi')()
File "C:\python\lib\site-packages\devpi\main.py", line 29, in main
return method(hub, hub.args)
File "C:\python\lib\site-packages\devpi\login.py", line 11, in main
r = hub.http_api("post", hub.current.login, input, quiet=False)
File "C:\python\lib\site-packages\devpi\main.py", line 96, in http_api
auth=auth)
File "C:\python\lib\site-packages\requests\sessions.py", line 385, in request
resp = self.send(prep, **send_kwargs)
File "C:\python\lib\site-packages\requests\sessions.py", line 489, in send
r = adapter.send(request, **kwargs)
File "C:\python\lib\site-packages\requests\adapters.py", line 311, in send
conn = self.get_connection(request.url, proxies)
File "C:\python\lib\site-packages\requests\adapters.py", line 203, in get_connection
import traceback ; traceback.print_stack()
using proxy http://proxy.example.com:80 for url http://private_machine:3141/+login
WARN: devpi-client-1.2.1 got an unversioned reply, assuming API-VERSION 1 (as implemented by devpi-server-1.1 and 1.2)
POST http://private_machine:3141/+login
407 Proxy Authentication Required
I'm not sure whether requests should be taught about urllib's OrderedDict([('http', 'http://proxy.example.com:80'), ('no', 'private_machine')]) "api" or whether devpi should rely on requests's proxy detection/handling.
With requests >= 2.2.1, no_proxy environment variable contents are now handled correctly (see https://github.com/kennethreitz/requests/pull/945 ), but -- as far as I can tell -- because devpi_common/requests.py's new_requests_session function is using urllib.getproxies() to set requests's Session object's proxies member, the Session doesn't understand urllib's getproxies() returned-dict's "no" key's value as the list of hosts for which it should ignore proxies. The requests's Session object just checks the "http"/scheme key, (perhaps) finds a proxy, and uses it.
It's a little tricky to demonstrate the problem, so I "instrumented" requests/{utils,sessions}.py:
You can see, given the below environment variables, the difference between what requests uses for proxies with the login POST request and what proxies devpi_common's use of requests causes to be used:
This is the DESIRED behaviour:
This is the UNDESIRED behaviour:
I'm not sure whether requests should be taught about urllib's OrderedDict([('http', 'http://proxy.example.com:80'), ('no', 'private_machine')]) "api" or whether devpi should rely on requests's proxy detection/handling.