Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Issues with HTTP proxy and accessing localhost - does requests ignore no_proxy? #879

Closed
victorhooi opened this Issue · 10 comments

6 participants

@victorhooi

Hi,

I have a Django app, exposing a REST API using TastyPie.

I have it running locally using Django's runserver, and I'm trying to access it using Requests.

For some reason though, accessing it with Curl or Firefox works fine:

curl "http://localhost:8000/api/v1/host/?name__regex=&format=json"

And on the Django runserver console, I see:

[02/Oct/2012 17:24:20] "GET /api/v1/host/?name__regex=&format=json HTTP/1.1" 200 2845

However, when I try to use the Python requests module (http://docs.python-requests.org/en/latest/), I get a 404:

>>> r = requests.get('http://localhost:8000/api/v1/host/?name__regex=&format=json')
>>> r
<Response [404]>

or:

>>> r = requests.get('http://localhost:8000/api/v1/host/?name__regex=&amp;format=json')
>>> r
<Response [404]>

or:

>>> payload = { 'format': 'json'}
>>> r = requests.get('http://localhost:8000/api/v1', params=payload)
>>> r
<Response [404]>
>>> r.url
u'http://localhost:8000/api/v1?format=json'

Also, on the Django runserver console, I see:

[02/Oct/2012 17:25:01] "GET http://localhost:8000/api/v1/host/?name__regex=&format=json HTTP/1.1" 404 161072

Notice that runserver prints the whole path in this case, rather than just the part after localhost:8000 - not sure why that is.

We have a HTTP proxy setup in this environment (Microsoft ISA, I believe), which uses authentication (I've put the username/password in the variable itself).

I've set the no_proxy environment variable and put localhost and 127.0.0.1 in there.

However, requests seems to still be doing something funky with the proxy.

If I actually unset http_proxy - requests works fine:

$ unset http_proxy
$ python
Python 2.7.3 (default, Sep  7 2012, 12:28:31) 
[GCC 4.1.2 20080704 (Red Hat 4.1.2-52)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> hostname = ''
>>> r = requests.get('http://localhost:8000/api/v1/host/?name__regex={}&format=json'.format(hostname))
>>> r
<Response [200]>

So firstly, I'm curious why requests doesn't work with the proxy variable set? And secondly, is there any any way to get it to respect no_proxy?

Cheers,
Victor

@timsavage

I have a similar problem where I just want to disable the proxy for certain requests (connecting to internal WebServices that are not available outside our DMZ). After digging through the requests code to find out how proxies are handled and how proxy configuration is loaded I have come up with a work-around to disable the proxy when needed:

NO_PROXY = {
    'no': 'pass',
}

r = requests.get(uri, proxies=NO_PROXY)

How this works, if the proxies parameter is passed to the get (or other http method) it is has any empty entries removed before checking if there are any values still in the dictionary. If the dictionary is empty then the variables are loaded from the environment using the method requests.utils.get_environ_proxies. By including a value in the 'no' key then the proxies parameter is used which includes empty http and https entries thereby disabling the proxy. The method is a bit hacky but does solve the issue.

@Lukasa
Collaborator

By default, requests trusts any environment variables the system has in place, and assumes you want to use them (which is usually the default behaviour of UNIXy programs). You can disable this behaviour either by using @timsavage's method (which is clever, nice workaround), or by setting the trust_env config value to False, like so:

r = requests.get('http://www.google.com', config={'trust_env': False})
s = requests.Session(config={'trust_env': False})

You can see everything you can change with the config dictionary here.

Hope that helps!

@Lukasa Lukasa closed this
@victorhooi

heya,

Hmm, yes, but by the same logic, wouldn't requests also trust the NO_PROXY environment variable as well?

This is meant to be a list of specific hosts for which we don't use the proxy - and it's supported by most *nix HTTP clients, as far as I'm aware

just to pick a few random ones.

I don't think it's a case of not trusting the http_proxy variable, but more just having support for the no_proxy variable as well?

Cheers,
Victor

@Lukasa
Collaborator

Ah, good point well made. I have genuinely never seen the NO_PROXY variable used, so didn't even know about it. In that case, we should probably endeavour to respect it. =)

@Lukasa Lukasa reopened this
@Lukasa
Collaborator

Oh, this is interesting! We do actually check for NO_PROXY, but we treat it like a standard proxy prefix, like HTTP_PROXY and FTP_PROXY. Put another way: we do totally the wrong thing with it. =D

@piotr-dobrogost

just to pick a few random ones.

In every list of command line http clients there must be curl even if it's a random list :)

@Lukasa Lukasa referenced this issue from a commit in Lukasa/requests
@Lukasa Lukasa Respect the no_proxy environment variable.
This change is in response to issue #879.
8da100f
@Lukasa
Collaborator

Resolved by #945 (hopefully). Next minor release of Requests should have the fix in it. Please let me know if this turns out not to have fixed it for you.

@Lukasa Lukasa closed this
@jonasdt jonasdt referenced this issue in coursera-dl/coursera-dl
Closed

NameError: global name 'className' is not defined #124

@tanmay9

Hi @Lukasa , I was having this issue with Requests 1.2. After reading this thread, I thought it will go away on upgrading to 2.0.1 (which contains your fix). However, it did not fix it for me.
The only way I got around it was to add the following code:

import os
os.environ['no_proxy'] = '127.0.0.1,localhost'
@Lukasa
Collaborator

@tanmay9 How are you setting the environment variable in question?

@bhechinger

Neither of these appear to work for me.

@timsavage's method does nothing at all and the config= trick results in:

TypeError: request() got an unexpected keyword argument 'config'

This is python 3.4.0 and requests 2.2.1.

I had to resort to @tanmay9's method of setting the no_http environmental.

Just FYI.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.