Proxy support broken #1074

Closed
mishari opened this Issue Dec 30, 2012 · 7 comments

Comments

Projects
None yet
5 participants

mishari commented Dec 30, 2012

I am using Python Requests with Polipo, after I updated Requests to the latest version, when I do a get, rather than giving me the output of the content of the website, I get a page that is equivalent to performing a request to http://localhost:8123

Update 1:
Wireshark gives me the following output for the request header.
GET / HTTP/1.1
Host: 127.0.0.1:8123
Content-Length: 0
Accept-Encoding: gzip, deflate, compress
Accept: /

MicksMix commented Jan 3, 2013

I am having the same issue. Wrote a script in Perl to download a file (http://live.sysinternals.com/streams.exe) and one in Python using 'requests' to do the same thing.

Here's the Python script I'm using (fiddler is acting as my proxy on 127.0.0.1:8888). The file fails to download.

#!/usr/bin/env python
import sys
import requests
import os

url = 'http://live.sysinternals.com/streams.exe'
http_proxy = "http://127.0.0.1:8888"

filename = 'c://temp//streams.exe'

proxyDict = {
    "http" : http_proxy,
    #"https" : https_proxy,
    #"ftp" : ftp_proxy
}

r = requests.get(url, proxies=proxyDict)

with open(filename, 'wb') as fp:
fp.write(r.content)

For my Python script using 'requests', Fiddler shows GET request as if it were requesting "streams.exe" directly from my proxy server:

GET /streams.exe HTTP/1.1
Host: 127.0.0.1:8888
Content-Length: 0
Accept-Encoding: gzip, deflate, compress
Accept: */*
User-Agent: python-requests/1.0.4 CPython/2.7.2 Windows/7

The Perl script (using LWP::UserAgent) works correctly by downloading the file and shows the following in Fiddler:

GET /streams.exe HTTP/1.1
TE: deflate,gzip;q=0.3
Connection: TE, close
Host: live.sysinternals.com
User-Agent: libwww-perl/6.04

Here's that Perl script:

#!/usr/bin/env perl
use strict;
require LWP::UserAgent;

my $ua = LWP::UserAgent->new();

$ua->proxy( [ 'http', 'ftp', 'https' ], 'http://127.0.0.1:8888' );

my $url      = "http://live.sysinternals.com/streams.exe";
my $response = $ua->get($url);
my $filename = "c://temp//streams.exe";

if ( $response->is_success ) {
    open( my $fh, '>', $filename );
    print $fh $response->decoded_content;
    close $fh;
}
else {
    die $response->status_line;
}
Contributor

erikcw commented Jan 5, 2013

I'm having the same issue. When proxying a request through Charles Proxy I get a "503 Malformed request URL "/"" error. requests seems to work fine when not using proxies.

trentvb commented Jan 7, 2013

Just downloaded the library to test it out....ran into the same issue trying to proxy through fiddler.....requests turns 'http://someurl.com/api/v1' into 'http://127.0.0.1:888/api/v1' when using {'http':'127.0.0.1:8888'} as proxy.

MicksMix commented Jan 7, 2013

This appears to ultimately be a problem with urllib3, which requests uses:
kennethreitz#905

According to this comment (kennethreitz#905 (comment)), Kenneth Reitz is "working on a rewrite using transport adapters.".

Ultimately, urllib3 needs to be patched in order to get proxy support working as-is with requests.

I just switched to urllib2 for my small project and everything is working fine in regard to the proxy support.

Owner

Lukasa commented Jan 8, 2013

@MicksMix: You've actually confused two different issues, which is totally understandable. =)

#905 is specifically a problem with HTTPS over proxies: in particular, urllib3 does not use the CONNECT verb. This issue is affecting all proxies, and is a result of the refactor.

@trentvb, @erikcw, @mishari, @MicksMix: Some similar issues were raised earlier: see #1056 and #1058. I submitted a fix for those in PR #1060, which has not yet made it to PyPI. Try using the version of Requests from master and see if that solves your problem.

MicksMix commented Jan 8, 2013

@Lukasa Thanks!

Owner

Lukasa commented Jan 19, 2013

Based on the lack of further complaints, I'm assuming that #1060 resolved the issue. Let me know if you're still suffering. Thanks for your contribution!

Lukasa closed this Jan 19, 2013

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment