New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pip doesn't use proxy for search command #1104

Closed
albgus opened this Issue Aug 2, 2013 · 10 comments

Comments

Projects
None yet
8 participants
@albgus

albgus commented Aug 2, 2013

I'm attempting to use pip to install packages to a python virtualenv. I have however discovered that when trying to search for commands, the proxy is not used. running for example pip search pika fails while pip install pika works fine.

I have set the environment variables http_proxy, https_proxy and no_proxy accordingly and I have tried to set the --proxy option with no success. It does however ask for a password if I specify a username but no password, so at least it acknowledges the options (the proxy is anonymous login)

I run into this issue on RedHat 6.4, with the default Python 2.6.6, and Python 2.7.5 from the tarball distribution.

Here are the logs:

Traceback (most recent call last):
  File "/home/gusson/.Python/vcdhelper26/lib/python2.6/site-packages/pip/basecommand.py", line 134, in main
    status = self.run(options, args)
  File "/home/gusson/.Python/vcdhelper26/lib/python2.6/site-packages/pip/commands/search.py", line 38, in run
    pypi_hits = self.search(query, index_url)
  File "/home/gusson/.Python/vcdhelper26/lib/python2.6/site-packages/pip/commands/search.py", line 52, in search
    hits = pypi.search({'name': query, 'summary': query}, 'or')
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1199, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1489, in __request
    verbose=self.__verbose
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1235, in request
    self.send_content(h, request_body)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1349, in send_content
    connection.endheaders()
  File "/usr/lib64/python2.6/httplib.py", line 908, in endheaders
    self._send_output()
  File "/usr/lib64/python2.6/httplib.py", line 780, in _send_output
    self.send(msg)
  File "/usr/lib64/python2.6/httplib.py", line 739, in send
    self.connect()
  File "/usr/lib64/python2.6/httplib.py", line 1112, in connect
    sock = socket.create_connection((self.host, self.port), self.timeout)
  File "/usr/lib64/python2.6/socket.py", line 567, in create_connection
    raise error, msg
timeout: timed out

Storing complete log in /home/gusson/.pip/pip.log

If I run a strace on the process it stops on this message, indicating that it tries to contact the source directly without going via the proxy, thus gets blocked by the firewall.

connect(3, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr("199.27.73.185")}, 16) = -1 EINPROGRESS (Operation now in progress)
poll([{fd=3, events=POLLOUT}], 1, 15000
@monkeypants

This comment has been minimized.

Show comment
Hide comment
@monkeypants

monkeypants Oct 10, 2013

pip search uses xmlrpclib, which isn't natively proxy aware. pip install uses urllib2, which is (uses the HTTP_PROXY environment variable etc).

There are various examples of people creating a urllib2 transport and tunneling xmlrpclib over it, see for example https://gist.github.com/nathforge/980961 .There is even something like this happening in vendor/distlib/util.py.

Simply patching commands/search.py to use code from nathforge's gist didn't work though, possibly because pip search it's using xmlrpclib from pip.backwardcompat rather than the native one.

monkeypants commented Oct 10, 2013

pip search uses xmlrpclib, which isn't natively proxy aware. pip install uses urllib2, which is (uses the HTTP_PROXY environment variable etc).

There are various examples of people creating a urllib2 transport and tunneling xmlrpclib over it, see for example https://gist.github.com/nathforge/980961 .There is even something like this happening in vendor/distlib/util.py.

Simply patching commands/search.py to use code from nathforge's gist didn't work though, possibly because pip search it's using xmlrpclib from pip.backwardcompat rather than the native one.

@kkdd

This comment has been minimized.

Show comment
Hide comment
@kkdd

kkdd Oct 19, 2013

The replacement of xmlrpclib.ServerProxy() in search.py by the xmlrpclib_ServerProxy() defined below works in my environment of Python 2.7.5 and pip 1.4.1.

import os

def get_proxy():
    proxy = os.getenv('http_proxy')
    if proxy and proxy.startswith("http://"):
        proxy = proxy[len('http://'):]
    return proxy

def xmlrpclib_ServerProxy(index_url):
    if get_proxy():
        p = ProxiedTransport()
        return xmlrpclib.ServerProxy(index_url, transport=p)
    else:
        return xmlrpclib.ServerProxy(index_url)

import httplib

class ProxiedTransport(xmlrpclib.Transport):
    def make_connection(self, host):
        self.realhost = host
        return httplib.HTTPConnection(get_proxy())
    def send_request(self, connection, handler, request_body):
        connection.putrequest("POST", 'http://%s%s' % (self.realhost, handler))
    def send_host(self, connection, host):
        connection.putheader('Host', self.realhost)

kkdd commented Oct 19, 2013

The replacement of xmlrpclib.ServerProxy() in search.py by the xmlrpclib_ServerProxy() defined below works in my environment of Python 2.7.5 and pip 1.4.1.

import os

def get_proxy():
    proxy = os.getenv('http_proxy')
    if proxy and proxy.startswith("http://"):
        proxy = proxy[len('http://'):]
    return proxy

def xmlrpclib_ServerProxy(index_url):
    if get_proxy():
        p = ProxiedTransport()
        return xmlrpclib.ServerProxy(index_url, transport=p)
    else:
        return xmlrpclib.ServerProxy(index_url)

import httplib

class ProxiedTransport(xmlrpclib.Transport):
    def make_connection(self, host):
        self.realhost = host
        return httplib.HTTPConnection(get_proxy())
    def send_request(self, connection, handler, request_body):
        connection.putrequest("POST", 'http://%s%s' % (self.realhost, handler))
    def send_host(self, connection, host):
        connection.putheader('Host', self.realhost)
@kkdd

This comment has been minimized.

Show comment
Hide comment
@kkdd

kkdd Nov 13, 2013

The above version of ProxiedTransport() used httplib. In contrast, the following uses urllib2. Also works in my environment of Python 2.7.5 and pip 1.4.1.

import urllib2

class ProxiedTransport(xmlrpclib.Transport):
    SCHEME = 'https'
    def __init__(self, *args, **argd):
        xmlrpclib.Transport.__init__(self, *args, **argd)
        handlers = []
        proxy = urllib2.ProxyHandler({'http':get_proxy()})
        handlers.append(proxy)
        self.opener = urllib2.build_opener(*handlers)
    def request(self, host, handler, request_body, verbose=0):
        self.verbose = 0
        scheme = self.SCHEME
        url = '%(scheme)s://%(host)s%(handler)s' % locals()
        req = urllib2.Request(url, data=request_body,
                              headers={'Content-Type':'text/xml'})
        fp = self.opener.open(req)
        return self.parse_response(fp)

kkdd commented Nov 13, 2013

The above version of ProxiedTransport() used httplib. In contrast, the following uses urllib2. Also works in my environment of Python 2.7.5 and pip 1.4.1.

import urllib2

class ProxiedTransport(xmlrpclib.Transport):
    SCHEME = 'https'
    def __init__(self, *args, **argd):
        xmlrpclib.Transport.__init__(self, *args, **argd)
        handlers = []
        proxy = urllib2.ProxyHandler({'http':get_proxy()})
        handlers.append(proxy)
        self.opener = urllib2.build_opener(*handlers)
    def request(self, host, handler, request_body, verbose=0):
        self.verbose = 0
        scheme = self.SCHEME
        url = '%(scheme)s://%(host)s%(handler)s' % locals()
        req = urllib2.Request(url, data=request_body,
                              headers={'Content-Type':'text/xml'})
        fp = self.opener.open(req)
        return self.parse_response(fp)
@drakeguan

This comment has been minimized.

Show comment
Hide comment
@drakeguan

drakeguan Nov 18, 2013

To help make this issue proceed, there is a pull request, #1180, which provides a HttpProxyTransport for xmlrpclib.ServerProxy to work perfectly with proxy. I have tested it and it worked well here. That pull request might be one of options for this issues?

drakeguan commented Nov 18, 2013

To help make this issue proceed, there is a pull request, #1180, which provides a HttpProxyTransport for xmlrpclib.ServerProxy to work perfectly with proxy. I have tested it and it worked well here. That pull request might be one of options for this issues?

@monkeypants

This comment has been minimized.

Show comment
Hide comment
@monkeypants

monkeypants Nov 18, 2013

I think that fixes the bug (--proxy ignored) and provides a work arround (pass in --proxy).

It would be nice if pip fell back to use HTTP_PROXY environment variable too, but not essential.

monkeypants commented Nov 18, 2013

I think that fixes the bug (--proxy ignored) and provides a work arround (pass in --proxy).

It would be nice if pip fell back to use HTTP_PROXY environment variable too, but not essential.

@ash211

This comment has been minimized.

Show comment
Hide comment
@ash211

ash211 Apr 16, 2014

Observed in pip 1.3.1 on CentOS 6.4 with EPEL's packages.

Works:

pip install $package

Doesn't work:

pip search $package

ash211 commented Apr 16, 2014

Observed in pip 1.3.1 on CentOS 6.4 with EPEL's packages.

Works:

pip install $package

Doesn't work:

pip search $package
@spuk-

This comment has been minimized.

Show comment
Hide comment
@spuk-

spuk- Apr 24, 2014

You don't need to set the proxy handler manually for urllib2 as it automatically detects *_proxy env vars. Also, the code above didn't work for me, didn't care to check throughly why, but it simply wasn't using the proxy although get_proxy() appeared to be finding it. Simplified patch here: https://gist.github.com/spuk-/11256147#file-pip-search-urllib2

spuk- commented Apr 24, 2014

You don't need to set the proxy handler manually for urllib2 as it automatically detects *_proxy env vars. Also, the code above didn't work for me, didn't care to check throughly why, but it simply wasn't using the proxy although get_proxy() appeared to be finding it. Simplified patch here: https://gist.github.com/spuk-/11256147#file-pip-search-urllib2

@jrosiek

This comment has been minimized.

Show comment
Hide comment
@jrosiek

jrosiek May 13, 2014

spuk's solutions is great! - simple and working. Please incorporate it into next release, thanks!

jrosiek commented May 13, 2014

spuk's solutions is great! - simple and working. Please incorporate it into next release, thanks!

@Ivoz

This comment has been minimized.

Show comment
Hide comment
@Ivoz

Ivoz Jun 12, 2014

Member

Relevant PRs #932, #1180

Alternate fix would be through #395

Member

Ivoz commented Jun 12, 2014

Relevant PRs #932, #1180

Alternate fix would be through #395

dstufft added a commit that referenced this issue Jul 3, 2014

@Ivoz

This comment has been minimized.

Show comment
Hide comment
@Ivoz

Ivoz Jul 4, 2014

Member

Should be fixed in 1.6 through #1902 as above

Member

Ivoz commented Jul 4, 2014

Should be fixed in 1.6 through #1902 as above

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment