Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

source_address support (rough) #1288

Closed
wants to merge 5 commits into from

2 participants

@infracanophile

First pass on source_address support (2.7 only).

@kennethreitz
Owner

This is what connection adapters are for.

@infracanophile

Thanks, I'll have a look at connection adapters

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Apr 3, 2013
  1. @infracanophile
  2. @infracanophile
  3. @infracanophile
  4. @infracanophile
Commits on Apr 4, 2013
  1. @infracanophile

    HTTPSConnection supports source_address

    infracanophile authored
    bit inconsistent with argument ordering I think, worth going back to
    double check that at some point
This page is out of date. Refresh to see the latest.
View
1  AUTHORS.rst
@@ -123,3 +123,4 @@ Patches and Suggestions
- Denis Ryzhkov <denisr@denisr.com>
- Wilfred Hughes <me@wilfred.me.uk> @dontYetKnow
- Dmitry Medvinsky <me@dmedvinsky.name>
+- Jesse Sherlock <jesse@jessesherlock.com>
View
89 README.rst
@@ -1,82 +1,31 @@
-Requests: HTTP for Humans
-=========================
+Requests: HTTP for Humans (now with source_address support)
+===========================================================
+Fork of Kenneth Reitz's fantastic Requests library with source_address support (and a forked version of the included urllib3 that's included with requests).
-.. image:: https://travis-ci.org/kennethreitz/requests.png?branch=master
- :target: https://travis-ci.org/kennethreitz/requests
+This means that this fork is only compatible with Python 2.7, when HTTPConnection added source_address support (and I have not yet tested it on 3.3). This first draft is a little specific to my Python 2.7 needs and I'll wait for feedback before I clean things up.
-Requests is an Apache2 Licensed HTTP library, written in Python, for human
-beings.
+Find the real documentation here: https://github.com/kennethreitz/requests
-Most existing Python modules for sending HTTP requests are extremely
-verbose and cumbersome. Python's builtin urllib2 module provides most of
-the HTTP capabilities you should need, but the api is thoroughly broken.
-It requires an enormous amount of work (even method overrides) to
-perform the simplest of tasks.
+I've removed the docs from this fork because I haven't had time to update all of them and no docs is better than docs that are wrong.
-Things shouldn't be this way. Not in Python.
-
-.. code-block:: pycon
-
- >>> r = requests.get('https://api.github.com', auth=('user', 'pass'))
- >>> r.status_code
- 204
- >>> r.headers['content-type']
- 'application/json'
- >>> r.text
- ...
-
-See `the same code, without Requests <https://gist.github.com/973705>`_.
-
-Requests allow you to send HTTP/1.1 requests. You can add headers, form data,
-multipart files, and parameters with simple Python dictionaries, and access the
-response data in the same way. It's powered by httplib and `urllib3
-<https://github.com/shazow/urllib3>`_, but it does all the hard work and crazy
-hacks for you.
-
-
-Features
---------
-
-- International Domains and URLs
-- Keep-Alive & Connection Pooling
-- Sessions with Cookie Persistence
-- Browser-style SSL Verification
-- Basic/Digest Authentication
-- Elegant Key/Value Cookies
-- Automatic Decompression
-- Unicode Response Bodies
-- Multipart File Uploads
-- Connection Timeouts
-- Thread-safety
+source_address is a kwarg on request building methods (get/post/put/etc)
+session objects have a source_address attribute that can be set on the session (and overridden on an individual request like all the other request related session object attributes).
-Installation
-------------
-
-To install requests, simply:
-
-.. code-block:: bash
-
- $ pip install requests
-
-Or, if you absolutely must:
-
-.. code-block:: bash
-
- $ easy_install requests
-
-But, you really shouldn't do that.
+If you pass in 0 for the port then the underlying HTTPConnection object falls back to "default behaviour" which, as far as I can tell, is the normal "any available port" behaviour.
+Perhaps the source_address should accept an ip string *or* a tuple so that you don't need to specify the 0 (and the subtlety of free source ports and the magic 0 argument don't need to be understood by the users)
+.. code-block:: pycon
-Contribute
-----------
+ >>> r = requests.get('https://api.github.com', auth=('user', 'pass'), source_address=('127.0.0.1', 54444))
+ >>> r = requests.get('https://api.github.com', auth=('user', 'pass'), source_address=('127.0.0.1', 0))
+ >>> sess = requests.session()
+ >>> sess.source_address = ('127.0.0.1', 54444)
+ >>> sess.auth = ('user', 'pass')
+ >>> r = sess.get('https://api.github.com')
-#. Check for open issues or open a fresh issue to start a discussion around a feature idea or a bug. There is a Contributor Friendly tag for issues that should be ideal for people who are not very familiar with the codebase yet.
-#. Fork `the repository`_ on Github to start making your changes to the **master** branch (or branch off of it).
-#. Write a test which shows that the bug was fixed or that the feature works as expected.
-#. Send a pull request and bug the maintainer until it gets merged and published. :) Make sure to add yourself to AUTHORS_.
+I don't have good test coverage yet (and none at all checked in at the moment because I was using netcat to set up a fake server). I will work out a reasonable way to test this and then hopefully get this merged into requests.
-.. _`the repository`: http://github.com/kennethreitz/requests
-.. _AUTHORS: https://github.com/kennethreitz/requests/blob/master/AUTHORS.rst
+I'm definitely open to suggestions of more elegant ways to thread the source_address parameter through the session, adapter, connectionpool and connection objects that are in between the requests api and the underlying HTTPConnection. It's not bad right now but I have a feeling that someone with a better mental model of how Requests works might see a nicer way to thread this argument through the class hierarchy.
View
10 requests/adapters.py
@@ -130,16 +130,16 @@ def build_response(self, req, resp):
return response
- def get_connection(self, url, proxies=None):
+ def get_connection(self, url, proxies=None, source_address=None):
"""Returns a connection for the given URL."""
proxies = proxies or {}
proxy = proxies.get(urlparse(url).scheme)
if proxy:
proxy = prepend_scheme_if_needed(proxy, urlparse(url).scheme)
- conn = ProxyManager(self.poolmanager.connection_from_url(proxy))
+ conn = ProxyManager(self.poolmanager.connection_from_url(proxy, source_address=source_address))
else:
- conn = self.poolmanager.connection_from_url(url)
+ conn = self.poolmanager.connection_from_url(url, source_address=source_address)
return conn
@@ -185,10 +185,10 @@ def add_headers(self, request, **kwargs):
request.headers['Proxy-Authorization'] = _basic_auth_str(username,
password)
- def send(self, request, stream=False, timeout=None, verify=True, cert=None, proxies=None):
+ def send(self, request, stream=False, timeout=None, verify=True, cert=None, proxies=None, source_address=None):
"""Sends PreparedRequest object. Returns Response object."""
- conn = self.get_connection(request.url, proxies)
+ conn = self.get_connection(request.url, proxies, source_address=source_address)
self.cert_verify(conn, request.url, verify, cert)
url = self.request_url(request, proxies)
View
1  requests/api.py
@@ -32,6 +32,7 @@ def request(method, url, **kwargs):
:param verify: (optional) if ``True``, the SSL cert will be verified. A CA_BUNDLE path can also be provided.
:param stream: (optional) if ``False``, the response content will be immediately downloaded.
:param cert: (optional) if String, path to ssl client cert file (.pem). If Tuple, ('cert', 'key') pair.
+ :param source_address: (optional) ('source_IP', source_port) pair.
Usage::
View
19 requests/packages/urllib3/connectionpool.py
@@ -125,9 +125,10 @@ class ConnectionPool(object):
scheme = None
QueueCls = LifoQueue
- def __init__(self, host, port=None):
+ def __init__(self, host, port=None, source_address=None):
self.host = host
self.port = port
+ self.source_address = source_address
def __str__(self):
return '%s(host=%r, port=%r)' % (type(self).__name__,
@@ -175,9 +176,9 @@ class HTTPConnectionPool(ConnectionPool, RequestMethods):
scheme = 'http'
- def __init__(self, host, port=None, strict=False, timeout=None, maxsize=1,
+ def __init__(self, host, port=None, source_address=None, strict=False, timeout=None, maxsize=1,
block=False, headers=None):
- ConnectionPool.__init__(self, host, port)
+ ConnectionPool.__init__(self, host, port, source_address=source_address)
RequestMethods.__init__(self, headers)
self.strict = strict
@@ -202,7 +203,8 @@ def _new_conn(self):
(self.num_connections, self.host))
return HTTPConnection(host=self.host,
port=self.port,
- strict=self.strict)
+ strict=self.strict,
+ source_address=self.source_address)
def _get_conn(self, timeout=None):
"""
@@ -521,13 +523,14 @@ class HTTPSConnectionPool(HTTPConnectionPool):
scheme = 'https'
def __init__(self, host, port=None,
+ source_address=None,
strict=False, timeout=None, maxsize=1,
block=False, headers=None,
key_file=None, cert_file=None, cert_reqs=None,
ca_certs=None, ssl_version=None,
assert_hostname=None, assert_fingerprint=None):
- HTTPConnectionPool.__init__(self, host, port,
+ HTTPConnectionPool.__init__(self, host, port, source_address,
strict, timeout, maxsize,
block, headers)
self.key_file = key_file
@@ -553,11 +556,13 @@ def _new_conn(self):
return HTTPSConnection(host=self.host,
port=self.port,
- strict=self.strict)
+ strict=self.strict,
+ source_address=self.source_address)
connection = VerifiedHTTPSConnection(host=self.host,
port=self.port,
- strict=self.strict)
+ strict=self.strict,
+ source_address=self.source_address)
connection.set_cert(key_file=self.key_file, cert_file=self.cert_file,
cert_reqs=self.cert_reqs, ca_certs=self.ca_certs,
assert_hostname=self.assert_hostname,
View
14 requests/packages/urllib3/poolmanager.py
@@ -61,7 +61,7 @@ def __init__(self, num_pools=10, headers=None, **connection_pool_kw):
self.pools = RecentlyUsedContainer(num_pools,
dispose_func=lambda p: p.close())
- def _new_pool(self, scheme, host, port):
+ def _new_pool(self, scheme, host, port, source_address):
"""
Create a new :class:`ConnectionPool` based on host, port and scheme.
@@ -76,7 +76,7 @@ def _new_pool(self, scheme, host, port):
for kw in SSL_KEYWORDS:
kwargs.pop(kw, None)
- return pool_cls(host, port, **kwargs)
+ return pool_cls(host, port, source_address=source_address, **kwargs)
def clear(self):
"""
@@ -87,7 +87,7 @@ def clear(self):
"""
self.pools.clear()
- def connection_from_host(self, host, port=None, scheme='http'):
+ def connection_from_host(self, host, port=None, scheme='http', source_address=None):
"""
Get a :class:`ConnectionPool` based on the host, port, and scheme.
@@ -97,7 +97,7 @@ def connection_from_host(self, host, port=None, scheme='http'):
scheme = scheme or 'http'
port = port or port_by_scheme.get(scheme, 80)
- pool_key = (scheme, host, port)
+ pool_key = (scheme, host, port, source_address)
# If the scheme, host, or port doesn't match existing open connections,
# open a new ConnectionPool.
@@ -106,11 +106,11 @@ def connection_from_host(self, host, port=None, scheme='http'):
return pool
# Make a fresh ConnectionPool of the desired type
- pool = self._new_pool(scheme, host, port)
+ pool = self._new_pool(scheme, host, port, source_address)
self.pools[pool_key] = pool
return pool
- def connection_from_url(self, url):
+ def connection_from_url(self, url, source_address=None):
"""
Similar to :func:`urllib3.connectionpool.connection_from_url` but
doesn't pass any additional parameters to the
@@ -120,7 +120,7 @@ def connection_from_url(self, url):
constructor.
"""
u = parse_url(url)
- return self.connection_from_host(u.host, port=u.port, scheme=u.scheme)
+ return self.connection_from_host(u.host, port=u.port, scheme=u.scheme, source_address=source_address)
def urlopen(self, method, url, redirect=True, **kw):
"""
View
11 requests/sessions.py
@@ -196,6 +196,11 @@ def __init__(self):
#: :class:`Request <Request>`.
self.auth = None
+ #: Default Source Address tuple (Source IP, Source Port)
+ #: to attach to
+ #: :class:`Request <Request>`.
+ self.source_address = None
+
#: Dictionary mapping protocol to the URL of the proxy (e.g.
#: {'http': 'foo.bar:3128'}) to be used on each
#: :class:`Request <Request>`.
@@ -251,7 +256,9 @@ def request(self, method, url,
hooks=None,
stream=None,
verify=None,
- cert=None):
+ cert=None,
+ source_address=None):
+
"""Constructs a :class:`Request <Request>`, prepares it and sends it.
Returns :class:`Response <Response>` object.
@@ -321,6 +328,7 @@ def request(self, method, url,
params = merge_kwargs(params, self.params)
headers = merge_kwargs(headers, self.headers)
auth = merge_kwargs(auth, self.auth)
+ source_address = merge_kwargs(source_address, self.source_address)
proxies = merge_kwargs(proxies, self.proxies)
hooks = merge_kwargs(hooks, self.hooks)
stream = merge_kwargs(stream, self.stream)
@@ -350,6 +358,7 @@ def request(self, method, url,
'cert': cert,
'proxies': proxies,
'allow_redirects': allow_redirects,
+ 'source_address': source_address
}
resp = self.send(prep, **send_kwargs)
Something went wrong with that request. Please try again.