Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

source_address support (rough) #1288

Closed
wants to merge 5 commits into from

2 participants

Jesse Sherlock Kenneth Reitz
Jesse Sherlock

First pass on source_address support (2.7 only).

Kenneth Reitz
Owner

This is what connection adapters are for.

Jesse Sherlock

Thanks, I'll have a look at connection adapters

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Apr 3, 2013
  1. Jesse Sherlock
  2. Jesse Sherlock
  3. Jesse Sherlock
  4. Jesse Sherlock
Commits on Apr 4, 2013
  1. Jesse Sherlock

    HTTPSConnection supports source_address

    jessesherlock authored
    bit inconsistent with argument ordering I think, worth going back to
    double check that at some point
This page is out of date. Refresh to see the latest.
1  AUTHORS.rst
View
@@ -123,3 +123,4 @@ Patches and Suggestions
- Denis Ryzhkov <denisr@denisr.com>
- Wilfred Hughes <me@wilfred.me.uk> @dontYetKnow
- Dmitry Medvinsky <me@dmedvinsky.name>
+- Jesse Sherlock <jesse@jessesherlock.com>
89 README.rst
View
@@ -1,82 +1,31 @@
-Requests: HTTP for Humans
-=========================
+Requests: HTTP for Humans (now with source_address support)
+===========================================================
+Fork of Kenneth Reitz's fantastic Requests library with source_address support (and a forked version of the included urllib3 that's included with requests).
-.. image:: https://travis-ci.org/kennethreitz/requests.png?branch=master
- :target: https://travis-ci.org/kennethreitz/requests
+This means that this fork is only compatible with Python 2.7, when HTTPConnection added source_address support (and I have not yet tested it on 3.3). This first draft is a little specific to my Python 2.7 needs and I'll wait for feedback before I clean things up.
-Requests is an Apache2 Licensed HTTP library, written in Python, for human
-beings.
+Find the real documentation here: https://github.com/kennethreitz/requests
-Most existing Python modules for sending HTTP requests are extremely
-verbose and cumbersome. Python's builtin urllib2 module provides most of
-the HTTP capabilities you should need, but the api is thoroughly broken.
-It requires an enormous amount of work (even method overrides) to
-perform the simplest of tasks.
+I've removed the docs from this fork because I haven't had time to update all of them and no docs is better than docs that are wrong.
-Things shouldn't be this way. Not in Python.
-
-.. code-block:: pycon
-
- >>> r = requests.get('https://api.github.com', auth=('user', 'pass'))
- >>> r.status_code
- 204
- >>> r.headers['content-type']
- 'application/json'
- >>> r.text
- ...
-
-See `the same code, without Requests <https://gist.github.com/973705>`_.
-
-Requests allow you to send HTTP/1.1 requests. You can add headers, form data,
-multipart files, and parameters with simple Python dictionaries, and access the
-response data in the same way. It's powered by httplib and `urllib3
-<https://github.com/shazow/urllib3>`_, but it does all the hard work and crazy
-hacks for you.
-
-
-Features
---------
-
-- International Domains and URLs
-- Keep-Alive & Connection Pooling
-- Sessions with Cookie Persistence
-- Browser-style SSL Verification
-- Basic/Digest Authentication
-- Elegant Key/Value Cookies
-- Automatic Decompression
-- Unicode Response Bodies
-- Multipart File Uploads
-- Connection Timeouts
-- Thread-safety
+source_address is a kwarg on request building methods (get/post/put/etc)
+session objects have a source_address attribute that can be set on the session (and overridden on an individual request like all the other request related session object attributes).
-Installation
-------------
-
-To install requests, simply:
-
-.. code-block:: bash
-
- $ pip install requests
-
-Or, if you absolutely must:
-
-.. code-block:: bash
-
- $ easy_install requests
-
-But, you really shouldn't do that.
+If you pass in 0 for the port then the underlying HTTPConnection object falls back to "default behaviour" which, as far as I can tell, is the normal "any available port" behaviour.
+Perhaps the source_address should accept an ip string *or* a tuple so that you don't need to specify the 0 (and the subtlety of free source ports and the magic 0 argument don't need to be understood by the users)
+.. code-block:: pycon
-Contribute
-----------
+ >>> r = requests.get('https://api.github.com', auth=('user', 'pass'), source_address=('127.0.0.1', 54444))
+ >>> r = requests.get('https://api.github.com', auth=('user', 'pass'), source_address=('127.0.0.1', 0))
+ >>> sess = requests.session()
+ >>> sess.source_address = ('127.0.0.1', 54444)
+ >>> sess.auth = ('user', 'pass')
+ >>> r = sess.get('https://api.github.com')
-#. Check for open issues or open a fresh issue to start a discussion around a feature idea or a bug. There is a Contributor Friendly tag for issues that should be ideal for people who are not very familiar with the codebase yet.
-#. Fork `the repository`_ on Github to start making your changes to the **master** branch (or branch off of it).
-#. Write a test which shows that the bug was fixed or that the feature works as expected.
-#. Send a pull request and bug the maintainer until it gets merged and published. :) Make sure to add yourself to AUTHORS_.
+I don't have good test coverage yet (and none at all checked in at the moment because I was using netcat to set up a fake server). I will work out a reasonable way to test this and then hopefully get this merged into requests.
-.. _`the repository`: http://github.com/kennethreitz/requests
-.. _AUTHORS: https://github.com/kennethreitz/requests/blob/master/AUTHORS.rst
+I'm definitely open to suggestions of more elegant ways to thread the source_address parameter through the session, adapter, connectionpool and connection objects that are in between the requests api and the underlying HTTPConnection. It's not bad right now but I have a feeling that someone with a better mental model of how Requests works might see a nicer way to thread this argument through the class hierarchy.
10 requests/adapters.py
View
@@ -130,16 +130,16 @@ def build_response(self, req, resp):
return response
- def get_connection(self, url, proxies=None):
+ def get_connection(self, url, proxies=None, source_address=None):
"""Returns a connection for the given URL."""
proxies = proxies or {}
proxy = proxies.get(urlparse(url).scheme)
if proxy:
proxy = prepend_scheme_if_needed(proxy, urlparse(url).scheme)
- conn = ProxyManager(self.poolmanager.connection_from_url(proxy))
+ conn = ProxyManager(self.poolmanager.connection_from_url(proxy, source_address=source_address))
else:
- conn = self.poolmanager.connection_from_url(url)
+ conn = self.poolmanager.connection_from_url(url, source_address=source_address)
return conn
@@ -185,10 +185,10 @@ def add_headers(self, request, **kwargs):
request.headers['Proxy-Authorization'] = _basic_auth_str(username,
password)
- def send(self, request, stream=False, timeout=None, verify=True, cert=None, proxies=None):
+ def send(self, request, stream=False, timeout=None, verify=True, cert=None, proxies=None, source_address=None):
"""Sends PreparedRequest object. Returns Response object."""
- conn = self.get_connection(request.url, proxies)
+ conn = self.get_connection(request.url, proxies, source_address=source_address)
self.cert_verify(conn, request.url, verify, cert)
url = self.request_url(request, proxies)
1  requests/api.py
View
@@ -32,6 +32,7 @@ def request(method, url, **kwargs):
:param verify: (optional) if ``True``, the SSL cert will be verified. A CA_BUNDLE path can also be provided.
:param stream: (optional) if ``False``, the response content will be immediately downloaded.
:param cert: (optional) if String, path to ssl client cert file (.pem). If Tuple, ('cert', 'key') pair.
+ :param source_address: (optional) ('source_IP', source_port) pair.
Usage::
19 requests/packages/urllib3/connectionpool.py
View
@@ -125,9 +125,10 @@ class ConnectionPool(object):
scheme = None
QueueCls = LifoQueue
- def __init__(self, host, port=None):
+ def __init__(self, host, port=None, source_address=None):
self.host = host
self.port = port
+ self.source_address = source_address
def __str__(self):
return '%s(host=%r, port=%r)' % (type(self).__name__,
@@ -175,9 +176,9 @@ class HTTPConnectionPool(ConnectionPool, RequestMethods):
scheme = 'http'
- def __init__(self, host, port=None, strict=False, timeout=None, maxsize=1,
+ def __init__(self, host, port=None, source_address=None, strict=False, timeout=None, maxsize=1,
block=False, headers=None):
- ConnectionPool.__init__(self, host, port)
+ ConnectionPool.__init__(self, host, port, source_address=source_address)
RequestMethods.__init__(self, headers)
self.strict = strict
@@ -202,7 +203,8 @@ def _new_conn(self):
(self.num_connections, self.host))
return HTTPConnection(host=self.host,
port=self.port,
- strict=self.strict)
+ strict=self.strict,
+ source_address=self.source_address)
def _get_conn(self, timeout=None):
"""
@@ -521,13 +523,14 @@ class HTTPSConnectionPool(HTTPConnectionPool):
scheme = 'https'
def __init__(self, host, port=None,
+ source_address=None,
strict=False, timeout=None, maxsize=1,
block=False, headers=None,
key_file=None, cert_file=None, cert_reqs=None,
ca_certs=None, ssl_version=None,
assert_hostname=None, assert_fingerprint=None):
- HTTPConnectionPool.__init__(self, host, port,
+ HTTPConnectionPool.__init__(self, host, port, source_address,
strict, timeout, maxsize,
block, headers)
self.key_file = key_file
@@ -553,11 +556,13 @@ def _new_conn(self):
return HTTPSConnection(host=self.host,
port=self.port,
- strict=self.strict)
+ strict=self.strict,
+ source_address=self.source_address)
connection = VerifiedHTTPSConnection(host=self.host,
port=self.port,
- strict=self.strict)
+ strict=self.strict,
+ source_address=self.source_address)
connection.set_cert(key_file=self.key_file, cert_file=self.cert_file,
cert_reqs=self.cert_reqs, ca_certs=self.ca_certs,
assert_hostname=self.assert_hostname,
14 requests/packages/urllib3/poolmanager.py
View
@@ -61,7 +61,7 @@ def __init__(self, num_pools=10, headers=None, **connection_pool_kw):
self.pools = RecentlyUsedContainer(num_pools,
dispose_func=lambda p: p.close())
- def _new_pool(self, scheme, host, port):
+ def _new_pool(self, scheme, host, port, source_address):
"""
Create a new :class:`ConnectionPool` based on host, port and scheme.
@@ -76,7 +76,7 @@ def _new_pool(self, scheme, host, port):
for kw in SSL_KEYWORDS:
kwargs.pop(kw, None)
- return pool_cls(host, port, **kwargs)
+ return pool_cls(host, port, source_address=source_address, **kwargs)
def clear(self):
"""
@@ -87,7 +87,7 @@ def clear(self):
"""
self.pools.clear()
- def connection_from_host(self, host, port=None, scheme='http'):
+ def connection_from_host(self, host, port=None, scheme='http', source_address=None):
"""
Get a :class:`ConnectionPool` based on the host, port, and scheme.
@@ -97,7 +97,7 @@ def connection_from_host(self, host, port=None, scheme='http'):
scheme = scheme or 'http'
port = port or port_by_scheme.get(scheme, 80)
- pool_key = (scheme, host, port)
+ pool_key = (scheme, host, port, source_address)
# If the scheme, host, or port doesn't match existing open connections,
# open a new ConnectionPool.
@@ -106,11 +106,11 @@ def connection_from_host(self, host, port=None, scheme='http'):
return pool
# Make a fresh ConnectionPool of the desired type
- pool = self._new_pool(scheme, host, port)
+ pool = self._new_pool(scheme, host, port, source_address)
self.pools[pool_key] = pool
return pool
- def connection_from_url(self, url):
+ def connection_from_url(self, url, source_address=None):
"""
Similar to :func:`urllib3.connectionpool.connection_from_url` but
doesn't pass any additional parameters to the
@@ -120,7 +120,7 @@ def connection_from_url(self, url):
constructor.
"""
u = parse_url(url)
- return self.connection_from_host(u.host, port=u.port, scheme=u.scheme)
+ return self.connection_from_host(u.host, port=u.port, scheme=u.scheme, source_address=source_address)
def urlopen(self, method, url, redirect=True, **kw):
"""
11 requests/sessions.py
View
@@ -196,6 +196,11 @@ def __init__(self):
#: :class:`Request <Request>`.
self.auth = None
+ #: Default Source Address tuple (Source IP, Source Port)
+ #: to attach to
+ #: :class:`Request <Request>`.
+ self.source_address = None
+
#: Dictionary mapping protocol to the URL of the proxy (e.g.
#: {'http': 'foo.bar:3128'}) to be used on each
#: :class:`Request <Request>`.
@@ -251,7 +256,9 @@ def request(self, method, url,
hooks=None,
stream=None,
verify=None,
- cert=None):
+ cert=None,
+ source_address=None):
+
"""Constructs a :class:`Request <Request>`, prepares it and sends it.
Returns :class:`Response <Response>` object.
@@ -321,6 +328,7 @@ def request(self, method, url,
params = merge_kwargs(params, self.params)
headers = merge_kwargs(headers, self.headers)
auth = merge_kwargs(auth, self.auth)
+ source_address = merge_kwargs(source_address, self.source_address)
proxies = merge_kwargs(proxies, self.proxies)
hooks = merge_kwargs(hooks, self.hooks)
stream = merge_kwargs(stream, self.stream)
@@ -350,6 +358,7 @@ def request(self, method, url,
'cert': cert,
'proxies': proxies,
'allow_redirects': allow_redirects,
+ 'source_address': source_address
}
resp = self.send(prep, **send_kwargs)
Something went wrong with that request. Please try again.