Add Option to Bind to a Local IP Address in httplib.py

[eldonzatlanticdbcom]

BPO	3972
Nosy	@gpshead, @pitrou, @giampaolo, @bitdancer
Files	httplib2.4.diff: Patch file httplibmod2.6.tar.gz: patch files and test file

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/gpshead'
closed_at = <Date 2010-01-03.02:07:07.022>
created_at = <Date 2008-09-26.09:58:50.737>
labels = ['easy', 'type-feature', 'library']
title = 'Add Option to Bind to a Local IP Address in httplib.py'
updated_at = <Date 2010-01-03.21:14:34.192>
user = 'https://bugs.python.org/eldonzatlanticdbcom'

bugs.python.org fields:

activity = <Date 2010-01-03.21:14:34.192>
actor = 'gregory.p.smith'
assignee = 'gregory.p.smith'
closed = True
closed_date = <Date 2010-01-03.02:07:07.022>
closer = 'gregory.p.smith'
components = ['Library (Lib)']
creation = <Date 2008-09-26.09:58:50.737>
creator = 'eldonz@atlanticdb.com'
dependencies = []
files = ['11607', '15687']
hgrepos = []
issue_num = 3972
keywords = ['patch', 'easy']
message_count = 18.0
messages = ['73840', '74184', '74185', '96963', '96964', '96966', '96968', '96970', '96971', '96973', '96974', '96975', '96976', '96982', '97154', '97162', '97167', '97179']
nosy_count = 6.0
nosy_names = ['gregory.p.smith', 'pitrou', 'giampaolo.rodola', 'eldonz@atlanticdb.com', 'r.david.murray', 'greg.hellings']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'test needed'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue3972'
versions = ['Python 2.7', 'Python 3.2']

[eldonzatlanticdbcom]

I updated httplib.py, python 2.4, to be able to bind to a specific IP
address when connecting to a remote site.

  conn = httplib.HTTPConnection('82.94.237.218', 80)

will connect to '82.94.237.218' using one of the local IP addresses. For
example, if a machine has an primary IP address and an alias such as

eth0 192.168.1.10
eth0:1 192.168.1.11

the outbound connection might use either eth0 or eth0:1. I'm not sure
how it picks now. I added a bind option both for http and https so we
can direct the connection through one or the other. For example,

  conn = httplib.HTTPConnection('82.94.237.218', 80, None, None, None,
'192.168.1.10')

would make sure it used 192.168.1.10 not 192.168.1.11.

I ran into this on a server that is contacted by an external legacy
server which requires a reverse connection over the same IP address that
the original connection came in on.

[gpshead]

This is useful but as its a feature we won't be able to add it until
2.7/3.1. Assigning to me to make sure it happens.

Until then, the approach I suggest is to subclass httplib.HTTPConnection
and HTTPSConnection to add your own variants that do bind before connecting.

[eldonzatlanticdbcom]

Sounds good. Let me know if there is anything I can do to help.

Eldon

On Thu, 2008-10-02 at 18:17 +0000, Gregory P. Smith wrote:

Gregory P. Smith <greg@krypto.org> added the comment:

This is useful but as its a feature we won't be able to add it until
2.7/3.1. Assigning to me to make sure it happens.

Until then, the approach I suggest is to subclass httplib.HTTPConnection
and HTTPSConnection to add your own variants that do bind before connecting.

----------
assignee: -> gregory.p.smith
nosy: +gregory.p.smith
priority: -> normal
versions: -Python 2.4, Python 2.5, Python 2.6, Python 3.0

---

Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue3972\>

---

[GregHellings]

Did this ever happen? It seems like overkill in the non-Python sort of
way to continue pointing people to over-riding classes and extending
objects when such a small patch adds so powerful and useful a
functionality to the library.

[bitdancer]

Since Gregory didn't update the ticket, it seems likely that it hasn't
happened yet. If you'd care to try the patch and report on your
results, that would help the process. We'll also need unit tests and
documentation updates, so patches for those would also speed the process.

[GregHellings]

Just looking at the indicated file in the 2.6.4 release tarball, it does
not seem that it would apply cleanly. The line numbers do not apply
properly anymore, though the edited lines themselves still appear to be
unaffected. Without context diff in the original patch, it's difficult
for me to asseess exactly which lines should be the affected ones.

Unfortunately, I'm not in a position right now with my job to spend the
time necessary to produce this as a patch since I'm on a timescale that
requires equivalent functionality by Wednesday on production systems.
Modifying it to apply cleanly to the latest versions of Python appears
like it would be easy, if the internals of that file have not changed
drastically in structure since 2.6.4.

Documentation should be relatively straightforward as well, since the
functionality the patch introduces is rather transparent. Unit tests
are beyond my expertise to comment on.

Would have loved to have seen this in the 2.7/3.1 series, as it would
make my task much easier! I'll keep it in mind for my "off time" this
holiday weekend.

[pitrou]

I'm not sure why this is needed in httplib. Isn't it a matter of
configuring the machine's routing tables properly?

[GregHellings]

For my own case, I have a machine with 50 IP addresses set and I need to
run a script to grab data that will randomly select one of those IP
addresses to use for its outgoing connection. That's something which
needs to be selected at the socket level, as best I understand the issue,
not on the routing end.

[pitrou]

For my own case, I have a machine with 50 IP addresses set and I need to
run a script to grab data that will randomly select one of those IP
addresses to use for its outgoing connection. That's something which
needs to be selected at the socket level, as best I understand the issue,
not on the routing end.

Why do you need to select manually between those 50 IP addresses?
Load-balancing? Fooling the remote server so that you don't get
blacklisted for opening too many connections?

The problem I see with adding this to httplib is that it's not
HTTP-specific. Other modules (smtplib etc.) might want to benefit,
depending on the use case.

By the way, what I mean by routing is your OS' internal routing table
for outbound packets. For example under Linux:

# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
X.Y.223.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
X.Y.222.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
0.0.0.0 X.Y.222.254 0.0.0.0 UG 100 0 0 eth0

The interface for each outbound route is part of the routing table.

[bitdancer]

You are correct that more than just httplib may need this; however, it
is a real need. Consider for example a machine that has multiple IPs on
the same network (perhaps there used to be two machines but the services
were consolidated onto one or something), and some other server it talks
to is using IP based security in its .htaccess file. So the client
machine needs to bind to the IP address that the server machine has
authorized to access the web service. This is why, for example, the ssh
command has a '-b' option to specify the port to which the client binds.
Or, for an even more apropos example, wget has a --bind-address option
for this reason.

[giampaolo]

In case it helps, a guy recently reported the same issue for pyftpdlib:
http://code.google.com/p/pyftpdlib/issues/detail?id=123

[eldonzatlanticdbcom]

The patch files for Python 2.6 and a test file are in the attached tar
file. Both httplib.py and socket.py need to be patched now.

httplib.HTTPConnection(host[, port[, strict[, timeout]]])

becomes

httplib.HTTPConnection(host[, port[, strict[, timeout[, bindip]]]])

and

httplib.HTTPSConnection(host[, port[, key_file[, cert_file[, strict[,
timeout]]]]])

becomes

httplib.HTTPSConnection(host[, port[, key_file[, cert_file[, strict[,
timeout[, bindip]]]]]])

[bitdancer]

If you want the maximum chance for this to get applied, it would be best
to have one or a set of patch files in 'svn diff' format against trunk
(see http://python.org/dev for information on how to produce such a
patch, specifically section 6.1 of the dev FAQ). The tests will need to
be in unittest format (they would go in test_httplib and/or test_socket).

I imagine that writing the unittests so they will run on any machine
might be a tad tricky. If I were doing it I'd try binding to 127.0.0.1
and checking to see that the socket is bound to that address rather than
0.0.0.0 in the binaddr case, and that it is bound to 0.0.0.0 in the
default case.

[gpshead]

yes its a real need and yes we should support this in the standard
library. no i have not had time to look at it since my comment a 14
months ago.

Thanks for the updated patch Eldon!

[gpshead]

trunk r77263 and r77264 add this feature, including documentation and tests.

[pitrou]

It seems to break at least one buildbot:

======================================================================
ERROR: testSourceAddress (test.test_socket.NetworkConnectionAttributesTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/var/lib/buildslave/trunk.murray-gentoo-wide/build/Lib/test/test_socket.py", line 120, in _tearDown
    self.fail(msg)
AssertionError: Tuples differ: ('199.125.120.107', 40732) != ('127.0.0.1', 40732)

First differing element 0:
199.125.120.107
127.0.0.1

• ('199.125.120.107', 40732)
  + ('127.0.0.1', 40732)

----------------------------------------------------------------------
Ran 100 tests in 11.505s

By the way, does the new "source_address" parameter have to be of the
same family as the target address? That is, if I pass e.g. "127.0.0.1"
as source address, what happens if the target host resolves to an IPv6
address?

[bitdancer]

Ah, and I was even the one that suggested the bind to 127.0.0.1 strategy for the test :(

My buildbots run in linux-vserver virtual hosts, and they way they handle localhost (127.0.0.1) is to alias it to the IP address assigned to the virthost (otherwise multiple virthosts opening ports on 127.0.0.1 would potentially conflict with each other).

I'd consider this a bug in the virthost software (lack of feature?). Not sure what the best workaround is, but if all else fails we could put in some special case skip code.

Would be good enough to have the test check that the returned IP address is *not* 0.0.0.0? Or perhaps issue a warning if the address is not 0.0.0.0 but isn't 127.0.0.1.

[gpshead]

I took the easy route and remove the test of the hostname all together. The fact that the source port was used is sufficient indication that the bind call was made.