circular reference in HTTPResponse by urllib2

[kristjanvalur]

BPO	7464
Nosy	@amauryfa, @orsenthil, @pitrou, @kristjanvalur, @vadmium, @serhiy-storchaka, @iritkatriel
Files	httpleak.py: file that demonstrates HTTPResponse leak

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/orsenthil'
closed_at = <Date 2021-06-18.11:07:14.501>
created_at = <Date 2009-12-09.17:27:12.055>
labels = ['library', 'performance']
title = 'circular reference in HTTPResponse by urllib2'
updated_at = <Date 2021-06-18.11:07:14.500>
user = 'https://github.com/kristjanvalur'

bugs.python.org fields:

activity = <Date 2021-06-18.11:07:14.500>
actor = 'iritkatriel'
assignee = 'orsenthil'
closed = True
closed_date = <Date 2021-06-18.11:07:14.501>
closer = 'iritkatriel'
components = ['Library (Lib)']
creation = <Date 2009-12-09.17:27:12.055>
creator = 'kristjan.jonsson'
dependencies = []
files = ['33205']
hgrepos = []
issue_num = 7464
keywords = ['patch']
message_count = 15.0
messages = ['96175', '96224', '96229', '96235', '96239', '96241', '110930', '112737', '113112', '206502', '206509', '206596', '207020', '207093', '396047']
nosy_count = 8.0
nosy_names = ['amaury.forgeotdarc', 'orsenthil', 'pitrou', 'kristjan.jonsson', 'dstanek', 'martin.panter', 'serhiy.storchaka', 'iritkatriel']
pr_nums = []
priority = 'normal'
resolution = 'out of date'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue7464'
versions = ['Python 2.7']

[kristjanvalur]

in urllib2, you will find these lines:
# Wrap the HTTPResponse object in socket's file object adapter
# for Windows. That adapter calls recv(), so delegate recv()
# to read(). This weird wrapping allows the returned object to
# have readline() and readlines() methods.

    # XXX It might be better to extract the read buffering code
    # out of socket._fileobject() and into a base class.

        r.recv = r.read
        fp = socket._fileobject(r, close=True)

This, storing a bound method in the instance, will cause a reference
cycle that the user knows nothing about.

I propose creating a wrapper instance with a recv() method instead. Or,
is there a standard way of storing bound methods on instances? A
'weakmethod', perhaps?

[kristjanvalur]

I have two solutions for this problem. The first is a mundane one, and
what I employed in our production environment:
class RecvAdapter(object):
def __init__(self, wrapped):
self.wrapped = wrapped
def recv(self, amt):
return self.wrapped.read(amt)
def close(self):
self.wrapped.close()

...
fp = socket._fileobject(RecvAdapter(r), close=True)

The second solution is a bit more interesting. It involves applying
what I call a weakmethod: A bound method that holds a weak ref to the
object instance:

import weakref
class WeakMethod(object):
    def __init__(self, bound):
        self.weakself = weakref.proxy(bound.im_self)
        self.methodname = bound.im_func.func_name
    def __call__(self, *args, **kw):
        return getattr(self.weakself, self.methodname)(*args, **kw)

We then do:
r.recv = WeakMethod(r.read)
fp = socket._fileobject(r, close=True)

I've had many uses for a WeakMethod through the years. I wonder if such
a class might be considered useful enough to be put into the weakref
module.

[amauryfa]

The WeakMethod idea is not new:

http://code.activestate.com/recipes/81253/

http://mindtrove.info/articles/python-weak-references/#toc-extending-weak-
references

[orsenthil]

weak method idea seems interesting. I have not used it anytime yet, and
in this case,it seems okay.

[kristjanvalur]

cool. The mindtrove one in particular seems nice. I didn't realize
that one could build boundobjects oneself.
Is there any chance of getting a weakmethod class into the weakref
module?

[orsenthil]

On Fri, Dec 11, 2009 at 09:40:40AM +0000, Kristján Valur Jónsson wrote:

Is there any chance of getting a weakmethod class into the weakref
module?

This is a separate feature request on weakref module. It may opened
and discussion carried out there?

[orsenthil]

Lets just investigate the circular reference part here for this ticket.

[dstanek]

Does this issue still exist? I did a little poking around at could not find the quoted code.

[kristjanvalur]

in python/trunk/Lib/urllib2.py, line 1161

It doesn't appear to be an issue in py3k.

[kristjanvalur]

This is still a horrible, horrible, cludge.
I've recently done some work in this area and will suggest a different approach.

[serhiy-storchaka]

Could you please provide some tests?

[kristjanvalur]

Here it is.
Notice the incredible nesting depth in python 2.7.
The socket itself is found at
response.fp._sock.fp._sock
There are two socket._fileobjects in use!