urllib2's urlopen() method causes a memory leak

[manekcz]

BPO	1208304
Nosy	@akuchling, @amauryfa, @orsenthil, @pitrou, @abadger
Files	urllib2leak.py: main test urllib2.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2013-04-11.09:27:26.277>
created_at = <Date 2005-05-25.09:20:22.000>
labels = ['invalid', 'library', 'performance']
title = "urllib2's urlopen() method causes a memory leak"
updated_at = <Date 2013-04-11.13:24:20.111>
user = 'https://bugs.python.org/manekcz'

bugs.python.org fields:

activity = <Date 2013-04-11.13:24:20.111>
actor = 'BreamoreBoy'
assignee = 'none'
closed = True
closed_date = <Date 2013-04-11.09:27:26.277>
closer = 'pitrou'
components = ['Library (Lib)']
creation = <Date 2005-05-25.09:20:22.000>
creator = 'manekcz'
dependencies = []
files = ['14171', '14827']
hgrepos = []
issue_num = 1208304
keywords = []
message_count = 21.0
messages = ['60743', '60744', '60745', '60746', '60747', '60748', '60749', '60750', '60751', '76298', '76300', '76350', '76368', '76683', '88812', '92245', '114503', '186550', '186552', '186556', '186560']
nosy_count = 16.0
nosy_names = ['jhylton', 'akuchling', 'holdenweb', 'jafo', 'amaury.forgeotdarc', 'orsenthil', 'pitrou', 'bwelling', 'schmir', 'stephbul', 'manekcz', 'nswinton', 'a.badger', 'peci', 'BreamoreBoy', 'gdub']
pr_nums = []
priority = 'normal'
resolution = 'not a bug'
stage = None
status = 'closed'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue1208304'
versions = ['Python 2.6', 'Python 3.1', 'Python 2.7', 'Python 3.2']

[manekcz]

It seems that the urlopen(url) methd of the urllib2 module
leaves some undestroyable objects in memory.

Please try the following code:
==========================

if __name__ == '__main__':
  import urllib2
  a = urllib2.urlopen('http://www.google.com')
  del a # or a = None or del(a)
  
  # check memory on memory leaks
  import gc
  gc.set_debug(gc.DEBUG_SAVEALL)
  gc.collect()
  for it in gc.garbage:
    print it

==========================

In our code, we're using lots of urlopens in a loop and
the number of unreachable objects grows beyond all
limits :) We also tried a.close() but it didn't help.

You can also try the following:
==========================

def print_unreachable_len():
  # check memory on memory leaks
  import gc
  gc.set_debug(gc.DEBUG_SAVEALL)
  gc.collect()
  unreachableL = []
  for it in gc.garbage:
    unreachableL.append(it)
  return len(str(unreachableL))
  
if __name__ == '__main__':
  print "at the beginning", print_unreachable_len()

  import urllib2
  print "after import of urllib2", print_unreachable_len()

  a = urllib2.urlopen('http://www.google.com')
  print 'after urllib2.urlopen', print_unreachable_len()

del a
print 'after del', print_unreachable_len()
==========================

We're using WindowsXP with latest patches, Python 2.4
(ActivePython 2.4 Build 243 (ActiveState Corp.) based on
Python 2.4 (#60, Nov 30 2004, 09:34:21) [MSC v.1310
32 bit (Intel)] on win32).

[akuchling]

Logged In: YES
user_id=11375

Confirmed. The objects involved seem to be an HTTPResponse and the
socket._fileobject wrapper; the assignment 'r.recv=r.read' around line 1013
of urllib2.py seems to be critical to creating the cycle.

[jafo]

Logged In: YES
user_id=81797

I can reproduce this in both the python.org 2.4 RPM and in a
freshly built copy from CVS. If I run a few thousand
urlopen()s, I get:

Traceback (most recent call last):
  File "/tmp/mt", line 26, in ?
  File "/tmp/python/dist/src/Lib/urllib2.py", line 130, in
urlopen
  File "/tmp/python/dist/src/Lib/urllib2.py", line 361, in open
  File "/tmp/python/dist/src/Lib/urllib2.py", line 379, in _open
  File "/tmp/python/dist/src/Lib/urllib2.py", line 340, in
_call_chain
  File "/tmp/python/dist/src/Lib/urllib2.py", line 1026, in
http_open
  File "/tmp/python/dist/src/Lib/urllib2.py", line 1001, in
do_open
urllib2.URLError: <urlopen error (24, 'Too many open files')>

Even if I do a a.close(). I'll investigate a bit further.

Sean

[jafo]

Logged In: YES
user_id=81797

I give up, this code is kind of a maze of twisty little
passages. I did try doing "a.fp.close()" and that didn't
seem to help at all. Couldn't really make any progress on
that though. I also tried doing a "if a.headers.fp:
a.headers.fp.close()", which didn't do anything noticable.

[bwelling]

Logged In: YES
user_id=63197

We just ran into this same problem, and worked around it by
simply removing the 'r.recv = r.read' line in urllib2.py,
and creating a recv alias to the read function in
HTTPResponse ('recv = read' in the class).

Not sure if this is the best solution, but it seems to work.

[jafo]

Logged In: YES
user_id=81797

I've just tried it again using the current CVS version as
well as the version installed with Fedora Core 4, and in
both cases I was able to run over 100,000 retrievals of
http://127.0.0.1/test.html and http://127.0.0.1/google.html.
test.html is just "it works" and google.html was generated
with "wget -O google.html http://google.com/".

I was able to reproduce this before, but now am not. My
urllib2.py includes the r.recv=r.read line. I have upgraded
from FC3 to FC4, could this be something related to an OS or
library interaction? I was going to try to confirm the last
message, but now I can't reproduce the failure.

[bwelling]

Logged In: YES
user_id=63197

The real problem we were seeing wasn't the memory leak, it
was a file descriptor leak. Leaking references within the
interpreter is bad, but the garbage collector will
eventually notice that the system is out of memory and clean
them. Leaking file descriptors is much worse, as gc won't
be triggered when the process has reached it's limit, and
the process will start failing with "Too many file descriptors".

To easily show this problem, run the following from an
interactive python interpreter:

import urllib2
f = urllib2.urlopen('http://www.google.com')
f.close()

and from another window, run "lsof -p <pid of interpreter>".
It should show a TCP socket in CLOSE_WAIT, which means the
file descriptor is still open. I'm seeing weirdness on
Fedora Core 4 today that I didn't see last week where after
a few seconds, the file descriptor is listed as "can't
identify protocol" instead of TCP, but that's not too
relevant, since it's still open.

Repeating the urllib2.urlopen()/close() pairs of statements
in the interpreter will cause more fds to be leaked, which
can also be seen by lsof.

[holdenweb]

Logged In: YES
user_id=88157

The Windows 2.4.1 build doesn't show this error, but the
Cygwin 2.4.1 build does still have uncollectable objects
after a urllib2.urlopen(), so there may be a platform
dependency here. No 2.4.2 on Cygwin yet, so nothing
conclusive as lsof isn't available.

[nswinton]

Logged In: YES
user_id=1363935

It's not the prettiest thing, but you can work around this
by setting the socket's recv method to None before closing it.

import urllib2
f = urllib2.urlopen('http://www.google.com')
text=f.read()
f.fp._sock.recv=None # hacky avoidance
f.close()

[abadger]

I tried to repeat the test in http://bugs.python.org/msg60749 and found
that the descriptors will close if you read from the file before closing.

so this leads to open descriptors::

  import urllib2
  f = urllib2.urlopen('http://www.google.com')
  f.close()

while this does not::

  import urllib2
  f = urllib2.urlopen('http://www.google.com')
  f.read(1)
  f.close()

[abadger]

One further data point. On two rhel5 systems with identical kernels,
both x86_64, both python-2.4.3... basically, everything I've thought to
check identical, I ran the test code with f.read() in an infinite loop.
One system only has one TCP socket in use at a time. The other one has
multiple TCP sockets in use, but they all close eventually.

/usr/sbin/lsof -p INTERPRETER_PID|wc -l reported

96 67 97 63 91 62 94 78

on subsequent runs.

[jhylton]

Python 2.4 is now in security-fix-only mode. No new features are being
added, and bugs are not fixed anymore unless they affect the stability
and security of the interpreter, or of Python applications.
http://www.python.org/download/releases/2.4.5/

This bug doesn't rise to the level of making into a 2.4.6.