popen4 doesn't close filedescriptors when in Threads

[mdoudoroff]

BPO	768649
Nosy	@gaul, @facundobatista

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2005-01-15.20:48:13.000>
created_at = <Date 2003-07-09.18:36:58.000>
labels = ['library']
title = "popen4 doesn't close filedescriptors when in Threads"
updated_at = <Date 2005-01-15.20:48:13.000>
user = 'https://bugs.python.org/mdoudoroff'

bugs.python.org fields:

activity = <Date 2005-01-15.20:48:13.000>
actor = 'facundobatista'
assignee = 'none'
closed = True
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2003-07-09.18:36:58.000>
creator = 'mdoudoroff'
dependencies = []
files = []
hgrepos = []
issue_num = 768649
keywords = []
message_count = 5.0
messages = ['17009', '17010', '17011', '17012', '17013']
nosy_count = 4.0
nosy_names = ['nnorwitz', 'gaul', 'facundobatista', 'mdoudoroff']
pr_nums = []
priority = 'normal'
resolution = 'wont fix'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue768649'
versions = ['Python 2.2']

[mdoudoroff]

Running the following code under Linux will result in a
crash on the 508th thread started. The error is

OSError: [Errno 24] Too many open files

The nature of the bug seems to be that Python isn't
closing filedescriptors cleanly when running a thread.

---------------------------------------

import os
from threading import Thread

class Crash(Thread):
    def run(self):
        a = os.popen4('ls')
        b = a[1].read()

    # uncommenting these lines fixes the problem
    #     but this isn't really documented as far as
    #     we can tell...
    # a[0].close()
    # a[1].close()

for i in range(1000):
    t = Crash()
    t.start()

    while t.isAlive():
         pass

print i

---------------------------------------

The same code without threads (Crash as a plain class)
doesn't crash, so the descriptor must be getting taken
care of when the run() method is exited.

import os

class Crash:
    def run(self):
        a = os.popen4('ls')
        b = a[1].read()

for i in range(1000):
    t = Crash()
    t.run()

print i

[nnorwitz]

Logged In: YES
user_id=33168

I can't duplicate this on Redhat 9. What OS, what version
of glibc and what kernel are you using? Does it always
crash on the 508th iteration?

I tested with both 2.2.3 and 2.3b2 from CVS without
problems. I even used ulimit to set my open files to 10.

Can you try the patch in bug bpo-761888 to see if that helps?
http://pythong.org/sf/761888

[gaul]

Logged In: YES
user_id=139865

Duplicated with Python 2.3 on Red Hat 7.3 using
glibc-2.2.5-43. Popen3.{poll,wait} are written under the
incorrect assumption that waitpid can monitor any process in
the same process group, when it only works for immediate
children. _active.remove is never called, so Popen3 objects
are never destroyed and the associated file descriptors are
not returned to the operating system.

A general solution for Popen[34] is not obvious to me. With
patch bpo-816059, popen2.popen[234] plugs the _active leak,
which in turn returns the file descriptors to the operating
system when the file objects that popen2.popen[234] return
go out of scope.

[facundobatista]

Logged In: YES
user_id=752496

Please, could you verify if this problem persists in Python 2.3.4
or 2.4?

If yes, in which version? Can you provide a test case?

If the problem is solved, from which version?

Note that if you fail to answer in one month, I'll close this bug
as "Won't fix".

Thank you!

. Facundo

[facundobatista]

Logged In: YES
user_id=752496

Works fine to me:

Python 2.3.4 (#1, Oct 26 2004, 16:42:40)
[GCC 3.4.2 20041017 (Red Hat 3.4.2-6.fc3)] on linux2

with glibc-2.3.4-2