multiprocessing Queue leaks a file descriptor associated with the pipe writer

[thehcma]

BPO	33081
Nosy	@pitrou, @applio, @thehcma, @pablogsal

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-03-23.16:57:13.221>
created_at = <Date 2018-03-15.18:05:17.001>
labels = ['library']
title = 'multiprocessing Queue leaks a file descriptor associated with the pipe writer'
updated_at = <Date 2019-03-22.07:47:37.646>
user = 'https://github.com/thehcma'

bugs.python.org fields:

activity = <Date 2019-03-22.07:47:37.646>
actor = 'fossilet'
assignee = 'none'
closed = True
closed_date = <Date 2018-03-23.16:57:13.221>
closer = 'pitrou'
components = ['Library (Lib)']
creation = <Date 2018-03-15.18:05:17.001>
creator = 'Henrique Andrade'
dependencies = []
files = []
hgrepos = []
issue_num = 33081
keywords = []
message_count = 21.0
messages = ['313899', '313900', '313921', '313922', '313924', '313926', '313927', '314014', '314037', '314043', '314045', '314047', '314048', '314049', '314050', '314066', '314068', '314316', '334640', '334641', '338588']
nosy_count = 6.0
nosy_names = ['pitrou', 'fossilet', 'davin', 'Henrique Andrade', 'pablogsal', 'Chris Langton']
pr_nums = []
priority = 'normal'
resolution = 'wont fix'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue33081'
versions = ['Python 2.7']

[thehcma]

A simple example like such demonstrates that one of the file descriptors associated with the underlying pipe will be leaked:

>> from multiprocessing.queues import Queue
>> x = Queue()
>> x.close()

Right after the queue is created we get (assuming the Python interpreter is associated with pid 8096 below):

ll /proc/8096/fd
total 0
dr-x------ 2 hcma hcma 0 2018-03-15 14:03:23.210089578 -0400 .
dr-xr-xr-x 9 hcma hcma 0 2018-03-15 14:03:23.190089760 -0400 ..
lrwx------ 1 hcma hcma 64 2018-03-15 14:03:33.145998954 -0400 0 -> /dev/pts/25
lrwx------ 1 hcma hcma 64 2018-03-15 14:03:33.145998954 -0400 1 -> /dev/pts/25
lrwx------ 1 hcma hcma 64 2018-03-15 14:03:23.210089578 -0400 2 -> /dev/pts/25
lr-x------ 1 hcma hcma 64 2018-03-15 14:03:33.145998954 -0400 3 -> pipe:[44076946]
l-wx------ 1 hcma hcma 64 2018-03-15 14:03:33.145998954 -0400 4 -> pipe:[44076946]
lr-x------ 1 hcma hcma 64 2018-03-15 14:03:33.145998954 -0400 5 -> /dev/urandom

After close():

ll /proc/8096/fd
total 0
dr-x------ 2 hcma hcma 0 2018-03-15 14:03:23.210089578 -0400 .
dr-xr-xr-x 9 hcma hcma 0 2018-03-15 14:03:23.190089760 -0400 ..
lrwx------ 1 hcma hcma 64 2018-03-15 14:03:33.145998954 -0400 0 -> /dev/pts/25
lrwx------ 1 hcma hcma 64 2018-03-15 14:03:33.145998954 -0400 1 -> /dev/pts/25
lrwx------ 1 hcma hcma 64 2018-03-15 14:03:23.210089578 -0400 2 -> /dev/pts/25
lr-x------ 1 hcma hcma 64 2018-03-15 14:03:33.145998954 -0400 3 -> pipe:[44076946]
l-wx------ 1 hcma hcma 64 2018-03-15 14:03:33.145998954 -0400 4 -> pipe:[44076946]
lr-x------ 1 hcma hcma 64 2018-03-15 14:03:33.145998954 -0400 5 -> /dev/urandom

[thehcma]

Correcting my original - after close():

ll /proc/8096/fd
total 0
dr-x------ 2 hcma hcma 0 2018-03-15 14:03:23.210089578 -0400 .
dr-xr-xr-x 9 hcma hcma 0 2018-03-15 14:03:23.190089760 -0400 ..
lrwx------ 1 hcma hcma 64 2018-03-15 14:03:33.145998954 -0400 0 -> /dev/pts/25
lrwx------ 1 hcma hcma 64 2018-03-15 14:03:33.145998954 -0400 1 -> /dev/pts/25
lrwx------ 1 hcma hcma 64 2018-03-15 14:03:23.210089578 -0400 2 -> /dev/pts/25
l-wx------ 1 hcma hcma 64 2018-03-15 14:03:33.145998954 -0400 4 -> pipe:[44076946]
lr-x------ 1 hcma hcma 64 2018-03-15 14:03:33.145998954 -0400 5 -> /dev/urandom

[pablogsal]

Is not leaked as the object is still alive. Only one side is closed when you call "closed". If you destroy the object:

>> del x

You will see that there are no more file descriptors in /proc/PID/fd associated with that queue.

Also, notice that closed() is called when the queue is destroyed.

[thehcma]

Pablo, but there is no way to close the other side.

Indeed, if you look in the implementation, you will see that the writer file descriptor can't be closed.

[pablogsal]

The way is to delete the object. IMHO I would not say is "leaked" as the object is still alive and holds resources and these resources are properly handled on destruction.

I cannot think of an immediate use case of closing both file descriptors but not deleting the object.

[pablogsal]

Notice that the writer gets closed when it receives a sentinel value (which how the queue knows when to close as part of the design):

>> x.put(multiprocessing.queues._sentinel)

If you call close after this line there will not be any fd associated with the queue open.

[thehcma]

Unfortunately this is not the case.

I will shrink my repro down to a more manageable size and post it here.

[thehcma]

Here is the repro (I am running this on Ubuntu 16 with the stock Python
version 2.7.12):

========================================================================================

#!/usr/bin/env python      
                
import os              
import subprocess                               
import sys                                        
                                                
from multiprocessing import Process, Queue      
from multiprocessing.queues import _sentinel                
                                                            
                                                            
def run_external_application(application_name, queue):      
    """Runs an Oxygen application"""                                     
    exit_status = 10            
    queue.put(exit_status)                         
    # none of the following help as far as making the pipe go away       
    queue.put(_sentinel)                                 
    queue.close()                                             
                                                              
                                                            
def run(application_name="external_application"):                                                       
    print "Starting '%s'" % application_name                                                            
                                                                                                              
    queue = Queue()                                                                                   
    application_process = Process(target=run_external_application, args=(application_name, queue))      
                                                                                                      
    application_process.start()         
                                             
    try:                                     
        application_process.join()         
    except KeyboardInterrupt:              
        application_process.terminate()      
                                           
    exit_status = queue.get()                                                                                    
    print "exit status", exit_status                                                                             
                                                                                                               
    queue.close()                                                                                         
    # the deletion below has no effect                                                                      
    del queue                                                                                               
    # the only thing that will make the pipe go away is to uncomment the below statement                      
    # queue._writer.close()                                                                                 
                                                                                                            
    print "\nthe '%s' application finished with exit status '%s'...\n" % (application_name, exit_status)      
     
    print "Note the file descriptor #4 below"     
    subprocess.call(["ls", "-la", "/proc/%d/fd" % os.getpid()])      
                                        
    return exit_status                  
                                                                                                                       
                                                                                                                       
if __name__ == "__main__":                                                                                             
    print "starting ", os.getpid()                                                                                     
    exit_status = run()                                                                                                
    sys.exit(exit_status)

[pablogsal]

When I run your script I do not see any file descriptor associated with the queue when

subprocess.call(["ls", "-la", "/proc/%d/fd" % os.getpid()])

is executed.

This is my output if I just execute your program:

starting 3728
Starting 'external_application'
exit status 10

the 'external_application' application finished with exit status '10'...

Note the file descriptor #4 below
total 0
dr-x------ 2 pablogsal pablogsal 0 Mar 18 12:17 .
dr-xr-xr-x 9 pablogsal pablogsal 0 Mar 18 12:17 ..
lrwx------ 1 pablogsal pablogsal 64 Mar 18 12:17 0 -> /dev/pts/1
lrwx------ 1 pablogsal pablogsal 64 Mar 18 12:17 1 -> /dev/pts/1
lrwx------ 1 pablogsal pablogsal 64 Mar 18 12:17 2 -> /dev/pts/1
lr-x------ 1 pablogsal pablogsal 64 Mar 18 12:17 5 -> /dev/urandom

This is my output if I remove the call to "del queue":

starting 3892
Starting 'external_application'
exit status 10

the 'external_application' application finished with exit status '10'...

Note the file descriptor #4 below
total 0
dr-x------ 2 pablogsal pablogsal 0 Mar 18 12:18 .
dr-xr-xr-x 9 pablogsal pablogsal 0 Mar 18 12:18 ..
lrwx------ 1 pablogsal pablogsal 64 Mar 18 12:18 0 -> /dev/pts/1
lrwx------ 1 pablogsal pablogsal 64 Mar 18 12:18 1 -> /dev/pts/1
lrwx------ 1 pablogsal pablogsal 64 Mar 18 12:18 2 -> /dev/pts/1
l-wx------ 1 pablogsal pablogsal 64 Mar 18 12:18 4 -> 'pipe:[104568]'
lr-x------ 1 pablogsal pablogsal 64 Mar 18 12:18 5 -> /dev/urandom

So at least on my side "del queue" is having the desired effect.

Notice that calling

del queue

does not destroy the object but just decrement in 1 the number of references to the object. To force collection of any possible cycles once you have destroyed all references on your side you can call

import gc
gc.collect()

[thehcma]

@pablo: I am using Python 2.7.12 (distributed with Ubuntu 16), what are you using? This might explain the difference between what we see.

Yet, irrespective of this difference, imho, it would be a better design to have "close" actually closing the underlying resources.

In general, if one has to delete and/or invoke the garbage collector on an object, it's an indication that the design needs a bit of polish. Just picture the small scenario I described amplified to a situation where a large number of queues is used, which is perhaps an artificial scenario, but one would end up with a bunch of file descriptors hanging around for no reason.

This is what files and sockets, for example, would do.

[pablogsal]

Notice that the documentation for close says:

Indicate that no more data will be put on this queue by the current process. The background thread will quit once it has flushed all buffered data to the pipe. This is called automatically when the queue is garbage collected.

The method does not promise to close any pipe, just "Indicate that no more data will be put on this queue by the current process". Closing prematurely the writer side could lead to issues. I still do not understand why you would want to close the pipes but maintain the queue alive.

I could be missing something, so let's see if other people think differently about this.

[thehcma]

I don't want to "close the pipes but maintain the queue alive" - I want to
terminate the queue and make sure that no resources are leaked. It's that
simple.

When one closes a file or a socket, there is no underlying OS resource
being held. That's what I would like to have here too.

Apparently the design does not support that and, if that's the case, it's
fine, it's just that it goes against most of the norm afaict.

On Sun, Mar 18, 2018 at 12:46 PM, Pablo Galindo Salgado <
report@bugs.python.org> wrote:

Pablo Galindo Salgado <pablogsal@gmail.com> added the comment:

Notice that the documentation for close says:

> Indicate that no more data will be put on this queue by the current
process. The background thread will quit once it has flushed all buffered
data to the pipe. This is called automatically when the queue is garbage
collected.

The method does not promise to close any pipe, just "Indicate that no more
data will be put on this queue by the current process". Closing prematurely
the writer side could lead to issues. I still do not understand why you
would want to close the pipes but maintain the queue alive.

I could be missing something, so let's see if other people think
differently about this.

----------

---

Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue33081\>

---

--

Henrique Andrade | +1-530-426-2123 | hcma@unscrambl.com |
https://unscrambl.com

[pablogsal]

"I want to terminate the queue and make sure that no resources are leaked.

Then you don't need to do anything special, those will be cleared on object destruction. This is not an unusual pattern even in other languages. For example, RAII in C++ is one of the most used patterns for acquiring resources and that works cleaning those resources on object destruction.

[thehcma]

Your comparison is not correct.

RAII in C++ ensures that, on object destruction, resources that have been
acquired will be closed and deallocated.

The closest analogy in Python is the use of a context manager, which, btw,
a Queue does not provide.

Indeed, such a design (with a context manager) would have been cleaner
because, on exit, both pipes would have been closed and file descriptors
would not hang around.

And, yes, that is what I'd prefer too - but one can't have everything. :)

With the current design, which is more akin to Java, one is exposed to the
vagaries of the garbage collector. Note that, even in Java,
try-with-resources and the auto-closeable interface would also take care of
this. In fact, most the Java classes that require external resources have
migrated to this model.

For these reasons, I think the design could be substantially improved
(i.e., with a context manager or with the provision of a method that really
terminates the queue, so resources are properly closed immediately).

On Sun, Mar 18, 2018 at 1:12 PM, Pablo Galindo Salgado <
report@bugs.python.org> wrote:

Pablo Galindo Salgado <pablogsal@gmail.com> added the comment:

>"I want to terminate the queue and make sure that no resources are leaked.

Then you don't need to do anything special, those will be cleared on
object destruction. This is not an unusual pattern even in other languages.
For example, RAII in C++ is one of the most used patterns for acquiring
resources and that works cleaning those resources on object destruction.

----------

---

Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue33081\>

---

--

Henrique Andrade | +1-530-426-2123 | hcma@unscrambl.com |
https://unscrambl.com