asyncio.create_subprocess_exec raises OSError

[SebastianKreftDeezer]

BPO	21594
Nosy	@gvanrossum, @pitrou, @vstinner, @giampaolo, @1st1

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2014-06-02.19:47:51.541>
created_at = <Date 2014-05-28.16:06:40.132>
labels = []
title = 'asyncio.create_subprocess_exec raises OSError'
updated_at = <Date 2014-06-02.19:47:51.540>
user = 'https://bugs.python.org/SebastianKreftDeezer'

bugs.python.org fields:

activity = <Date 2014-06-02.19:47:51.540>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2014-06-02.19:47:51.541>
closer = 'vstinner'
components = []
creation = <Date 2014-05-28.16:06:40.132>
creator = 'Sebastian.Kreft.Deezer'
dependencies = []
files = []
hgrepos = []
issue_num = 21594
keywords = []
message_count = 5.0
messages = ['219285', '219286', '219584', '219602', '219625']
nosy_count = 6.0
nosy_names = ['gvanrossum', 'pitrou', 'vstinner', 'giampaolo.rodola', 'yselivanov', 'Sebastian.Kreft.Deezer']
pr_nums = []
priority = 'normal'
resolution = 'wont fix'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue21594'
versions = ['Python 3.4']

[SebastianKreftDeezer]

In some cases asyncio.create_subprocess_exec raises an OSError because there are no file descriptors available.

I don't know if that is expected, but IMO I think it would be better to just block until the required numbers of fds are available. Otherwise one would need to do this handling, which is not a trivial task.

This issue is happening in Debian 7, with a 3.2.0-4-amd64 kernel, and python 3.4.1 compiled from source.

[vstinner]

"I don't know if that is expected, but IMO I think it would be better to just block until the required numbers of fds are available."

Does it mean that it can block forever? It sounds strange to try to make such error silent.

Why not retrying in case of such error in your application? asyncio has no idea how to release file descriptors.

[SebastianKreftDeezer]

I agree that blocking is not ideal, however there are already some other methods that can eventually block forever, and for such cases a timeout is provided. A similar approach could be used here.

I think this method should retry until it can actually access the resources, because knowing when and how many files descriptors are going to be used is very implementation dependent. So handling the retry logic on the application side, would be probably very inefficient as lot os information is missing, as the subprocess mechanism is a black box.

[gvanrossum]

I'm not sure. Running out of file descriptors is really not something a
library can handle on its own -- this needs to be kicked back to the app to
handle. E.g. by pacing itself, or closing some connections, or changing the
system limit... The library really can't know what to do, and just waiting
until the condition magically clears seems asking for mysterious hangs.

On Mon, Jun 2, 2014 at 7:30 AM, STINNER Victor <report@bugs.python.org>
wrote:

Changes by STINNER Victor <victor.stinner@gmail.com>:

----------
nosy: +giampaolo.rodola, gvanrossum, pitrou, yselivanov

---

Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue21594\>

---

[vstinner]

"I agree that blocking is not ideal, however there are already some other methods that can eventually block forever, and for such cases a timeout is provided."

Functions like read() can "block" during several minutes, but it's something expect from network functions. Blocking until the application releases a file descriptor is more surprising.

"I think this method should retry until it can actually access the resources,"

You can easily implement this in your application.

"knowing when and how many files descriptors are going to be used is very implementation dependent"

I don't think that asyncio is the right place to handle file descriptors.

Usually, the file descriptor limit is around 1024. How did you reach such high limit? How many processes are running at the same time? asyncio should not "leak" file descriptors. It's maybe a bug in your application?

I'm now closing the bug.