Asyncio subprocess `.returncode` never gets set when busy

[WillyJL]

Bug report

Bug description:

When checking asyncio subprocess .returncode for example in a busy loop, it will never be set.

Minimal example:

import asyncio

async def main():
    code = """
import time, sys
time.sleep(1)
print('proc exits')
    """
    proc = await asyncio.create_subprocess_exec("python", "-c", code)
    while proc.returncode is None:
        pass
    print("proc.returncode:", proc.returncode)

if __name__ == "__main__":
    asyncio.run(main())

One would expect this code to wait 1 second, print proc exits, followed by proc.returncode: 0. However only proc exits is printed, meaning that the parent process's asyncio never detected the process exiting.

This is problematic for what I would imagine is a relatively common and useful usecase (as was for me finding this bug): using a loop to get output from a child process.

import asyncio
import subprocess

async def main():
    code = """
import time, sys
time.sleep(1)
print('proc exits')
    """
    proc = await asyncio.create_subprocess_exec("python", "-c", code, stdout=subprocess.PIPE)
    while proc.returncode is None:
        print("proc said:", await proc.stdout.readline())
    print("proc.returncode:", proc.returncode)

if __name__ == "__main__":
    asyncio.run(main())

This will print:

proc said: b'proc exits\r\n'
proc said: b''
proc said: b''
proc said: b''
proc said: b''
proc said: b''
proc said: b''
proc said: b''
proc said: b''
...

I noticed that adding a small await asyncio.sleep(0.1) before looping again will allow the returncode to be set, but this is not ideal. Especially considering the bug happens even when using await proc.stdout.readline() in the loop, which hints to the user that this is done efficiently and asynchronously. Having to instead think about leaving enough CPU time to the event loop to detect the process exiting is not ideal and doesn't seem pythonic to me.

For the time being I worked around this issue for my usecase by checking proc.stdout.at_eof() which seems to work fine.

Maybe when accessing .returncode which is a property, it could use some CPU time to see if an exit event is pending? Not sure how it is implemented under the hood so that might not even make sense.

CPython versions tested on:

3.11

Operating systems tested on:

Windows

[kumaraditya303]

The minimal example isn't working because you are creating a busy loop without yielding to the event loop to process events.
The following code would make it work for you:

import asyncio

async def main():
    code = """
import time, sys
time.sleep(1)
print('proc exits')
    """
    proc = await asyncio.create_subprocess_exec("python", "-c", code)
    while proc.returncode is None:
        await asyncio.sleep(0)
    print("proc.returncode:", proc.returncode)

if __name__ == "__main__":
    asyncio.run(main())

Output:

❯ python main.py 
proc exits
proc.returncode: 0

[asvetlov]

I don't think we should care about such bug reports; busy loop is not compatible with asyncio in many aspects.

[WillyJL]

I understand that the busy loop without ever yielding to the event loop would have no chance to process the events in this case, but what about the example where the loop repeatedly calls await proc.stdout.readline()? Shouldn't that give the chance to process events?

If that's not the case, thanks for pointing out await asyncio.sleep(0), i had forgotten that this was an option to yield to event loop.

EDIT: perhaps it could be a good idea to note this behavior in the documentation for .returncode?

[kumaraditya303]

The following code works on Python 3.12 and above:

import asyncio
import subprocess

async def main():
    code = """
import time, sys
time.sleep(1)
print('proc exits')
    """
    proc = await asyncio.create_subprocess_exec("python", "-c", code, stdout=subprocess.PIPE)
    while proc.returncode is None:
        print("proc said:", await proc.stdout.readline())
    print("proc.returncode:", proc.returncode)

if __name__ == "__main__":
    asyncio.run(main())

❯ python main.py
proc said: b'proc exits\n'
proc said: b''
proc.returncode: 0

It doesn't work on 3.11 but that is in security fixes mode so closing this as won't fix.