asyncio subprocess AttributeError: 'NoneType' object has no attribute '_add_reader' / '_remove_reader'

[decaz]

BPO	38019
Nosy	@asvetlov, @1st1, @decaz, @miss-islington
PRs	bpo-38019: correctly handle pause/resume reading of closed asyncio unix pipe #16472 [3.8] bpo-38019: correctly handle pause/resume reading of closed asyncio unix pipe (GH-16472) #16473 [3.7] bpo-38019: correctly handle pause/resume reading of closed asyncio unix pipe (GH-16472) #16474
Files	f1_and_f2.zip

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/asvetlov'
closed_at = <Date 2019-09-29.13:21:00.810>
created_at = <Date 2019-09-03.16:55:09.357>
labels = ['3.8', 'type-bug', '3.7', '3.9', 'expert-asyncio']
title = "asyncio subprocess AttributeError: 'NoneType' object has no attribute '_add_reader' / '_remove_reader'"
updated_at = <Date 2019-09-30.09:21:10.830>
user = 'https://github.com/decaz'

bugs.python.org fields:

activity = <Date 2019-09-30.09:21:10.830>
actor = 'decaz'
assignee = 'asvetlov'
closed = True
closed_date = <Date 2019-09-29.13:21:00.810>
closer = 'asvetlov'
components = ['asyncio']
creation = <Date 2019-09-03.16:55:09.357>
creator = 'decaz'
dependencies = []
files = ['48589']
hgrepos = []
issue_num = 38019
keywords = ['patch']
message_count = 8.0
messages = ['351090', '352489', '352490', '352779', '353496', '353497', '353498', '353559']
nosy_count = 4.0
nosy_names = ['asvetlov', 'yselivanov', 'decaz', 'miss-islington']
pr_nums = ['16472', '16473', '16474']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue38019'
versions = ['Python 3.7', 'Python 3.8', 'Python 3.9']

[decaz]

I'm trying to use dwdiff (https://os.ghalkes.nl/dwdiff.html) to compare two text files. While running subprocess I'm getting the following errors:

Traceback (most recent call last):
  File "test_dwdiff.py", line 17, in <module>
    asyncio.run(main())
  File "/home/decaz/.pyenv/versions/3.7.4/lib/python3.7/asyncio/runners.py", line 43, in run
    return loop.run_until_complete(main)
  File "/home/decaz/.pyenv/versions/3.7.4/lib/python3.7/asyncio/base_events.py", line 579, in run_until_complete
    return future.result()
  File "test_dwdiff.py", line 14, in main
    await asyncio.gather(*(dwdiff() for __ in range(25)))
  File "test_dwdiff.py", line 10, in dwdiff
    await process.communicate()
  File "/home/decaz/.pyenv/versions/3.7.4/lib/python3.7/asyncio/subprocess.py", line 187, in communicate
    loop=self._loop)
  File "/home/decaz/.pyenv/versions/3.7.4/lib/python3.7/asyncio/subprocess.py", line 166, in _read_stream
    output = await stream.read()
  File "/home/decaz/.pyenv/versions/3.7.4/lib/python3.7/asyncio/streams.py", line 633, in read
    block = await self.read(self._limit)
  File "/home/decaz/.pyenv/versions/3.7.4/lib/python3.7/asyncio/streams.py", line 646, in read
    self._maybe_resume_transport()
  File "/home/decaz/.pyenv/versions/3.7.4/lib/python3.7/asyncio/streams.py", line 417, in _maybe_resume_transport
    self._transport.resume_reading()
  File "/home/decaz/.pyenv/versions/3.7.4/lib/python3.7/asyncio/unix_events.py", line 498, in resume_reading
    self._loop._add_reader(self._fileno, self._read_ready)
AttributeError: 'NoneType' object has no attribute '_add_reader'

and

Exception in callback SubprocessStreamProtocol.pipe_data_received(1, b'\xd0\x9f\xd...b8\xd1\x8f.\n')
handle: <Handle SubprocessStreamProtocol.pipe_data_received(1, b'\xd0\x9f\xd...b8\xd1\x8f.\n')>
Traceback (most recent call last):
  File "/home/decaz/.pyenv/versions/3.7.4/lib/python3.7/asyncio/events.py", line 88, in _run
    self._context.run(self._callback, *self._args)
  File "/home/decaz/.pyenv/versions/3.7.4/lib/python3.7/asyncio/subprocess.py", line 71, in pipe_data_received
    reader.feed_data(data)
  File "/home/decaz/.pyenv/versions/3.7.4/lib/python3.7/asyncio/streams.py", line 440, in feed_data
    self._transport.pause_reading()
  File "/home/decaz/.pyenv/versions/3.7.4/lib/python3.7/asyncio/unix_events.py", line 495, in pause_reading
    self._loop._remove_reader(self._fileno)
AttributeError: 'NoneType' object has no attribute '_remove_reader'

The code is the following:

import asyncio

async def dwdiff():
    process = await asyncio.create_subprocess_exec(
        '/usr/local/bin/dwdiff', 'f1.txt', 'f2.txt',
        stdout=asyncio.subprocess.PIPE,
        stderr=asyncio.subprocess.PIPE,
    )
    await process.communicate()

async def main():
    await asyncio.gather(*(dwdiff() for __ in range(25)))

asyncio.run(main())

Two text files f1.txt and f2.txt are attached as archive.

Perhaps it relates to https://bugs.python.org/issue29704

[asvetlov]

Thanks. The correct fix is 'do nothing if stream._transport is None'.
We have only two weeks before 3.8rc, thus I'll prepare a fix myself for the sake of fast merging.

[asvetlov]

This fix cannot land on 3.6, the version is in security-fix only mode.

[decaz]

Andrew, it would be nice to have it fixed within 3.8. Hope you'll find time for this.

[asvetlov]

New changeset 58498bc by Andrew Svetlov in branch 'master':
bpo-38019: correctly handle pause/resume reading of closed asyncio unix pipe (GH-16472)
58498bc

[miss-islington]

New changeset 1c3e469 by Miss Islington (bot) in branch '3.7':
bpo-38019: correctly handle pause/resume reading of closed asyncio unix pipe (GH-16472)
1c3e469

[miss-islington]

New changeset 19cd595 by Miss Islington (bot) in branch '3.8':
bpo-38019: correctly handle pause/resume reading of closed asyncio unix pipe (GH-16472)
19cd595

[decaz]

Andrew, thank you!