Python 3.8 improperly warns about closing properly closed streams

[RonFrederick]

BPO	38529
Nosy	@asvetlov, @cjrh, @ambv, @1st1, @thehesiod, @miss-islington
PRs	bpo-38529: Fix asyncio stream warning #17474 [3.8] bpo-38529: Fix asyncio stream warning (GH-17474) #17492

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-12-07.11:40:47.255>
created_at = <Date 2019-10-19.18:40:28.968>
labels = ['type-bug', '3.8', '3.9', 'release-blocker', 'expert-asyncio']
title = 'Python 3.8 improperly warns about closing properly closed streams'
updated_at = <Date 2019-12-07.11:40:47.254>
user = 'https://bugs.python.org/RonFrederick'

bugs.python.org fields:

activity = <Date 2019-12-07.11:40:47.254>
actor = 'asvetlov'
assignee = 'none'
closed = True
closed_date = <Date 2019-12-07.11:40:47.255>
closer = 'asvetlov'
components = ['asyncio']
creation = <Date 2019-10-19.18:40:28.968>
creator = 'Ron Frederick'
dependencies = []
files = []
hgrepos = []
issue_num = 38529
keywords = ['patch']
message_count = 6.0
messages = ['354953', '355481', '355482', '355485', '357968', '357970']
nosy_count = 8.0
nosy_names = ['falsetru', 'asvetlov', 'cjrh', 'lukasz.langa', 'yselivanov', 'thehesiod', 'Ron Frederick', 'miss-islington']
pr_nums = ['17474', '17492']
priority = 'release blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue38529'
versions = ['Python 3.8', 'Python 3.9']

[RonFrederick]

In testing AsyncSSH against Python 3.8, I noticed a large number of the following errors, even though I was properly closing streams before the objects holding them were garbage-collected.

An open stream object is being garbage collected; call "stream.close()" explicitly.

After some investigation, the problem appears to be that closing a stream is not good enough to prevent the error. The check in asyncio doesn't properly handle the case where the stream is closing, but has not fully closed. Here's a simple test program that demonstrates this:

import asyncio

async def tcp_client():
    reader, writer = await asyncio.open_connection('127.0.0.1', 22)

    writer.close()

asyncio.run(tcp_client())

It's possible to avoid this message by awaiting on writer.wait_closed(), but wait_closed() doesn't exist until Python 3.7, making it very difficult to write portable code and still avoid this message.

[RonFrederick]

I think the following change might address this problem:

---

*** 233,239 ****

  def \_on_reader_gc(self, wr):
      transport = self.\_transport

! if transport is not None:
# connection_made was called
context = {
'message': ('An open stream object is being garbage '
--- 233,239 ----

  
      def _on_reader_gc(self, wr):
          transport = self._transport
!         if transport is not None and not transport.is_closing():
              # connection_made was called
              context = {
                  'message': ('An open stream object is being garbage '

[RonFrederick]

Sorry, I should have said that the change below was in the file asyncio/streams.py.

[1st1]

Yes, I've experienced this bug. We need to fix this in 3.8.1.

[asvetlov]

New changeset 7ddcd0c by Andrew Svetlov in branch 'master':
bpo-38529: Fix asyncio stream warning (GH-17474)
7ddcd0c

[miss-islington]

New changeset 7fde4f4 by Miss Islington (bot) in branch '3.8':
bpo-38529: Fix asyncio stream warning (GH-17474)
7fde4f4