asyncio: emit ResourceWarning warnings if transports/event loops are not explicitly closed

[vstinner]

BPO	23243
Nosy	@gvanrossum, @vstinner, @1st1
Files	resource_warnings.patch server_close.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2015-01-29.16:55:05.859>
created_at = <Date 2015-01-14.23:18:04.084>
labels = ['expert-asyncio']
title = 'asyncio: emit ResourceWarning warnings if transports/event loops are not explicitly closed'
updated_at = <Date 2015-01-29.16:55:05.858>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2015-01-29.16:55:05.858>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2015-01-29.16:55:05.859>
closer = 'vstinner'
components = ['asyncio']
creation = <Date 2015-01-14.23:18:04.084>
creator = 'vstinner'
dependencies = []
files = ['37708', '37720']
hgrepos = []
issue_num = 23243
keywords = ['patch']
message_count = 7.0
messages = ['234043', '234071', '234074', '234103', '234858', '234979', '234980']
nosy_count = 4.0
nosy_names = ['gvanrossum', 'vstinner', 'python-dev', 'yselivanov']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue23243'
versions = ['Python 3.4', 'Python 3.5']

[vstinner]

I propose to add destructors to transports and event loops which emit a ResourceWarning if they are not closed.

The change should help to detect resource leaks and bugs.

Attached patch implements this issue. It only adds destructors on Python 3.4 and later, because older Python versions don't implement the PEP-442 (Safe object finalization) and so objects part of reference cycle would never be deleted.

The patch adds a new _closed attribute to BaseSubprocessTransport and _SSLProtocolTransport classes, to track if the transport was closed or not.

The patch should help to find bugs like this one:
https://code.google.com/p/tulip/issues/detail?id=218

[python-dev]

New changeset 2f13d53f4680 by Victor Stinner in branch '3.4':
Issue bpo-23243: Fix asyncio._UnixWritePipeTransport.close()
https://hg.python.org/cpython/rev/2f13d53f4680

New changeset aef0f9b4e729 by Victor Stinner in branch '3.4':
Issue bpo-23243: Close explicitly event loops in asyncio tests
https://hg.python.org/cpython/rev/aef0f9b4e729

New changeset f9b127188d43 by Victor Stinner in branch '3.4':
Issue bpo-23243: Close explicitly transports in asyncio tests
https://hg.python.org/cpython/rev/f9b127188d43

[vstinner]

I fixed a lot of methods and tests to ensure that all event loops and transports are properly closed. These changes were required to prepare the integration of this change.

Many changed were real bug fixes: closing explicitly tranports on error avoids to leak resources.

Some ResourceWarning are still emited on SSL server tests.

[vstinner]

Some ResourceWarning are still emited on SSL server tests.

Attached server_close.patch is a work-in-process patch to fix this issue on Linux.

If I understood correctly, there are two issues:

• SelectorEventLoop._accept_connection() doesn't care of the creation of the transport failed
• SSLProtocol.eof_received() does not notify the waiter that EOF was received

So an application cannot control SSL incoming connections which fail during the handshake? Control: log errors, do something with the socket, etc.

See also the STARTTLS feature request:
https://code.google.com/p/tulip/issues/detail?id=79

[vstinner]

• SelectorEventLoop._accept_connection() doesn't care of the creation of the transport failed

Fixing this issue requires to change the design of asyncio, so I chose to open a new issue: issue bpo-23333.

[python-dev]

New changeset 543f770f62f0 by Victor Stinner in branch '3.4':
Issue bpo-23243, asyncio: Emit a ResourceWarning when an event loop or a transport
https://hg.python.org/cpython/rev/543f770f62f0

[vstinner]

Ok, I commited my change adding destructors on Python 3.4 and newer to emit warnings when event loops or transports are not explicitly closed.