Bind logger and waninigs modules for asyncio __del__ methods

[asvetlov]

BPO	25074
Nosy	@gvanrossum, @vstinner, @asvetlov, @1st1
Files	asyncio_bind_modules.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/asvetlov'
closed_at = <Date 2017-12-20.21:11:54.594>
created_at = <Date 2015-09-11.19:16:05.093>
labels = ['type-feature', 'expert-asyncio']
title = 'Bind logger and waninigs modules for asyncio __del__ methods'
updated_at = <Date 2017-12-20.21:11:54.593>
user = 'https://github.com/asvetlov'

bugs.python.org fields:

activity = <Date 2017-12-20.21:11:54.593>
actor = 'asvetlov'
assignee = 'asvetlov'
closed = True
closed_date = <Date 2017-12-20.21:11:54.594>
closer = 'asvetlov'
components = ['asyncio']
creation = <Date 2015-09-11.19:16:05.093>
creator = 'asvetlov'
dependencies = []
files = ['40442']
hgrepos = []
issue_num = 25074
keywords = ['patch']
message_count = 4.0
messages = ['250490', '250701', '250778', '254624']
nosy_count = 4.0
nosy_names = ['gvanrossum', 'vstinner', 'asvetlov', 'yselivanov']
pr_nums = []
priority = 'normal'
resolution = 'wont fix'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue25074'
versions = ['Python 3.5', 'Python 3.6']

[asvetlov]

See aio-libs/aiohttp#497 for the reason.

Desctructors and descendant code (loop.call_exception_handler()) can be called on interpreter shutdown stage, which leads to printouts like 'None object has no attribute "errror"'.

[vstinner]

I don't like such "hacks". IMHO It's too late to try to log errors, to execute code to cleanup objects, etc.

You should try to implement something in aiohttp or even in the application to cleanup objects at exit. For example, it's probably wrong if you still have tasks when the event loop is closed. Especially if tasks are still pending. See this part of the doc which lists "pending tasks at exit":
https://docs.python.org/dev/library/asyncio-dev.html#chain-coroutines-correctly

For "exceptions never consumed", I proposed a different enhancement in asyncio directly:
https://bugs.python.org/issue24598

What do you think?

[asvetlov]

You should try to implement something in aiohttp or even in the application to cleanup objects at exit. For example, it's probably wrong if you still have tasks when the event loop is closed. Especially if tasks are still pending.

The problem is for client API. For server I have more control and implemented checks like you suggest.

For for client lazy user writes floppy code like:

    client = aiohttp.ClientSession()
    resp = yield from client.get(url)
    text = yield from resp.text()

Client is closed by GC, at the moment is not determined is loop has been closed or not (GC releases object in non-determenistic way).

So without changes I just cannot use loop.call_exception_handler() in __del__ methods.

It's not only aiohttp issue but we should get rid of call_exception_handler() reports in asyncio itself for the same reason.

We have logger and warnings modules usage e.g. in asyncio transports, it's not safe if transport is not closed properly before interpreter shutdown.

subprecess module has tricks for proper destruction like I've used in my patch.

[asvetlov]

Yes, the approach use hackery shadows for binding global variables on shutdown stage.
The same I've found in subprocess module: see https://github.com/python/cpython/blob/master/Lib/subprocess.py#L1544

The problem is: I want to inform users about unclosed resources.
Printouts like
TypeError: 'NoneType' object is not callable
or
AttributeError: 'NoneType' object has no attribute 'warn' are too confusing for casual developer.

On shutdown logger and warnings modules may be destroyed at the moment of del call, so warnings.warn, loop.call_exception_handler etc. are forbidden.

As an option we can add atexit handler to store shutdown flag into loop object (I don't know other way to get info is interpreter in shutdown or not).

According to the flag we can eihter print some short text about non-closed resources or raise proper ResourceWarning along with comprehensive logging with stacktrace if PYTHONASYNCIO is on.

Maybe I've missed something and you can suggest better solution.