asyncio: ugly error related to signal handlers at exit if the loop is not closed explicitly

[AlexBrandt]

BPO	26133
Nosy	@gvanrossum, @vstinner, @asvetlov, @1st1
PRs	bpo-26133: Dont unsubscribe signals in UNIX even loop on interpreter shutdown #4956 [3.6] bpo-26133: Dont unsubscribe signals in UNIX even loop on interpreter shutdown (GH-4956) #4962 bpo-26133: Clear signals list on interpreter finalizing #5002 [3.6] bpo-26133: Clear signals list on interpreter finalizing (GH-5002) #5003 bpo-26133: Fix typos #5010 [3.6] bpo-26133: Fix typos (GH-5010) #5014

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/asvetlov'
closed_at = <Date 2017-12-24.12:32:05.842>
created_at = <Date 2016-01-16.19:06:06.630>
labels = ['3.7', 'type-crash', 'expert-asyncio']
title = 'asyncio: ugly error related to signal handlers at exit if the loop is not closed explicitly'
updated_at = <Date 2017-12-26.10:29:35.838>
user = 'https://bugs.python.org/AlexBrandt'

bugs.python.org fields:

activity = <Date 2017-12-26.10:29:35.838>
actor = 'asvetlov'
assignee = 'asvetlov'
closed = True
closed_date = <Date 2017-12-24.12:32:05.842>
closer = 'asvetlov'
components = ['asyncio']
creation = <Date 2016-01-16.19:06:06.630>
creator = 'Alex Brandt'
dependencies = []
files = []
hgrepos = []
issue_num = 26133
keywords = ['patch']
message_count = 15.0
messages = ['258401', '258402', '258429', '258432', '259592', '308819', '308828', '308830', '308859', '308875', '308890', '308995', '308996', '309054', '309056']
nosy_count = 6.0
nosy_names = ['gvanrossum', 'vstinner', 'asvetlov', 'yselivanov', 'Alex Brandt', 'hlellelid']
pr_nums = ['4956', '4962', '5002', '5003', '5010', '5014']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue26133'
versions = ['Python 3.6', 'Python 3.7']

[AlexBrandt]

When using the suggested practice of setting a stop loop signal handler with:

loop.add_signal_handler(signal.SIGTERM, loop.stop)

The following stack trace is given when the signal runs:

ligament_1 | Exception ignored in: <bound method BaseEventLoop.__del__ of <_UnixSelectorEventLoop running=False closed=True debug=False>>
ligament_1 | Traceback (most recent call last):
ligament_1 |   File "/usr/lib/python3.5/asyncio/base_events.py", line 387, in __del__
ligament_1 |   File "/usr/lib/python3.5/asyncio/unix_events.py", line 58, in close
ligament_1 |   File "/usr/lib/python3.5/asyncio/unix_events.py", line 139, in remove_signal_handler
ligament_1 |   File "/usr/lib/python3.5/signal.py", line 47, in signal
ligament_1 | TypeError: signal handler must be signal.SIG_IGN, signal.SIG_DFL, or a callable object

Since this happens during shutdown of the application I wouldn't consider this a high priority bug but it is quite annoying. I've also not investigated if this interrupts the loop stopping procedure yet.

[gvanrossum]

Heh, this is weird. If the signal being removed is SIGTERM, the logic looks like it is definitely going to call signal.signal(signal.SIGTERM, signal.SIG_DFL). And it doesn't look like the signal module's globals have been eradicated yet (or you'd have gotten something like "TypeError: 'NoneType' object is not callable" instead).

You seem to be using Python 3.5.0. Can you repro this with 3.5.1 or with the asyncio from github? Do you have a small self-contained program that repos it?

[vstinner]

The problem is that the signal module has been cleared when the destruction
has been called. You must not rely on destructors but call explicitly close
methods. Please run your app in asyncio debug mode and enable logs. See
asyncio doc.

[gvanrossum]

Victor, if the signal module has been cleared, how could it emit that error
message? signal.signal itself would no longer exist.

(I do agree with the solution you suggest -- though note that it may be
tricky to close the loop from inside the signal callback for SIGTERM, which
is presumably what is going on here.)

[hlellelid]

FWIW, I am experiencing the issue described here with Python 3.5.1.

[asvetlov]

remove_signal_handler() should do nothing if sys.is_finalizing() is true.

[1st1]

remove_signal_handler() should do nothing if sys.is_finalizing() is true.

Probably a good idea.

See also python/asyncio#456.

[vstinner]

remove_signal_handler() should do nothing if sys.is_finalizing() is true.

I dislike this option.

If you want to use sys.is_finalizing(), I would prefer to modify _UnixSelectorEventLoop.close() to not try to remove signal handler if close() has been called too late during Python finalization. But I also expect a warning in this case, not hide bugs silently.

If you reach this case, close() has probably been called by BaseEventLoop.__del__().

[asvetlov]

Implemented PR 4956 following Victor's suggestion.

[asvetlov]

New changeset 4a02543 by Andrew Svetlov in branch 'master':
bpo-26133: Dont unsubscribe signals in UNIX even loop on interpreter shutdown (bpo-4956)
4a02543

[asvetlov]

New changeset 3bc68cf by Andrew Svetlov (Miss Islington (bot)) in branch '3.6':
bpo-26133: Dont unsubscribe signals in UNIX even loop on interpreter shutdown (GH-4956) (bpo-4962)
3bc68cf

[asvetlov]

New changeset 4f146f9 by Andrew Svetlov in branch 'master':
bpo-26133: Clear signals list on interpreter finalizing (bpo-5002)
4f146f9

[asvetlov]

New changeset 5ff5d11 by Andrew Svetlov (Miss Islington (bot)) in branch '3.6':
bpo-26133: Clear signals list on interpreter finalizing (GH-5002) (bpo-5003)
5ff5d11

[asvetlov]

New changeset a8f4e15 by Andrew Svetlov in branch 'master':
bpo-26133: Fix typos (bpo-5010)
a8f4e15

[asvetlov]

New changeset 32518b4 by Andrew Svetlov (Miss Islington (bot)) in branch '3.6':
bpo-26133: Fix typos (GH-5010) (bpo-5014)
32518b4