Python doesn't handle SIGINT well if it arrives during interpreter startup

[gjb1002]

BPO	3137
Nosy	@vstinner, @benjaminp
Files	SIGINT-2.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2010-06-14.23:14:35.701>
created_at = <Date 2008-06-19.08:04:36.509>
labels = ['interpreter-core', 'type-bug']
title = "Python doesn't handle SIGINT well if it arrives during interpreter startup"
updated_at = <Date 2010-06-14.23:14:35.700>
user = 'https://bugs.python.org/gjb1002'

bugs.python.org fields:

activity = <Date 2010-06-14.23:14:35.700>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2010-06-14.23:14:35.701>
closer = 'vstinner'
components = ['Interpreter Core']
creation = <Date 2008-06-19.08:04:36.509>
creator = 'gjb1002'
dependencies = []
files = ['16501']
hgrepos = []
issue_num = 3137
keywords = ['patch']
message_count = 19.0
messages = ['68392', '68402', '100617', '100619', '100636', '100637', '100638', '100639', '100640', '100667', '100682', '100814', '100825', '100935', '100940', '101426', '101462', '101466', '107841']
nosy_count = 3.0
nosy_names = ['gjb1002', 'vstinner', 'benjamin.peterson']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue3137'
versions = ['Python 2.5']

[gjb1002]

If a python script receives SIGINT while the interpreter is starting up,
it's possible to get the message "import site failed; use -v for
traceback" printed on standard error and for execution to proceed. It
also seems to be possible to get half-imported modules and for the
script to fail later claiming that something like "os.getenv" doesn't exist.

If I do as instructed and use -v for traceback I get something like:

'import site' failed; traceback:
Traceback (most recent call last):
  File "/usr/lib/python2.4/site.py", line 61, in ?
    import os
  File "/usr/lib/python2.4/os.py", line 683, in ?
    import copy_reg as _copy_reg
  File "/usr/lib/python2.4/copy_reg.py", line 5, in ?
    """
KeyboardInterrupt 

I imagine there exists some code like
try:
import site
except:
sys.stderr.write("import site failed; use -v for traceback\n")

though I couldn't find any. If so, it seems clear that KeyboardInterrupt
needs to be re-raised, or Python's special handler for SIGINT installed
rather later.

[benjaminp]

Yes, the C startup code in pythonrun.c has a lot of bare-except statements.

[vstinner]

I think initsite() should be atomic: if any kind of error occurs, Python should print it and exit directly (exit code 1 to notice the error). If initsite() fails, Python is not complelty initialized.

site is responsible to initialiaze a lot of things:

• Set all module' __file__ attribute to an absolute path
• Remove duplicate entries from sys.path along with making them absolute
• Add a per user site-package to sys.path
• Add site-packages (and possibly site-python) to sys.path
• Define new built-ins 'quit' and 'exit'.
• Set 'copyright' and 'credits' in __builtin__"
• Create builtin help() function
• On Windows, some default encodings are not provided by Python, while they are always available as "mbcs" in each locale. Make them usable by aliasing to "mbcs" in such a case.
• import sitecustomize
• import usercustomize
• del sys.setdefaultencoding
• etc.

Be able to ignore the site initializations might be a security vulnerability.

Attached patch consider any exception as fatal: display the error and exit. I moved the call to _PyGILState_Init() before initsite() to avoid a crash in Py_Finalize() => see bpo-8063.

[vstinner]

Using "valgrind --log-file=/tmp/trace ./python" to slow down Python, I found another bug in the interactive interpreter: if you press CTRL+c just after the initialization of the site module the exception will be ignored.

When a signal is catched, it's stored in a table, and the real handler is called later (by Py_MakePendingCalls()). Py_AddPendingCall() is called to ensure that the signal will be handled before the next Python instruction. In my case, the next Python instruction is "decode the string written by the user using the terminal encoding" called by the tokenizer... but the tokenizer ignores all errors (not only unicode errors): see tok_stdin_decode().

Recipe to catch the code responsible to ignore the keyboard interrupt exception (using gdb):
---------------------

$ gdb ./python
(gdb) b initsite
Breakpoint 1, initsite ()
(gdb) run
...
Breakpoint 1, initsite ()
(gdb) next
<execution of the site module>
(gdb) b signal_default_int_handler
Breakpoint 2
(gdb) signal SIGINT
Breakpoint 2, signal_default_int_handler
(gdb) b PyErr_Clear
Breakpoint 3
(gdb) cont
...
Breakpoint 3, PyErr_Clear ()
(gdb) where
<HERE YOU HAVE>

---

Attached patch calls Py_MakePendingCalls() before PyRun_AnyFileExFlags() to avoid the tokenizer bug. It's just a workaround, not a real bugfix.

[vstinner]

PyImport_Import() clears errors if a module has no globals, whereas PyEval_GetGlobals() doesn't set any exception on failure. => import_no_global.patch removes this unnecessary PyErr_Clear().

[vstinner]

Py_InitializeEx() calls PyErr_Clear() if PyCodec_Encoder() fails without checking for the exception type.

Attached initiliaze_pycodec_error.patch checks for error type: if the error is not an LookupError, display the error and exit (as done for initsite() iny my initsite.patch).

[vstinner]

tok_stdin_decode() clears error without check for the type.

tokenizer_error.patch stops the tokenizer with an E_ERROR if the exception is not an UnicodeDecodeError. Fix also err_input() to support E_ERROR: leave the global exception (PyErr_*) unchanged.

[vstinner]

Remove main_pending_calls.patch hack: replaced by tokenizer_error.patch.

[vstinner]

Remove main_pending_calls.patch hack: replaced by tokenizer_error.patch.

With extra tests, I realized that this patchs is still useful to process a SIGINT emitted between Py_Initialize() and PyRun_AnyFileExFlags(). Without the patch, if a SIGINT is emitted just after Py_Initialize(): it's only proceed when the user press enter in the interactive interpreter (when the tokenizer decodes the input string to the terminal charset).

[vstinner]

SIGINT.patch is the sum of all patches, it should be easier to review it.

[vstinner]

As requested by Guido, here is a new patch including the fix for the site module: catch errors in execsitecustomize() and execusercustomize().

[vstinner]

Commited to trunk: r78826 + r78827. I will later backport to py3k.

[vstinner]

My change about site initialization error raised 2 old issues: bpo-7774 and bpo-7880. flox commited a fix for bpo-7880 and a workaround for bpo-7774 (just enough to fix test_subprocess).

[vstinner]

Commited to trunk: r78826

Backported to py3k as r78872. But py3k will require extra work: there are some PyErr_Clear() somewhere, eating the errors.

[vstinner]

Bug related to this issue: bpo-8124, PySys_WriteStdout() and PySys_WriteStderr() ignore signal handlers errors.

[vstinner]

Commited to trunk: r78826 + r78827

Partial backport to 2.6 as r79204: leave import site error handler unchanged (print the error and continue). I don't want to change Python behaviour between minor releases.

[vstinner]

Backported to py3k as r78872

And backported to 3.1 as r79247.

But py3k will require extra work: there are some PyErr_Clear() somewhere, eating the errors.

Leave this issue open until bpo-8124 is fixed.

[vstinner]

> Backported to py3k as r78872
And backported to 3.1 as r79247.

I reverted the change on initsite(): as for Python 2.6, I don't want to change import site error handler between minor releases.

[vstinner]

Bug related to this issue: bpo-8124, PySys_WriteStdout() and PySys_WriteStderr() ignore signal handlers errors.
...
Leave this issue open until bpo-8124 is fixed.

I fixed both issues, so it's time to close this one.