Misc fixes and cleanups in error handling C code

[serhiy-storchaka]

BPO	28994
Nosy	@terryjreedy, @pfmoore, @larryhastings, @tjguk, @ned-deily, @zware, @serhiy-storchaka, @zooba, @ZackerySpytz
PRs	bpo-28994: Fixed errors handling in atexit._run_exitfuncs(). #2034 bpo-28994: PyErr_NormalizeException() no longer recursive. #2035 gh-73180: Chain exceptions rather than dropping the old exception. #2072 [3.6] bpo-28994: Fixed errors handling in atexit._run_exitfuncs(). (GH-2034) #2121 [3.5] bpo-28994: Fixed errors handling in atexit._run_exitfuncs(). (GH-2034) #2122 [2.7] bpo-28994: Fixed errors handling in atexit._run_exitfuncs(). (GH-2034) #2123 [2.7] bpo-28994: Remove mistakenly backported atexitmodule.c #9214
Files	errors.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2016-12-16.23:38:16.853>
labels = ['interpreter-core', 'type-bug', '3.7']
title = 'Misc fixes and cleanups in error handling C code'
updated_at = <Date 2018-09-12.11:28:24.267>
user = 'https://github.com/serhiy-storchaka'

bugs.python.org fields:

activity = <Date 2018-09-12.11:28:24.267>
actor = 'serhiy.storchaka'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Interpreter Core']
creation = <Date 2016-12-16.23:38:16.853>
creator = 'serhiy.storchaka'
dependencies = []
files = ['45932']
hgrepos = []
issue_num = 28994
keywords = ['patch']
message_count = 14.0
messages = ['283451', '295618', '295743', '295744', '295745', '295747', '295817', '295820', '295821', '295822', '295823', '305588', '325129', '325133']
nosy_count = 9.0
nosy_names = ['terry.reedy', 'paul.moore', 'larry', 'tim.golden', 'ned.deily', 'zach.ware', 'serhiy.storchaka', 'steve.dower', 'ZackerySpytz']
pr_nums = ['2034', '2035', '2072', '2121', '2122', '2123', '9214']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue28994'
versions = ['Python 3.5', 'Python 3.6', 'Python 3.7']

[serhiy-storchaka]

Proposed patch makes few minor changes related to handling errors in C.

• Fixed possible NULL dereference in PyErr_NormalizeException().

• Fixed error checking in atexit._run_exitfuncs().

• Fixed possible memory leaks in _Py_FatalError_PrintExc().

• PyErr_NormalizeException() no longer recursive.

• If an error is occurred during handling other error, the original exception is chained as the __contex__ attribute to the new exception instead of just be dropped.

• PyTraceBack_Print() no longer fails OverflowError if tracebacklimit is very large negative or positive value.

• ctype error message now include the name of the initial exception instead of the repr of its class.

• Py_XDECREFs is replaced with Py_DECREFs if this is safe.

• Added few asserts.

• Other minor cleanups.

[serhiy-storchaka]

I'm trying to split the patch on smaller parts.

[serhiy-storchaka]

New changeset 3fd54d4 by Serhiy Storchaka in branch 'master':
bpo-28994: Fixed errors handling in atexit._run_exitfuncs(). (bpo-2034)
3fd54d4

[serhiy-storchaka]

New changeset d89dc84 by Serhiy Storchaka in branch '3.6':
[3.6] bpo-28994: Fixed errors handling in atexit._run_exitfuncs(). (GH-2034) (bpo-2121)
d89dc84

[serhiy-storchaka]

New changeset 7d8c1eb by Serhiy Storchaka in branch '3.5':
[3.5] bpo-28994: Fixed errors handling in atexit._run_exitfuncs(). (GH-2034) (bpo-2122)
7d8c1eb

[serhiy-storchaka]

New changeset 0cc43df by Serhiy Storchaka in branch '2.7':
[2.7] bpo-28994: Fixed errors handling in atexit._run_exitfuncs(). (GH-2034) (bpo-2123)
0cc43df

[terryjreedy]

3.6 no longer compiles for me on Windows. A repeat of pcbuild\build.bat -d gives the same error report as below.

My previous build was 25 hours ago. atexitmodule.c is the only C file in the 3.6 merge update, so the backport might be to blame. Git status say that Objects/listobject is changed, even though not changed by PR 2121, and that there is an untracked Objects/clinic/listobject.c.h. (but there is one in 3.7, so...?).

chkdsk F: /F found one unrelated orphaned file but not other specific errors. It ended with this:

Stage 3: Examining security descriptors ...
Security descriptor verification completed.
40783 data files processed.
CHKDSK is verifying Usn Journal...
3224 USN bytes processed.
Usn Journal verification completed.
An unspecified error occurred (6e74667363686b2e 1475).

(which it also did a week ago) I don't remember what was done after the USN check. Web search has not helped yet for the 63... error. In any case, I thing someone else should verify building on Windows before 3.6.2 is tagged, if it has not yet been.

Building heads/3.6-dirty:8399a177de 3.6
atexitmodule.c
listobject.c
f:\dev\36\objects\clinic/listobject.c.h(24): warning C4133: 'function': incompatible type
s - from 'char [10]' to 'PyObject *' [f:\dev\36\PCbuild\pythoncore.vcxproj]
f:\dev\36\objects\clinic/listobject.c.h(25): warning C4133: 'function': incompatible type
s - from 'Py_ssize_t *' to '_PyArg_Parser *' [f:\dev\36\PCbuild\pythoncore.vcxproj]
f:\dev\36\objects\clinic/listobject.c.h(29): warning C4013: '_PyArg_NoStackKeywords' unde
fined; assuming extern returning int [f:\dev\36\PCbuild\pythoncore.vcxproj]
f:\dev\36\objects\clinic/listobject.c.h(112): warning C4133: 'function': incompatible typ
es - from 'char [7]' to 'PyObject *' [f:\dev\36\PCbuild\pythoncore.vcxproj]
f:\dev\36\objects\clinic/listobject.c.h(113): warning C4133: 'function': incompatible typ
es - from 'Py_ssize_t *' to '_PyArg_Parser *' [f:\dev\36\PCbuild\pythoncore.vcxproj]
f:\dev\36\objects\clinic/listobject.c.h(147): warning C4013: '_PyArg_ParseStackAndKeyword
s' undefined; assuming extern returning int [f:\dev\36\PCbuild\pythoncore.vcxproj]
f:\dev\36\objects\clinic/listobject.c.h(198): warning C4133: 'function': incompatible typ
es - from 'char [13]' to 'PyObject *' [f:\dev\36\PCbuild\pythoncore.vcxproj]
f:\dev\36\objects\clinic/listobject.c.h(199): warning C4047: 'function': '_PyArg_Parser *
' differs in levels of indirection from 'PyObject **' [f:\dev\36\PCbuild\pythoncore.vcxpr
oj]
f:\dev\36\objects\clinic/listobject.c.h(199): warning C4024: '_PyArg_ParseStack': differe
nt types for formal and actual parameter 4 [f:\dev\36\PCbuild\pythoncore.vcxproj]
..\Objects\listobject.c(1979): error C2059: syntax error: '<<' [f:\dev\36\PCbuild\pythonc
ore.vcxproj]
..\Objects\listobject.c(1984): error C2014: preprocessor command must start as first nonw
hite space [f:\dev\36\PCbuild\pythoncore.vcxproj]
..\Objects\listobject.c(2015): error C2014: preprocessor command must start as first nonw
hite space [f:\dev\36\PCbuild\pythoncore.vcxproj]
..\Objects\listobject.c(2171): error C2143: syntax error: missing ';' before '<<' [f:\dev
\36\PCbuild\pythoncore.vcxproj]
..\Objects\listobject.c(2173): error C2143: syntax error: missing ';' before '==' [f:\dev
\36\PCbuild\pythoncore.vcxproj]
..\Objects\listobject.c(2175): error C2143: syntax error: missing ';' before '>>' [f:\dev
\36\PCbuild\pythoncore.vcxproj]
..\Objects\listobject.c(2175): error C2014: preprocessor command must start as first nonw
hite space [f:\dev\36\PCbuild\pythoncore.vcxproj]
..\Objects\listobject.c(2256): error C2143: syntax error: missing ';' before '<<' [f:\dev
\36\PCbuild\pythoncore.vcxproj]
..\Objects\listobject.c(2261): error C2143: syntax error: missing ';' before '==' [f:\dev
\36\PCbuild\pythoncore.vcxproj]
..\Objects\listobject.c(2262): error C2014: preprocessor command must start as first nonw
hite space [f:\dev\36\PCbuild\pythoncore.vcxproj]
..\Objects\listobject.c(2268): error C2059: syntax error: 'if' [f:\dev\36\PCbuild\pythonc
ore.vcxproj]
..\Objects\listobject.c(2273): error C2059: syntax error: 'for' [f:\dev\36\PCbuild\python
core.vcxproj]
..\Objects\listobject.c(2273): error C2143: syntax error: missing '{' before '<' [f:\dev\
36\PCbuild\pythoncore.vcxproj]
..\Objects\listobject.c(2273): error C2059: syntax error: '<' [f:\dev\36\PCbuild\pythonco
re.vcxproj]
..\Objects\listobject.c(2273): error C2143: syntax error: missing '{' before '++' [f:\dev
\36\PCbuild\pythoncore.vcxproj]
..\Objects\listobject.c(2273): error C2059: syntax error: '++' [f:\dev\36\PCbuild\pythonc
ore.vcxproj]
..\Objects\listobject.c(2273): error C2059: syntax error: ')' [f:\dev\36\PCbuild\pythonco
re.vcxproj]
..\Objects\listobject.c(2280): error C2059: syntax error: 'string' [f:\dev\36\PCbuild\pyt
honcore.vcxproj]
..\Objects\listobject.c(2280): error C2040: 'PyErr_Format': 'int ()' differs in levels of
indirection from 'PyObject *(PyObject *,const char *,...)' [f:\dev\36\PCbuild\pythoncore
.vcxproj]
..\Objects\listobject.c(2281): error C2059: syntax error: 'return' [f:\dev\36\PCbuild\pyt
honcore.vcxproj]
..\Objects\listobject.c(2282): error C2059: syntax error: '}' [f:\dev\36\PCbuild\pythonco
re.vcxproj]
..\Objects\listobject.c(2422): error C2059: syntax error: '<<' [f:\dev\36\PCbuild\pythonc
ore.vcxproj]
..\Objects\listobject.c(2443): error C2014: preprocessor command must start as first nonw
hite space [f:\dev\36\PCbuild\pythoncore.vcxproj]
..\Objects\listobject.c(3113): fatal error C1004: unexpected end-of-file found [f:\dev\36
\PCbuild\pythoncore.vcxproj]
Generating Code...

f:\dev\36>git log Modules/atexitmodule.c
commit d89dc84
Author: Serhiy Storchaka <storchaka@gmail.com>
Date: Mon Jun 12 09:02:13 2017 +0300

[3.6] bpo-28994: Fixed errors handling in atexit._run_exitfuncs(). (GH-2034) (bpo-2121)

The traceback no longer displayed for SystemExit raised in a callback registered by atexit..
(cherry picked from commit 3fd54d4a7e604067e2bc0f8cfd58bdbdc09fa7f4)

When I reverted the listobject.c change and rebuilt, compilation finished and the test suite ran to SUCCESS.

[serhiy-storchaka]

Last time Objects/listobject.c was changed 2 months ago. This looks as an error in your workspace. Maybe something wrong happened when you switched from master to 3.6.

[zware]

Agreed, looks like something went wrong in your checkout, Terry. Just confirmed that a 3.6 build on a fresh checkout on Windows is fine.

[zooba]

I just updated to the latest 3.6 commit and listobject.c doesn't even use clinic in this branch.

Serhiy is probably right - this is a workspace issue.

[terryjreedy]

Since I fetch, merge, and build with a .bat file that has worked fine at least 10 times, with pushes in between, the glitch is a puzzle. Next time I will just revert, as I did, or reclone and rebuild the 3.6 worktree.

[serhiy-storchaka]

New changeset cf29653 by Serhiy Storchaka in branch 'master':
bpo-28994: PyErr_NormalizeException() no longer use C stack for recursion. (bpo-2035)
cf29653

[ZackerySpytz]

In PR 2123, it was reported that the Modules/atexitmodule.c file was backported to 2.7. PR 9214 addresses this.

[serhiy-storchaka]

New changeset b36567b by Serhiy Storchaka (Zackery Spytz) in branch '2.7':
[2.7] bpo-28994: Remove mistakenly backported atexitmodule.c (GH-9214)
b36567b

[arhadthedev]

The feature is implemented before Python 3.3 (an enhancement of this feature is listed in 3.3's What's New, didn't try to search further) so closing as done.