"SystemError: bad call flags" exceptions added as part of BPO-33012 are difficult to debug

[ngie-eign]

BPO	39884
Nosy	@vstinner, @ngie-eign
PRs	bpo-39884: Add method name in "bad call flags" error #18944 [3.8] bpo-39884: Add method name in "bad call flags" error (GH-18944) #18956 [3.7] bpo-39884: Add method name in "bad call flags" error (GH-18944) #18957

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-03-12.12:38:29.055>
created_at = <Date 2020-03-07.04:11:33.983>
labels = ['extension-modules', '3.8', 'type-bug', '3.7', '3.9']
title = '"SystemError: bad call flags" exceptions added as part of BPO-33012 are difficult to debug'
updated_at = <Date 2020-03-12.12:38:29.051>
user = 'https://github.com/ngie-eign'

bugs.python.org fields:

activity = <Date 2020-03-12.12:38:29.051>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2020-03-12.12:38:29.055>
closer = 'vstinner'
components = ['Extension Modules']
creation = <Date 2020-03-07.04:11:33.983>
creator = 'ngie'
dependencies = []
files = []
hgrepos = []
issue_num = 39884
keywords = ['patch']
message_count = 5.0
messages = ['363575', '364000', '364012', '364013', '364014']
nosy_count = 2.0
nosy_names = ['vstinner', 'ngie']
pr_nums = ['18944', '18956', '18957']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue39884'
versions = ['Python 3.7', 'Python 3.8', 'Python 3.9']

[ngie-eign]

When a body of C extensions needs to be ported from python <3.8 to 3.8, one of the issues one might run into is improperly defined methods in a C extension, which results in SystemErrors stating:

>> SystemError: bad call flags

This new behavior was added as part of Issue bpo-33012.

While the issues definitely need to be resolved in the C extensions, where to start is not completely clear. I had to put printfs in PyCFunction_NewEx and PyDescr_NewMethod to track down the issues, e.g.,

>> printf("method name: %s\n", method->ml_name);

While this might be misleading for duplicate method definitions, it definitely helps narrow down the offending code.

Adding the method name to the SystemError would be a big step in the right direction in terms of making it easier to resolve these issues.

PS I realize that this might be masked by casting PyCFunction on methods or by not using gcc 8+, but I'd argue that C extensions need to have developer issues like this be clearer to the end-reader.

[vstinner]

New changeset c7d2d69 by Victor Stinner in branch 'master':
bpo-39884: Add method name in "bad call flags" error (GH-18944)
c7d2d69

[vstinner]

New changeset 03ac090 by Victor Stinner in branch '3.8':
bpo-39884: Add method name in "bad call flags" error (GH-18944) (GH-18956)
03ac090

[vstinner]

New changeset 6a12676 by Victor Stinner in branch '3.7':
bpo-39884: Add method name in "bad call flags" error (GH-18944) (GH-18957)
6a12676

[vstinner]

While the issues definitely need to be resolved in the C extensions, where to start is not completely clear. I had to put printfs in PyCFunction_NewEx and PyDescr_NewMethod to track down the issues, e.g., (...)

I agree, I had the same issue :-) I fixed the issue in 3.7, 3.8 and master branches.

In 3.7, the modified functions are different and the error message was different but similar.

Thanks for your bug report ;-)