Pass explicitly tstate to function calls

[vstinner]

BPO	38644
Nosy	@rhettinger, @vstinner, @encukou
PRs	bpo-38644: Pass tstate to Py_EnterRecursiveCall() #16997 bpo-38644: Add Py_EnterRecursiveCall() to the limited API #17046 bpo-38644: Pass tstate to _Py_CheckFunctionResult() #17050 bpo-38644: Add _PyObject_VectorcallTstate() #17052 bpo-38644: Add _PyObject_Call() #17089 bpo-38644: Add _PyEval_EvalFrame() with tstate #17131 bpo-38644: Add _PyEval_EvalCode() #17183 bpo-38644: Cleanup ceval.h #17185 bpo-38644: Pass tstate to _Py_FinishPendingCalls() #17990 bpo-38644: Pass tstate in ceval.c #18222 bpo-38644: Add Py_EnterRecursiveCall() to python3.def #18399 bpo-38644: Rephrase What's New entry #18461 [WIP] bpo-39984: Add PyInterpreterState.ceval #19034 bpo-38644: Add _PySys_Audit() which takes tstate #19180 bpo-38644: Make tstate more explicit inside pystate.c #19182 bpo-38644: Use _PySys_Audit(): pass tstate explicitly #19183 bpo-38644: Pass tstate explicitly in signalmodule.c #19184

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-04-14.00:37:59.857>
created_at = <Date 2019-10-30.14:31:24.358>
labels = ['interpreter-core', '3.9']
title = 'Pass explicitly tstate to function calls'
updated_at = <Date 2021-09-07.10:32:41.166>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2021-09-07.10:32:41.166>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2020-04-14.00:37:59.857>
closer = 'vstinner'
components = ['Interpreter Core']
creation = <Date 2019-10-30.14:31:24.358>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 38644
keywords = ['patch']
message_count = 24.0
messages = ['355716', '355717', '355974', '355984', '355987', '356234', '356495', '356597', '356601', '356732', '356734', '356878', '359916', '359920', '360835', '361558', '361559', '361825', '365104', '365112', '365113', '365152', '401213', '401225']
nosy_count = 3.0
nosy_names = ['rhettinger', 'vstinner', 'petr.viktorin']
pr_nums = ['16997', '17046', '17050', '17052', '17089', '17131', '17183', '17185', '17990', '18222', '18399', '18461', '19034', '19180', '19182', '19183', '19184']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue38644'
versions = ['Python 3.9']

[vstinner]

Follow-up of bpo-36710 for function calls.

[vstinner]

PR 16997 is a first step. The second step will be to pass tstate to:

• _PyObject_Vectorcall()
• _PyObject_MakeTpCall()
• _Py_CheckFunctionResult()

(I have a local branch that I have to rewrite on top of PR 16997.)

[vstinner]

New changeset f4b1e3d by Victor Stinner in branch 'master':
bpo-38644: Add Py_EnterRecursiveCall() to the limited API (GH-17046)
f4b1e3d

[vstinner]

New changeset be434dc by Victor Stinner in branch 'master':
bpo-38644: Pass tstate to Py_EnterRecursiveCall() (GH-16997)
be434dc

[vstinner]

New changeset 1726909 by Victor Stinner in branch 'master':
bpo-38644: Pass tstate to _Py_CheckFunctionResult() (GH-17050)
1726909

[vstinner]

New changeset 7e43373 by Victor Stinner in branch 'master':
bpo-38644: Add _PyObject_VectorcallTstate() (GH-17052)
7e43373

[vstinner]

I started a thread on python-dev about this issue:
"Pass the Python thread state to internal C functions"
https://mail.python.org/archives/list/python-dev@python.org/thread/PQBGECVGVYFTVDLBYURLCXA3T7IPEHHO/

[vstinner]

New changeset b9e6812 by Victor Stinner in branch 'master':
bpo-38644: Add _PyEval_EvalFrame() with tstate (GH-17131)
b9e6812

[vstinner]

New changeset 4d231bc by Victor Stinner in branch 'master':
bpo-38644: Add _PyObject_Call() (GH-17089)
4d231bc

[vstinner]

New changeset b5e170f by Victor Stinner in branch 'master':
bpo-38644: Add _PyEval_EvalCode() (GH-17183)
b5e170f

[vstinner]

New changeset 51edf8a by Victor Stinner in branch 'master':
bpo-38644: Cleanup ceval.h (GH-17185)
51edf8a

[vstinner]

One further step would be to change the VECTORCALL/FASTCALL calling convention to pass tstate. But I am not sure what is the risk to do that in Python 3.9? Cython uses FASTCALL internally for example.

[vstinner]

One further step would be to change the VECTORCALL/FASTCALL calling convention to pass tstate. But I am not sure what is the risk to do that in Python 3.9? Cython uses FASTCALL internally for example.

I started a thread on python-dev:
https://mail.python.org/archives/list/python-dev@python.org/thread/PIXJAJPWKDGHSQD65VOO2B7FDLU2QLHH/

I also wrote an article on this issue, "Pass the Python thread state explicitly":
https://vstinner.github.io/cpython-pass-tstate.html

[vstinner]

New changeset 2b1df45 by Victor Stinner in branch 'master':
bpo-38644: Pass tstate to _Py_FinishPendingCalls() (GH-17990)
2b1df45

[vstinner]

New changeset 61f4db8 by Victor Stinner in branch 'master':
bpo-38644: Pass tstate in ceval.c (GH-18222)
61f4db8

[vstinner]

New changeset 877ea88 by Victor Stinner in branch 'master':
bpo-38644: Add Py_EnterRecursiveCall() to python3.def (GH-18399)
877ea88

[vstinner]

One further step would be to change the VECTORCALL/FASTCALL calling convention to pass tstate.

My first strong motivation is to get None singleton from tstate. It's now being discussed in bpo-39511: "[subinterpreters] Per-interpreter singletons (None, True, False, etc.)".

[encukou]

New changeset f3fda37 by Victor Stinner in branch 'master':
bpo-38644: Rephrase What's New entry (GH-18461)
f3fda37

[vstinner]

New changeset 08faf00 by Victor Stinner in branch 'master':
bpo-38644: Add _PySys_Audit() which takes tstate (GH-19180)
08faf00

[vstinner]

New changeset 7281898 by Victor Stinner in branch 'master':
bpo-38644: Pass tstate explicitly in signalmodule.c (GH-19184)
7281898

[vstinner]

New changeset 71a3522 by Victor Stinner in branch 'master':
bpo-38644: Make tstate more explicit inside pystate.c (GH-19182)
71a3522

[vstinner]

New changeset 1c1e68c by Victor Stinner in branch 'master':
bpo-38644: Use _PySys_Audit(): pass tstate explicitly (GH-19183)
1c1e68c

[rhettinger]

This adds cost to our most critical code paths. For example, type_call() will now be slower for every single object instantiation.

[vstinner]

Raymond:

This adds cost to our most critical code paths. For example, type_call() will now be slower for every single object instantiation.

This issue is now closed. Please open a new issue if you consider that the code should be modified. It would also help to have a benchmark if it's a performance regression ;-)