[C API] Promote PEP 523 functions to public functions: add `PyUnstable_InterpreterState_SetEvalFrameFunc()`

[vstinner]

C extensions should not use private functions (functions prefixed by _Py), but PEP 523 only added private functions:

• _PyInterpreterState_GetEvalFrameFunc()
• _PyInterpreterState_SetEvalFrameFunc()
• _PyEval_EvalFrameDefault()

I propose promoting these functions as public functions (just remove the _ prefix):

• PyUnstable_InterpreterState_GetEvalFrameFunc()
• PyUnstable_InterpreterState_SetEvalFrameFunc()
• PyUnstable_Eval_EvalFrameDefault()

cc @emmatyping

Update: I added PyUnstable_ prefix to function names.

Linked PRs

• gh-141518: Add PyUnstable_InterpreterState_SetEvalFrameFunc() #141665

[vstinner]

A code search on PyPI top 15,000 projects using regex _PyInterpreterState_GetEvalFrameFunc|_PyInterpreterState_SetEvalFrameFunc|_PyEval_EvalFrameDefault didn't find _PyInterpreterState_GetEvalFrameFunc() or _PyInterpreterState_SetEvalFrameFunc() usage, but found _PyEval_EvalFrameDefault() usage in 7 projects:

• cython (3.1.4)
• debugpy (1.8.17)
• memray (1.18.0)
• numpy (2.3.3)
• ptvsd (4.3.2)
• pystack (1.5.1)
• river (0.22.0)

cython-3.1.4.tar.gz: cython-3.1.4/Cython/Debugger/libpython.py: EVALFRAME = '_PyEval_EvalFrameDefault'
debugpy-1.8.17.tar.gz: debugpy-1.8.17/src/debugpy/_vendored/pydevd/_pydevd_frame_eval/pydevd_frame_evaluator.pxd: #define CALL_EvalFrameDefault_39(a, b, c)    _PyEval_EvalFrameDefault(a, b, c)
debugpy-1.8.17.tar.gz: debugpy-1.8.17/src/debugpy/_vendored/pydevd/_pydevd_frame_eval/pydevd_frame_evaluator.pxd: #define CALL_EvalFrameDefault_38(a, b)    _PyEval_EvalFrameDefault(a, b)
debugpy-1.8.17.tar.gz: debugpy-1.8.17/src/debugpy/_vendored/pydevd/_pydevd_frame_eval/pydevd_frame_evaluator.pxd: PyObject* _PyEval_EvalFrameDefault(...)
debugpy-1.8.17.tar.gz: debugpy-1.8.17/src/debugpy/_vendored/pydevd/_pydevd_frame_eval/pydevd_frame_evaluator.pyx: state.interp.eval_frame = _PyEval_EvalFrameDefault
debugpy-1.8.17.tar.gz: debugpy-1.8.17/src/debugpy/_vendored/pydevd/_pydevd_frame_eval/pydevd_frame_evaluator.template.pyx: state.interp.eval_frame = _PyEval_EvalFrameDefault
memray-1.18.0.tar.gz: memray-1.18.0/src/memray/_memray.pyx: if pidx >= 0 and "_PyEval_EvalFrameDefault" in symbol:
memray-1.18.0.tar.gz: memray-1.18.0/src/memray/_memray.pyx: if function != "_PyEval_EvalFrameDefault":
memray-1.18.0.tar.gz: memray-1.18.0/src/memray/reporters/frame_tools.py: if "PyEval_EvalFrameEx" in symbol or "_PyEval_EvalFrameDefault" in symbol:
numpy-2.3.3.tar.gz: numpy-2.3.3/numpy/_core/src/multiarray/temp_elide.c: #define PYFRAMEEVAL_FUNC "_PyEval_EvalFrameDefault"
ptvsd-4.3.2.zip: ptvsd-4.3.2/src/ptvsd/_vendored/pydevd/_pydevd_frame_eval/pydevd_frame_evaluator.pxd: PyObject* _PyEval_EvalFrameDefault(PyFrameObject *frame, int exc)
ptvsd-4.3.2.zip: ptvsd-4.3.2/src/ptvsd/_vendored/pydevd/_pydevd_frame_eval/pydevd_frame_evaluator.pyx: return _PyEval_EvalFrameDefault(frame_obj, exc)
ptvsd-4.3.2.zip: ptvsd-4.3.2/src/ptvsd/_vendored/pydevd/_pydevd_frame_eval/pydevd_frame_evaluator.pyx: return _PyEval_EvalFrameDefault(frame_obj, exc)
ptvsd-4.3.2.zip: ptvsd-4.3.2/src/ptvsd/_vendored/pydevd/_pydevd_frame_eval/pydevd_frame_evaluator.pyx: return _PyEval_EvalFrameDefault(frame_obj, exc)
ptvsd-4.3.2.zip: ptvsd-4.3.2/src/ptvsd/_vendored/pydevd/_pydevd_frame_eval/pydevd_frame_evaluator.pyx: return _PyEval_EvalFrameDefault(frame_obj, exc)
ptvsd-4.3.2.zip: ptvsd-4.3.2/src/ptvsd/_vendored/pydevd/_pydevd_frame_eval/pydevd_frame_evaluator.pyx: return _PyEval_EvalFrameDefault(frame_obj, exc)
ptvsd-4.3.2.zip: ptvsd-4.3.2/src/ptvsd/_vendored/pydevd/_pydevd_frame_eval/pydevd_frame_evaluator.pyx: return _PyEval_EvalFrameDefault(frame_obj, exc)
ptvsd-4.3.2.zip: ptvsd-4.3.2/src/ptvsd/_vendored/pydevd/_pydevd_frame_eval/pydevd_frame_evaluator.pyx: return _PyEval_EvalFrameDefault(frame_obj, exc)
ptvsd-4.3.2.zip: ptvsd-4.3.2/src/ptvsd/_vendored/pydevd/_pydevd_frame_eval/pydevd_frame_evaluator.pyx: state.interp.eval_frame = _PyEval_EvalFrameDefault
pystack-1.5.1.tar.gz: pystack-1.5.1/src/pystack/types.py: return "_PyEval_EvalFrameDefault" in symbol
river-0.22.0.tar.gz: river-0.22.0/.venv/lib/python3.13/site-packages/debugpy/_vendored/pydevd/_pydevd_frame_eval/pydevd_frame_evaluator.pxd: #define CALL_EvalFrameDefault_39(a, b, c)    _PyEval_EvalFrameDefault(a, b, c)
river-0.22.0.tar.gz: river-0.22.0/.venv/lib/python3.13/site-packages/debugpy/_vendored/pydevd/_pydevd_frame_eval/pydevd_frame_evaluator.pxd: #define CALL_EvalFrameDefault_38(a, b)    _PyEval_EvalFrameDefault(a, b)
river-0.22.0.tar.gz: river-0.22.0/.venv/lib/python3.13/site-packages/debugpy/_vendored/pydevd/_pydevd_frame_eval/pydevd_frame_evaluator.pxd: PyObject* _PyEval_EvalFrameDefault(...)
river-0.22.0.tar.gz: river-0.22.0/.venv/lib/python3.13/site-packages/debugpy/_vendored/pydevd/_pydevd_frame_eval/pydevd_frame_evaluator.template.pyx: state.interp.eval_frame = _PyEval_EvalFrameDefault

[vstinner]

PyCharm blog post from 2017 about PEP 523 APIs: https://blog.jetbrains.com/pycharm/2017/03/inside-the-debugger-interview-with-elizaveta-shashkova/

What changed in Python 3.6 to allow this?
The new frame evaluation API was introduced to CPython in PEP 523 and it allows to specify a per-interpreter function pointer to handle the evaluation of frames.

[vstinner]

PyTorch Dynamo says that it uses the PEP 523 API: https://docs.pytorch.org/docs/stable/torch.compiler_dynamo_overview.html

Dynamo hooks into the frame evaluation API in CPython (PEP 523) to dynamically modify Python bytecode right before it is executed.

[emmatyping]

Perhaps it would be better to make these PyUnstable functions? I believe there was discussion at PyCon this year (and it seems in the past) about issues with the frame evaluation C API. Looping in a few people who probably have stronger opinions than I:
@markshannon
@jansel
@itamaro

[jansel]

PyTorch's torch.compile uses PEP 523. PyUnstable would be fine for us. We already need to do extra work to support each new Python version primarily due to Python bytecode changes.

[itamaro]

As long as the frame evaluation API provides the functionality along PEP 523 spirit, I don't think it matters to us (cinder jit in this context) what's the exact spelling. Right @DinoV ?

[markshannon]

A code search on PyPI top 15,000 projects using regex...

Those are either:

• a vendored old version of pydevd. All debuggers now use sys.monitoring
• pystack/memray -- They both probe into CPython internals, so don't much care about APIs.

The only use case that matters, and probably the only use case at all, is PyTorch.

I don't want to make the API public because that entails a lot of defining semantics and guarantees about future compatibility. It also effectively makes the interpreter frame part of the public API, and I don't want that either.

Making them PyUnstable seems best.

[DinoV]

As long as the frame evaluation API provides the functionality along PEP 523 spirit, I don't think it matters to us (cinder jit in this context) what's the exact spelling. Right @DinoV ?

Correct, a rename is no big deal, I think this probably just predated the PyUnstable convention.