Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Py_TRACE_REFS is not compatible with Py_LIMITED_API #108634

Closed
2 tasks done
vstinner opened this issue Aug 29, 2023 · 1 comment
Closed
2 tasks done

Py_TRACE_REFS is not compatible with Py_LIMITED_API #108634

vstinner opened this issue Aug 29, 2023 · 1 comment
Labels
type-bug An unexpected behavior, bug, or error

Comments

@vstinner
Copy link
Member

vstinner commented Aug 29, 2023 •
edited by bedevere-bot

Bug report

Checklist

  • I am confident this is a bug in CPython, not a bug in a third-party project
  • I have searched the CPython issue tracker,
    and am confident this bug has not been reported before

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

No response

A clear and concise description of the bug:

Trying to build a C extension targetting the limited C API (defining Py_LIMITED_API) fails if Python is configured with ./configure --with-trace-refs. Example with PR #108573:

In file included from ./Include/Python.h:44,
                 from ./Modules/_stat.c:16:
./Include/object.h:62:4: error: #error Py_LIMITED_API is incompatible with Py_TRACE_REFS
   62 | #  error Py_LIMITED_API is incompatible with Py_TRACE_REFS
      |    ^~~~~
make: *** [Makefile:3194: Modules/_stat.o] Error 1

The #error comes from Include/object.h:

#if defined(Py_LIMITED_API) && defined(Py_TRACE_REFS)
#  error Py_LIMITED_API is incompatible with Py_TRACE_REFS
#endif

The problem is that the PyObject ABI is different: Py_TRACE_REFS adds two members to PyObject structure:

    PyObject *_ob_next;
    PyObject *_ob_prev;

One solution to make Py_TRACE_REFS compatible with Py_LIMITED_API would be to not add these two members to PyObject, but store the double-linked list outside PyObject. Store it in a different structure, so PyObject stays ABI compatible.

Linked PRs
@vstinner vstinner added the type-bug An unexpected behavior, bug, or error label Aug 29, 2023
@vstinner vstinner mentioned this issue Aug 29, 2023
Closed
@erlend-aasland
Copy link
Contributor

See also #63083 and #73104

@vstinner vstinner mentioned this issue Aug 29, 2023
Closed
vstinner added a commit to vstinner/cpython that referenced this issue Aug 30, 2023
@vstinner
pythongh-108634: Use a hashtable for Py_TRACE_REFS
f7e7106
vstinner added a commit to vstinner/cpython that referenced this issue Aug 30, 2023
@vstinner
pythongh-108634: Use a hashtable for Py_TRACE_REFS
9820187 
Change Py_TRACE_REFS build:

* Remove _PyObject_EXTRA_INIT macro.
* The PyObject structure no longer has two extra members (_ob_prev
  and _ob_next).
* Use an hash table (_Py_hashtable_t) to trace references (all
  objects): PyInterpreterState.object_state.refchain.
* test_sys now tests sys.getobjects().
* Limited C API extensions can now be built with Py_TRACE_REFS:
  xxlimited, xxlimited_35, _testclinic_limited.
* Py_TRACE_REFS build is now ABI compatible with release build and
  debug build.
* No longer rename PyModule_Create2() and PyModule_FromDefAndSpec2()
  functions to PyModule_Create2TraceRefs() and
  PyModule_FromDefAndSpec2TraceRefs().
* Add test.support.Py_TRACE_REFS constant.
* test_exceptions skips test_recursion_normalizing_with_no_memory()
  and test_memory_error_in_PyErr_PrintEx() if Python is built with
  Py_TRACE_REFS.
* test_repl skips test_no_memory().
* test_capi skisp test_set_nomemory().
vstinner added a commit to vstinner/cpython that referenced this issue Aug 30, 2023
@vstinner
pythongh-108634: Py_TRACE_REFS uses a hash table
b45005c 
Change Py_TRACE_REFS build:

* Remove _PyObject_EXTRA_INIT macro.
* The PyObject structure no longer has two extra members (_ob_prev
  and _ob_next).
* Use a hash table (_Py_hashtable_t) to trace references (all
  objects): PyInterpreterState.object_state.refchain.
* test_sys now tests sys.getobjects().
* Limited C API extensions can now be built with Py_TRACE_REFS:
  xxlimited, xxlimited_35, _testclinic_limited.
* Py_TRACE_REFS build is now ABI compatible with release build and
  debug build.
* No longer rename PyModule_Create2() and PyModule_FromDefAndSpec2()
  functions to PyModule_Create2TraceRefs() and
  PyModule_FromDefAndSpec2TraceRefs().
* Add test.support.Py_TRACE_REFS constant.
* test_exceptions skips test_recursion_normalizing_with_no_memory()
  and test_memory_error_in_PyErr_PrintEx() if Python is built with
  Py_TRACE_REFS.
* test_repl skips test_no_memory().
* test_capi skisp test_set_nomemory().
vstinner added a commit to vstinner/cpython that referenced this issue Aug 30, 2023
@vstinner
pythongh-108634: Py_TRACE_REFS uses a hash table
c3f537d 
Python built with "configure --with-trace-refs" (tracing references)
is now ABI compatible with Python built in release mode. Moreover, it
now also supports the Limited API.

Change Py_TRACE_REFS build:

* Remove _PyObject_EXTRA_INIT macro.
* The PyObject structure no longer has two extra members (_ob_prev
  and _ob_next).
* Use a hash table (_Py_hashtable_t) to trace references (all
  objects): PyInterpreterState.object_state.refchain.
* test_sys now tests sys.getobjects().
* Limited C API extensions can now be built with Py_TRACE_REFS:
  xxlimited, xxlimited_35, _testclinic_limited.
* Py_TRACE_REFS build is now ABI compatible with release build and
  debug build.
* No longer rename PyModule_Create2() and PyModule_FromDefAndSpec2()
  functions to PyModule_Create2TraceRefs() and
  PyModule_FromDefAndSpec2TraceRefs().
* Add test.support.Py_TRACE_REFS constant.
* test_exceptions skips test_recursion_normalizing_with_no_memory()
  and test_memory_error_in_PyErr_PrintEx() if Python is built with
  Py_TRACE_REFS.
* test_repl skips test_no_memory().
* test_capi skisp test_set_nomemory().
vstinner added a commit to vstinner/cpython that referenced this issue Aug 30, 2023
@vstinner
pythongh-108634: Py_TRACE_REFS uses a hash table
b382d58 
Python built with "configure --with-trace-refs" (tracing references)
is now ABI compatible with Python release build and debug build.
Moreover, it now also supports the Limited API.

Change Py_TRACE_REFS build:

* Remove _PyObject_EXTRA_INIT macro.
* The PyObject structure no longer has two extra members (_ob_prev
  and _ob_next).
* Use a hash table (_Py_hashtable_t) to trace references (all
  objects): PyInterpreterState.object_state.refchain.
* test_sys now tests sys.getobjects().
* Limited C API extensions can now be built with Py_TRACE_REFS:
  xxlimited, xxlimited_35, _testclinic_limited.
* Py_TRACE_REFS build is now ABI compatible with release build and
  debug build.
* No longer rename PyModule_Create2() and PyModule_FromDefAndSpec2()
  functions to PyModule_Create2TraceRefs() and
  PyModule_FromDefAndSpec2TraceRefs().
* Add test.support.Py_TRACE_REFS constant.
* test_exceptions skips test_recursion_normalizing_with_no_memory()
  and test_memory_error_in_PyErr_PrintEx() if Python is built with
  Py_TRACE_REFS.
* test_repl skips test_no_memory().
* test_capi skisp test_set_nomemory().
vstinner added a commit to vstinner/cpython that referenced this issue Aug 30, 2023
@vstinner
pythongh-108634: Py_TRACE_REFS uses a hash table
a44bea2 
Python built with "configure --with-trace-refs" (tracing references)
is now ABI compatible with Python release build and debug build.
Moreover, it now also supports the Limited API.

Change Py_TRACE_REFS build:

* Remove _PyObject_EXTRA_INIT macro.
* The PyObject structure no longer has two extra members (_ob_prev
  and _ob_next).
* Use a hash table (_Py_hashtable_t) to trace references (all
  objects): PyInterpreterState.object_state.refchain.
* Py_TRACE_REFS build is now ABI compatible with release build and
  debug build.
* Limited C API extensions can now be built with Py_TRACE_REFS:
  xxlimited, xxlimited_35, _testclinic_limited.
* No longer rename PyModule_Create2() and PyModule_FromDefAndSpec2()
  functions to PyModule_Create2TraceRefs() and
  PyModule_FromDefAndSpec2TraceRefs().
* _Py_PrintReferenceAddresses() is now called before
  finalize_interp_delete() which deletes the refchain hash table.

Test changes for Py_TRACE_REFS:

* Add test.support.Py_TRACE_REFS constant.
* Add test_sys.test_getobjects() to test sys.getobjects() function.
* test_exceptions skips test_recursion_normalizing_with_no_memory()
  and test_memory_error_in_PyErr_PrintEx() if Python is built with
  Py_TRACE_REFS.
* test_repl skips test_no_memory().
* test_capi skisp test_set_nomemory().
@bedevere-bot bedevere-bot mentioned this issue Aug 30, 2023
Merged
vstinner added a commit to vstinner/cpython that referenced this issue Aug 30, 2023
@vstinner
pythongh-108634: Py_TRACE_REFS uses a hash table
28f41a5 
Python built with "configure --with-trace-refs" (tracing references)
is now ABI compatible with Python release build and debug build.
Moreover, it now also supports the Limited API.

Change Py_TRACE_REFS build:

* Remove _PyObject_EXTRA_INIT macro.
* The PyObject structure no longer has two extra members (_ob_prev
  and _ob_next).
* Use a hash table (_Py_hashtable_t) to trace references (all
  objects): PyInterpreterState.object_state.refchain.
* Py_TRACE_REFS build is now ABI compatible with release build and
  debug build.
* Limited C API extensions can now be built with Py_TRACE_REFS:
  xxlimited, xxlimited_35, _testclinic_limited.
* No longer rename PyModule_Create2() and PyModule_FromDefAndSpec2()
  functions to PyModule_Create2TraceRefs() and
  PyModule_FromDefAndSpec2TraceRefs().
* _Py_PrintReferenceAddresses() is now called before
  finalize_interp_delete() which deletes the refchain hash table.

Test changes for Py_TRACE_REFS:

* Add test.support.Py_TRACE_REFS constant.
* Add test_sys.test_getobjects() to test sys.getobjects() function.
* test_exceptions skips test_recursion_normalizing_with_no_memory()
  and test_memory_error_in_PyErr_PrintEx() if Python is built with
  Py_TRACE_REFS.
* test_repl skips test_no_memory().
* test_capi skisp test_set_nomemory().
vstinner added a commit that referenced this issue Aug 31, 2023
@vstinner
gh-108634: Py_TRACE_REFS uses a hash table (#108663)
13a0007 
Python built with "configure --with-trace-refs" (tracing references)
is now ABI compatible with Python release build and debug build.
Moreover, it now also supports the Limited API.

Change Py_TRACE_REFS build:

* Remove _PyObject_EXTRA_INIT macro.
* The PyObject structure no longer has two extra members (_ob_prev
  and _ob_next).
* Use a hash table (_Py_hashtable_t) to trace references (all
  objects): PyInterpreterState.object_state.refchain.
* Py_TRACE_REFS build is now ABI compatible with release build and
  debug build.
* Limited C API extensions can now be built with Py_TRACE_REFS:
  xxlimited, xxlimited_35, _testclinic_limited.
* No longer rename PyModule_Create2() and PyModule_FromDefAndSpec2()
  functions to PyModule_Create2TraceRefs() and
  PyModule_FromDefAndSpec2TraceRefs().
* _Py_PrintReferenceAddresses() is now called before
  finalize_interp_delete() which deletes the refchain hash table.
* test_tracemalloc find_trace() now also filters by size to ignore
  the memory allocated by _PyRefchain_Trace().

Test changes for Py_TRACE_REFS:

* Add test.support.Py_TRACE_REFS constant.
* Add test_sys.test_getobjects() to test sys.getobjects() function.
* test_exceptions skips test_recursion_normalizing_with_no_memory()
  and test_memory_error_in_PyErr_PrintEx() if Python is built with
  Py_TRACE_REFS.
* test_repl skips test_no_memory().
* test_capi skisp test_set_nomemory().
vstinner added a commit to vstinner/cpython that referenced this issue Sep 1, 2023
@vstinner
pythongh-108634: init_interpreter() returns Py_Status
6e8ba06 
* Replace Py_FatalError() with PyStatus in init_interpreter().
* init_runtime() cannot be called with _initialized=1: add an
  assertion in the caller.
vstinner added a commit to vstinner/cpython that referenced this issue Sep 1, 2023
@vstinner
pythongh-108634: init_interpreter() returns Py_Status
905c4c4 
* Replace Py_FatalError() with PyStatus in init_interpreter() and
  _PyObject_InitState().
* PyInterpreterState_New() falls Py_FatalError() if
  init_interpreter() failed.
* init_runtime() cannot be called with _initialized=1: add an
  assertion in the caller.
* _PyErr_SetFromPyStatus() now raises RuntimeError, instead of
  ValueError. It can no call PyErr_NoMemory() if it detects
  _PyStatus_NO_MEMORY().
vstinner added a commit to vstinner/cpython that referenced this issue Sep 1, 2023
@vstinner
pythongh-108634: init_interpreter() returns Py_Status
c6a35fa 
* Add _PyInterpreterState_New() function.
* Replace Py_FatalError() with PyStatus in init_interpreter() and
  _PyObject_InitState().
* PyInterpreterState_New() now raises a RuntimeError instead of
  calling Py_FatalError() on init_interpreter() failure.
* _PyErr_SetFromPyStatus() now raises RuntimeError, instead of
  ValueError. It can no call PyErr_NoMemory() if it detects
  _PyStatus_NO_MEMORY().
* init_runtime() cannot be called with _initialized=1: add an
  assertion in the caller instead.
vstinner added a commit to vstinner/cpython that referenced this issue Sep 1, 2023
@vstinner
pythongh-108634: PyInterpreterState_New() no longer calls Py_FatalErr…
bc2bb7b 
…or()

PyInterpreterState_New() now raises a RuntimeError instead of calling
Py_FatalError() on init_interpreter() failure. And
pycore_create_interpreter() now reports status exception to its
caller rather than calling Py_FatalError().

* Add _PyInterpreterState_New() function.
* Replace Py_FatalError() with PyStatus in init_interpreter() and
  _PyObject_InitState().
* _PyErr_SetFromPyStatus() now raises RuntimeError, instead of
  ValueError. It can no call PyErr_NoMemory() if it detects
  _PyStatus_NO_MEMORY().
* init_runtime() cannot be called with _initialized=1: add an
  assertion in the caller instead.
vstinner added a commit to vstinner/cpython that referenced this issue Sep 1, 2023
@vstinner
pythongh-108634: PyInterpreterState_New() no longer calls Py_FatalErr…
fc03c9b 
…or()

PyInterpreterState_New() now raises a RuntimeError, instead of calling
Py_FatalError() on error. And pycore_create_interpreter() now returns
a status exception on error, rather than calling Py_FatalError().

* Add _PyInterpreterState_New() function.
* Replace Py_FatalError() with PyStatus in init_interpreter() and
  _PyObject_InitState().
* _PyErr_SetFromPyStatus() now raises RuntimeError, instead of
  ValueError. It can no call PyErr_NoMemory() if it detects
  _PyStatus_NO_MEMORY().
* init_runtime() cannot be called with _initialized=1: add an
  assertion in the caller instead.
vstinner added a commit to vstinner/cpython that referenced this issue Sep 1, 2023
@vstinner
pythongh-108634: PyInterpreterState_New() no longer calls Py_FatalErr…
d25c5da 
…or()

PyInterpreterState_New() now raises a RuntimeError, instead of calling
Py_FatalError() on error. And pycore_create_interpreter() now returns
a status exception on error, rather than calling Py_FatalError().

* Add _PyInterpreterState_New() function.
* Replace Py_FatalError() with PyStatus in init_interpreter() and
  _PyObject_InitState().
* _PyErr_SetFromPyStatus() now raises RuntimeError, instead of
  ValueError. It can now call PyErr_NoMemory() if it detects
  _PyStatus_NO_MEMORY().
* init_runtime() cannot be called with _initialized=1: add an
  assertion in the caller instead.
@bedevere-bot bedevere-bot mentioned this issue Sep 1, 2023
Merged
vstinner added a commit to vstinner/cpython that referenced this issue Sep 1, 2023
@vstinner
pythongh-108634: Add _PyInterpreterState_New() function
04627a8 
pycore_create_interpreter() now returns a status exception on error,
rather than calling Py_FatalError().

* PyInterpreterState_New() now calls Py_ExitStatusException() instead
  of calling Py_FatalError() directly.
* Replace Py_FatalError() with PyStatus in init_interpreter() and
  _PyObject_InitState().
* _PyErr_SetFromPyStatus() now raises RuntimeError, instead of
  ValueError. It can now call PyErr_NoMemory() if it detects
  _PyStatus_NO_MEMORY().
* init_runtime() cannot be called with _initialized=1: add an
  assertion in the caller instead.
vstinner added a commit that referenced this issue Sep 1, 2023
@vstinner
gh-108634: PyInterpreterState_New() no longer calls Py_FatalError() (#…
b936cf4 
…108748)

pycore_create_interpreter() now returns a status, rather than
calling Py_FatalError().

* PyInterpreterState_New() now calls Py_ExitStatusException() instead
  of calling Py_FatalError() directly.
* Replace Py_FatalError() with PyStatus in init_interpreter() and
  _PyObject_InitState().
* _PyErr_SetFromPyStatus() now raises RuntimeError, instead of
  ValueError. It can now call PyErr_NoMemory(), raise MemoryError,
  if it detects _PyStatus_NO_MEMORY() error message.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type-bug An unexpected behavior, bug, or error
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@vstinner @erlend-aasland