ctypes does crash on AMD64 FreeBSD Shared 3.x

[vstinner]

test_gc.test_refcount_errors() fails on AMD64 FreeBSD Shared 3.x since build 2367 (May 12 at 6:57 AM, 17 hours ago):
https://buildbot.python.org/all/#/builders/483/builds/2367

FAIL: test_refcount_errors (test.test_gc.GCCallbackTests.test_refcount_errors)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/test_gc.py", line 1269, in test_refcount_errors
    self.assertRegex(stderr,
    ^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: Regex didn't match: b'gcmodule\\.c:[0-9]+: gc_decref: Assertion "gc_get_refs\\(g\\) > 0" failed.' not found in b''

The problem is the call to Py_DecRef() function using ctypes:

            # Simulate the refcount of "a" being too low (compared to the
            # references held on it by live data), but keeping it above zero
            # (to avoid deallocating it):
            import ctypes
            ctypes.pythonapi.Py_DecRef(ctypes.py_object(a))

It does crash with SIGILL on this FreeBSD machine.

test_ctypes also crash:

test_2 (ctypes.test.test_array_in_pointer.Test.test_2) ... Fatal Python error: Segmentation fault

Current thread 0x0000000826cf2000 (most recent call first):
  File "/usr/home/haypo/cpython/Lib/ctypes/__init__.py", line 512 in cast
  File "/usr/home/haypo/cpython/Lib/ctypes/test/test_array_in_pointer.py", line 50 in test_2

test_bytes:

test_from_format (test.test_bytes.BytesTest.test_from_format) ... Fatal Python error: Segmentation fault

Current thread 0x0000000826ffd000 (most recent call first):
  File "/usr/home/haypo/cpython/Lib/test/test_bytes.py", line 1055 in test_from_format

In fact, many tests are failing:

1 test failed:
    test_gc

20 tests skipped:
    test_dbm_gnu test_devpoll test_epoll test_idle test_ioctl
    test_launcher test_msilib test_readline test_spwd test_startfile
    test_tcl test_tix test_tk test_ttk_guionly test_ttk_textonly
    test_turtle test_winconsoleio test_winreg test_winsound
    test_zipfile64

7 re-run tests:
    test_bytes test_code test_ctypes test_embed test_gc test_threading
    test_unicode

6 tests run no tests:
    test_bytes test_code test_ctypes test_embed test_threading
    test_unicode

[vstinner]

The _ctypes extension is linked to system /usr/local/lib/libffi.so.8:

% ldd /usr/home/haypo/cpython/build/lib.freebsd-14.0-CURRENT-amd64-3.12-pydebug/_ctypes.cpython-312d.so
/usr/home/haypo/cpython/build/lib.freebsd-14.0-CURRENT-amd64-3.12-pydebug/_ctypes.cpython-312d.so:
	libffi.so.8 => /usr/local/lib/libffi.so.8 (0x17f2bde4c000)
	(...)

[vstinner]

@koobs: Did you update libffi recently on this machine?

[koobs]

Yep. 3.4.2 via https://cgit.freebsd.org/ports/commit/?id=abcd54a26fee084382d799f5ce14f397464bba80

[vstinner]

% pkg version -n libffi 
libffi-3.4.2                       =

Yep. 3.4.2 via https://cgit.freebsd.org/ports/commit/?id=abcd54a26fee084382d799f5ce14f397464bba80

It says:

devel/libffi: Update to 3.4.2 Drop lots of upstreamed patches.

Maybe some patches were useful.

[koobs]

Initial isolation test. Testing of the locally installed python 3.7 and 3.9 passes test_gc (pythons build against latest libffi)

[koobs@140-CURRENT-amd64-564d:/usr/home/koobs/repos/freebsd/ports] python3.9 -m test test.test_gc
0:00:00 load avg: 3.63 Run tests sequentially
0:00:00 load avg: 3.63 [1/1] test.test_gc

== Tests result: SUCCESS ==

1 test OK.

Total duration: 6.6 sec
Tests result: SUCCESS
[koobs@140-CURRENT-amd64-564d:/usr/home/koobs/repos/freebsd/ports] python3.7 -m test test.test_gc
0:00:00 load avg: 3.56 Run tests sequentially
0:00:00 load avg: 3.56 [1/1] test.test_gc

== Tests result: SUCCESS ==

1 test OK.

Total duration: 3.4 sec
Tests result: SUCCESS

Edit: I note test_refcount_errors is skipped for non-debug builds:

python3.9 -m test test.test_gc -v |grep refcount
test_refcount_errors (test.test_gc.GCCallbackTests) ... skipped 'built with -NDEBUG'

[koobs]

Rebuilding python 3.9 with debug ...

[koobs]

Reproduced with a Python 3.9 debug build:

======================================================================
FAIL: test_refcount_errors (test.test_gc.GCCallbackTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/test/test_gc.py", line 1200, in test_refcount_errors
    self.assertRegex(stderr,
AssertionError: Regex didn't match: b'gcmodule\\.c:[0-9]+: gc_decref: Assertion "gc_get_refs\\(g\\) > 0" failed.' not found in b''

----------------------------------------------------------------------
Ran 47 tests in 4.045s

FAILED (failures=1)
test test.test_gc failed
test.test_gc failed (1 failure)

== Tests result: FAILURE ==

1 test failed:
    test.test_gc

Total duration: 4.2 sec
Tests result: FAILURE

[koobs]

Likewise ctypes.test.test_array_in_pointer

[koobs@140-CURRENT-amd64-564d:/usr/home/koobs/repos/freebsd/ports] python3.9 -m ctypes.test.test_array_in_pointer
.zsh: segmentation fault (core dumped)  python3.9 -m ctypes.test.test_array_in_pointer

Rebuilding local gdb (libpython.so ABI / filename changed with debug)...

[koobs]

gdb --args python3.9 -m ctypes.test.test_array_in_pointer

GNU gdb (GDB) 12.1 [GDB v12.1 for FreeBSD]
...<snip>...
Reading symbols from python3.9...
(gdb) run
Starting program: /usr/local/bin/python3.9 -m ctypes.test.test_array_in_pointer

...<snip>...

Program received signal SIGSEGV, Segmentation fault.
Address not mapped to object.
0x0000000801960404 in _Py_INCREF (op=0x7fff00000004) at ./Include/object.h:408
408     ./Include/object.h: No such file or directory.
(gdb) bt full
#0  0x0000000801960404 in _Py_INCREF (op=0x7fff00000004) at ./Include/object.h:408
No locals.
#1  0x0000000801964535 in O_get (ptr=0x7fffffffa880, size=8) at Modules/_ctypes/cfield.c:1120
        ob = 0x7fff00000004
#2  0x000000080195dfa1 in GetResult (restype=0x801645830, result=0x7fffffffa880, checker=0x0) at Modules/_ctypes/callproc.c:983
        dict = 0x8018ebc80
        retval = 0x8018ebcc8
        v = 0x7fffffffa840
#3  0x000000080195d649 in _ctypes_callproc (pProc=0x801952870 <cast>, argtuple=0x801bc73b0, flags=4357, argtypes=0x8019a4a10, restype=0x801645830, checker=0x0) at Modules/_ctypes/callproc.c:1301
        i = 3
        n = 3
        argcount = 3
        argtype_count = 3
        resbuf = 0x7fffffffa880
        args = 0x7fffffffa890
        pa = 0x7fffffffa8f0
        atypes = 0x7fffffffa840
        rtype = 0x8018ebcc8
        avalues = 0x7fffffffa860
        retval = 0x0
#4  0x0000000801950e8d in PyCFuncPtr_call (self=0x8018fc590, inargs=0x801bc73b0, kwds=0x0) at Modules/_ctypes/_ctypes.c:4201
        restype = 0x801645830
        converters = 0x8019a4a10
        checker = 0x0
        argtypes = 0x8019a4110
        dict = 0x80193e9b0
        result = 0x0
        callargs = 0x801bc73b0
        errcheck = 0x0
        pProc = 0x801952870 <cast>
        inoutmask = 0
        outmask = 0
        numretvals = 0
#5  0x000000080041d4a9 in _PyObject_MakeTpCall () from /usr/local/lib/libpython3.9d.so.1.0
No symbol table info available.
#6  0x000000080052d088 in call_function () from /usr/local/lib/libpython3.9d.so.1.0
No symbol table info available.
#7  0x0000000800527d5c in _PyEval_EvalFrameDefault () from /usr/local/lib/libpython3.9d.so.1.0
No symbol table info available.
#8  0x000000080041e350 in function_code_fastcall () from /usr/local/lib/libpython3.9d.so.1.0
No symbol table info available.
#9  0x000000080052f887 in _PyObject_VectorcallTstate () from /usr/local/lib/libpython3.9d.so.1.0
No symbol table info available.
#10 0x000000080052d088 in call_function () from /usr/local/lib/libpython3.9d.so.1.0
No symbol table info available.
#11 0x0000000800527d5c in _PyEval_EvalFrameDefault () from /usr/local/lib/libpython3.9d.so.1.0
No symbol table info available.
#12 0x000000080052de88 in _PyEval_EvalCode () from /usr/local/lib/libpython3.9d.so.1.0
No symbol table info available.
#13 0x000000080041e170 in _PyFunction_Vectorcall () from /usr/local/lib/libpython3.9d.so.1.0
No symbol table info available.
#14 0x00000008004218c7 in _PyObject_VectorcallTstate () from /usr/local/lib/libpython3.9d.so.1.0
No symbol table info available.
#15 0x0000000800420c99 in method_vectorcall () from /usr/local/lib/libpython3.9d.so.1.0
No symbol table info available.
#16 0x000000080052f887 in _PyObject_VectorcallTstate () from /usr/local/lib/libpython3.9d.so.1.0
No symbol table info available.
#17 0x000000080052d088 in call_function () from /usr/local/lib/libpython3.9d.so.1.0
No symbol table info available.
#18 0x0000000800527d5c in _PyEval_EvalFrameDefault () from /usr/local/lib/libpython3.9d.so.1.0
No symbol table info available.
#19 0x000000080041e350 in function_code_fastcall () from /usr/local/lib/libpython3.9d.so.1.0
No symbol table info available.
#20 0x000000080052f887 in _PyObject_VectorcallTstate () from /usr/local/lib/libpython3.9d.so.1.0
No symbol table info available.
#21 0x000000080052d088 in call_function () from /usr/local/lib/libpython3.9d.so.1.0
No symbol table info available.
#22 0x0000000800527c2b in _PyEval_EvalFrameDefault () from /usr/local/lib/libpython3.9d.so.1.0
No symbol table info available.
#23 0x000000080052de88 in _PyEval_EvalCode () from /usr/local/lib/libpython3.9d.so.1.0
No symbol table info available.
#24 0x000000080041e170 in _PyFunction_Vectorcall () from /usr/local/lib/libpython3.9d.so.1.0
No symbol table info available.
#25 0x00000008004218c7 in _PyObject_VectorcallTstate () from /usr/local/lib/libpython3.9d.so.1.0
No symbol table info available.
#26 0x0000000800420d11 in method_vectorcall () from /usr/local/lib/libpython3.9d.so.1.0
No symbol table info available.
#27 0x0000000800528376 in _PyEval_EvalFrameDefault () from /usr/local/lib/libpython3.9d.so.1.0
No symbol table info available.
#28 0x000000080052de88 in _PyEval_EvalCode () from /usr/local/lib/libpython3.9d.so.1.0
No symbol table info available.
#29 0x000000080041e170 in _PyFunction_Vectorcall () from /usr/local/lib/libpython3.9d.so.1.0
No symbol table info available.
#30 0x000000080041d1a8 in _PyObject_FastCallDictTstate () from /usr/local/lib/libpython3.9d.so.1.0
No symbol table info available.
#31 0x000000080041e81f in _PyObject_Call_Prepend () from /usr/local/lib/libpython3.9d.so.1.0

[koobs]

After rebuilding libffi without -fcf-protection and Python, test_gc and ctypes.test.test_array_in_pointer now pass.

You can force-rebuild of the failed builds if you like, or if there are builds in the queue, let them run, they will pickup the rebuilt libffi library.

[koobs@140-CURRENT-amd64-564d:/usr/home/koobs/repos/freebsd/ports/lang/python39] python3.9 -m ctypes.test.test_array_in_pointer
..
----------------------------------------------------------------------
Ran 2 tests in 0.001s

OK

 python3.9 -m test test.test_gc
0:00:00 load avg: 0.64 Run tests sequentially
0:00:00 load avg: 0.64 [1/1] test.test_gc

== Tests result: SUCCESS ==

1 test OK.

Total duration: 4.1 sec
Tests result: SUCCESS

[vstinner]

Tests pass again, I close the issue. Thanks @koobs for investigating. The issue seems to be in libffi, not directly related to Python. The issue is now tracked in the libffi bug tracker: libffi/libffi#710