Segfault in test suite with Python 3.14

[cjwatson]

I've been looking at https://bugs.debian.org/1117901, reporting that the test suite segfaults with Python 3.14. I'd assumed that the latest release would fix this since you're publishing 3.14 wheels, but it seems that it doesn't. Indeed, it's consistently reproducible on git master.

To reproduce, I started a Debian unstable container, installed all the build-dependencies (sudo apt build-dep python-confluent-kafka is a quick way to get most of them; I commented out tink from requirements/requirements-rules.txt since building that needs installing Bazel which is a lot of effort that's incidental to this problem), and ran tox -e py314 -- -v tests/test_Admin.py::test_create_topics_api:

platform linux -- Python 3.14.0, pytest-9.0.1, pluggy-1.5.0 -- /home/cjwatson/src/python/confluent-kafka-python/.tox/py314/bin/python3
cachedir: .tox/py314/.pytest_cache
rootdir: /home/cjwatson/src/python/confluent-kafka-python
configfile: pyproject.toml (WARNING: ignoring pytest config in tox.ini!)
plugins: cov-7.0.0, anyio-4.12.0, timeout-2.4.0, asyncio-1.3.0, requests-mock-1.12.1, respx-0.22.0
timeout: 600.0s
timeout method: signal
timeout func_only: False
asyncio: mode=Mode.AUTO, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function
collected 1 item

tests/test_Admin.py::test_create_topics_api Fatal Python error: Segmentation fault

Current thread 0x00007f11c5ee3100 [python3] (most recent call first):
  File "/home/cjwatson/src/python/confluent-kafka-python/.tox/py314/lib/python3.14/site-packages/pluggy/_callers.py", line 69 in _multicall
  File "/home/cjwatson/src/python/confluent-kafka-python/.tox/py314/lib/python3.14/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/home/cjwatson/src/python/confluent-kafka-python/.tox/py314/lib/python3.14/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/home/cjwatson/src/python/confluent-kafka-python/.tox/py314/lib/python3.14/site-packages/_pytest/runner.py", line 249 in call_and_report
  File "/home/cjwatson/src/python/confluent-kafka-python/.tox/py314/lib/python3.14/site-packages/_pytest/runner.py", line 137 in runtestprotocol
  File "/home/cjwatson/src/python/confluent-kafka-python/.tox/py314/lib/python3.14/site-packages/_pytest/runner.py", line 118 in pytest_runtest_protocol
  File "/home/cjwatson/src/python/confluent-kafka-python/.tox/py314/lib/python3.14/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/home/cjwatson/src/python/confluent-kafka-python/.tox/py314/lib/python3.14/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/home/cjwatson/src/python/confluent-kafka-python/.tox/py314/lib/python3.14/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/home/cjwatson/src/python/confluent-kafka-python/.tox/py314/lib/python3.14/site-packages/_pytest/main.py", line 396 in pytest_runtestloop
  File "/home/cjwatson/src/python/confluent-kafka-python/.tox/py314/lib/python3.14/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/home/cjwatson/src/python/confluent-kafka-python/.tox/py314/lib/python3.14/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/home/cjwatson/src/python/confluent-kafka-python/.tox/py314/lib/python3.14/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/home/cjwatson/src/python/confluent-kafka-python/.tox/py314/lib/python3.14/site-packages/_pytest/main.py", line 372 in _main
  File "/home/cjwatson/src/python/confluent-kafka-python/.tox/py314/lib/python3.14/site-packages/_pytest/main.py", line 318 in wrap_session
  File "/home/cjwatson/src/python/confluent-kafka-python/.tox/py314/lib/python3.14/site-packages/_pytest/main.py", line 365 in pytest_cmdline_main
  File "/home/cjwatson/src/python/confluent-kafka-python/.tox/py314/lib/python3.14/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/home/cjwatson/src/python/confluent-kafka-python/.tox/py314/lib/python3.14/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/home/cjwatson/src/python/confluent-kafka-python/.tox/py314/lib/python3.14/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/home/cjwatson/src/python/confluent-kafka-python/.tox/py314/lib/python3.14/site-packages/_pytest/config/__init__.py", line 197 in main
  File "/home/cjwatson/src/python/confluent-kafka-python/.tox/py314/lib/python3.14/site-packages/_pytest/config/__init__.py", line 221 in console_main
  File "/home/cjwatson/src/python/confluent-kafka-python/.tox/py314/lib/python3.14/site-packages/pytest/__main__.py", line 9 in <module>
  File "<frozen runpy>", line 88 in _run_code
  File "<frozen runpy>", line 198 in _run_module_as_main

Current thread's C stack trace (most recent call first):
  Binary file "/home/cjwatson/src/python/confluent-kafka-python/.tox/py314/bin/python3", at _Py_DumpStack+0x4a [0x471b89]
  Binary file "/home/cjwatson/src/python/confluent-kafka-python/.tox/py314/bin/python3" [0x4eb56f]
  Binary file "/lib/x86_64-linux-gnu/libc.so.6", at +0x3fdf0 [0x7f11c5f23df0]
  Binary file "/home/cjwatson/src/python/confluent-kafka-python/.tox/py314/bin/python3", at PyList_New+0x31 [0x4f3eb1]
  Binary file "/home/cjwatson/src/python/confluent-kafka-python/.tox/py314/bin/python3", at _PyEval_EvalFrameDefault+0x2d52 [0x537352]
  Binary file "/home/cjwatson/src/python/confluent-kafka-python/.tox/py314/bin/python3" [0x572bc4]
  Binary file "/home/cjwatson/src/python/confluent-kafka-python/.tox/py314/bin/python3" [0x51de3e]
  Binary file "/home/cjwatson/src/python/confluent-kafka-python/.tox/py314/bin/python3" [0x6152df]
  Binary file "/home/cjwatson/src/python/confluent-kafka-python/.tox/py314/bin/python3", at _PyObject_MakeTpCall+0x36c [0x51657c]
  Binary file "/home/cjwatson/src/python/confluent-kafka-python/.tox/py314/bin/python3", at _PyEval_EvalFrameDefault+0x65a0 [0x53aba0]
  Binary file "/home/cjwatson/src/python/confluent-kafka-python/.tox/py314/bin/python3" [0x572bc4]
  Binary file "/home/cjwatson/src/python/confluent-kafka-python/.tox/py314/bin/python3" [0x51de3e]
  Binary file "/home/cjwatson/src/python/confluent-kafka-python/.tox/py314/bin/python3" [0x6152df]
  Binary file "/home/cjwatson/src/python/confluent-kafka-python/.tox/py314/bin/python3", at _PyObject_MakeTpCall+0x36c [0x51657c]
  Binary file "/home/cjwatson/src/python/confluent-kafka-python/.tox/py314/bin/python3", at _PyEval_EvalFrameDefault+0xe9fb [0x542ffb]
  Binary file "/home/cjwatson/src/python/confluent-kafka-python/.tox/py314/bin/python3" [0x572bc4]
  Binary file "/home/cjwatson/src/python/confluent-kafka-python/.tox/py314/bin/python3" [0x51de3e]
  Binary file "/home/cjwatson/src/python/confluent-kafka-python/.tox/py314/bin/python3" [0x6152df]
  Binary file "/home/cjwatson/src/python/confluent-kafka-python/.tox/py314/bin/python3", at _PyObject_MakeTpCall+0x36c [0x51657c]
  Binary file "/home/cjwatson/src/python/confluent-kafka-python/.tox/py314/bin/python3", at _PyEval_EvalFrameDefault+0xe9fb [0x542ffb]
  Binary file "/home/cjwatson/src/python/confluent-kafka-python/.tox/py314/bin/python3" [0x572bc4]
  Binary file "/home/cjwatson/src/python/confluent-kafka-python/.tox/py314/bin/python3" [0x51de3e]
  Binary file "/home/cjwatson/src/python/confluent-kafka-python/.tox/py314/bin/python3" [0x6152df]
  Binary file "/home/cjwatson/src/python/confluent-kafka-python/.tox/py314/bin/python3", at _PyObject_MakeTpCall+0x36c [0x51657c]
  Binary file "/home/cjwatson/src/python/confluent-kafka-python/.tox/py314/bin/python3", at _PyEval_EvalFrameDefault+0xe9fb [0x542ffb]
  Binary file "/home/cjwatson/src/python/confluent-kafka-python/.tox/py314/bin/python3", at PyEval_EvalCode+0xf1 [0x52f451]
  Binary file "/home/cjwatson/src/python/confluent-kafka-python/.tox/py314/bin/python3" [0x686577]
  Binary file "/home/cjwatson/src/python/confluent-kafka-python/.tox/py314/bin/python3", at PyObject_Vectorcall+0x32 [0x51cb82]
  Binary file "/home/cjwatson/src/python/confluent-kafka-python/.tox/py314/bin/python3", at _PyEval_EvalFrameDefault+0x37ed [0x537ded]
  Binary file "/home/cjwatson/src/python/confluent-kafka-python/.tox/py314/bin/python3" [0x572bc4]
  Binary file "/home/cjwatson/src/python/confluent-kafka-python/.tox/py314/bin/python3" [0x6a0fa0]
  Binary file "/home/cjwatson/src/python/confluent-kafka-python/.tox/py314/bin/python3", at Py_RunMain+0x1a9 [0x69fde9]
  <truncated rest of calls>

Extension modules: charset_normalizer.md, requests.packages.charset_normalizer.md, requests.packages.chardet.md, confluent_kafka.cimpl (total: 4)
py314: exit -11 (0.95 seconds) /home/cjwatson/src/python/confluent-kafka-python> python3 -m pytest --timeout 600 --ignore=tmp-build -v tests/test_Admin.py::test_create_topics_api pid=11937
  py314: FAIL code -11 (8.52=setup[4.67]+cmd[1.28,1.59,0.03,0.95] seconds)
  evaluation failed :( (8.55 seconds)

I suspect 3.14 may not be fully in your test matrix yet, so I'd suggest adding it and seeing if you can reproduce this, which I'm sure would be a lot quicker than round-tripping through me. But if you can't, please let me know what I can do to help debug this - the traceback is pretty obscure and it would probably take me a while to figure it out on my own, but I'm happy to be guided in an investigation if needed. Thanks!

[MSeal]

Hmm, I've narrowed this down to the last line which was introduced as part of the mypy changes. I'll see if @fangnx can take a look but if you're able to debug why 3.14 is exploding on this line -- it appears to happen after the check for the TypeError on cleanup.

    with pytest.raises(TypeError):
        a.create_topics([NewTopic("mytopic", 3, 2, config=["fails", "because not a dict"])])

Further checks show it's caused by creating NewTopic with a list config, which is an invalid input. What's confusing is the C code for this is pretty straight forward in AdminTypes.c and I don't understand why it is failing in this instance:

static int NewTopic_init(PyObject *self0, PyObject *args, PyObject *kwargs) {
        NewTopic *self = (NewTopic *)self0;
        const char *topic;
        static char *kws[] = {"topic",
                              "num_partitions",
                              "replication_factor",
                              "replica_assignment",
                              "config",
                              NULL};

        self->num_partitions     = -1;
        self->replication_factor = -1;
        self->replica_assignment = NULL;
        self->config             = NULL;

        if (!PyArg_ParseTupleAndKeywords(
                args, kwargs, "s|iiOO", kws, &topic, &self->num_partitions,
                &self->replication_factor, &self->replica_assignment,
                &self->config))
                return -1;



        if (self->config) {
                if (!PyDict_Check(self->config)) {
                        PyErr_SetString(PyExc_TypeError,
                                        "config must be a dict of strings");
                        return -1;
                }
                Py_INCREF(self->config);
        }

As for testing, our 3.14 wheel build check doesn't run the full suite, which is how this got missed. I was developing on 3.14 earlier as we just added build support for it and wasn't hitting this failure back before we added mypy enforcement.

[cjwatson]

Thanks, this was enough for me to track it down. PR incoming.