Replace upstream.yml with new python312.yml

[rwgk]

Description

Add new python312.yml, replacing obsolete upstream.yml. The new name is to make it completely obvious what the workflow is for. (We will have to update the name only once a year.)

Main purpose: testing builtins / state dict change started under PR #4293 and continued under the pair of PRs #4329/#4349 (master/smart_holder).

The Python312 workflow was working completely with python-3.12.0-alpha.1-linux-22.04-x64.tar.gz, but the C++ tests fail with alpha.2 and alpha.3. Debugging led deep into the Python sources. This is captured in a comment; further debugging is left for later (help appreciated). The main purpose (above) is sufficiently met via the Python tests + interactive testing of test_embed (ignoring the one failures).

Suggested changelog entry:

[rwgk]

Hi @vstinner, just hoping that a minute of your time may give us a clue: Does the traceback below ring any bells? — It involves Py_InitializeFromConfig().

Stack frame #18 in of the traceback points to cpython code that was added 3 years ago but has a suspiciously looking comment:

https://github.com/python/cpython/blame/54289f85b2af1ecf046089ddf535dda1bdf6af24/Python/import.c#L521-L524

                /* Somebody already imported the module,
                   likely under a different name.
                   XXX this should really not happen. */
                Py_CLEAR(def->m_base.m_copy);

The same pybind11 code works with Python 3.6 through 3.11 and 3.12 alpha.1 (freshly confirmed via interactive testing). What change in 3.12, between alpha.1 and alpha.2 (also freshly confirmed), could have gotten us here?

For easy reference, here is:

1. the unit test code flagged by Catch2 (tb frame #27):

   pybind11/tests/test_embed/test_interpreter.cpp

   Line 184 in 9727dcd

   py::initialize_interpreter();

   
   2.the Python C API call from pybind11 that does not return before the crash (tb frame #25):

   pybind11/include/pybind11/embed.h

   Line 157 in 9727dcd

   status = Py_InitializeFromConfig(config);

$ PYTHONPATH=/usr/local/google/home/rwgk/forked/build_clang_py312a3/lib gdb /usr/local/google/home/rwgk/forked/build_clang_py312a3/bin/test_embed
GNU gdb (GDB) 10.0-gg5
...
(gdb) run
Starting program: /usr/local/google/home/rwgk/forked/build_clang_py312a3/bin/test_embed
free(): invalid pointer

Program received signal SIGABRT, Aborted.
__pthread_kill_implementation (threadid=<optimized out>, signo=6, no_tid=no_tid@entry=0) at ./nptl/pthread_kill.c:44
44      ./nptl/pthread_kill.c: No such file or directory.
(gdb) bt
#0  __pthread_kill_implementation (threadid=<optimized out>, signo=6, no_tid=no_tid@entry=0) at ./nptl/pthread_kill.c:44
#1  0x00007ffff78895df in __pthread_kill_internal (signo=<optimized out>, threadid=<optimized out>) at ./nptl/pthread_kill.c:89
#2  __GI___pthread_kill (threadid=<optimized out>, signo=<optimized out>) at ./nptl/pthread_kill.c:89
#3  0x00007ffff79f5e70 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#4  0x00007ffff7828469 in __GI_abort () at ./stdlib/abort.c:79
#5  0x00007ffff787d888 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7ffff79b66fb "%s\n")
    at ../sysdeps/posix/libc_fatal.c:155
#6  0x00007ffff78931ea in malloc_printerr (str=str@entry=0x7ffff79b41d0 "free(): invalid pointer") at ./malloc/malloc.c:5659
#7  0x00007ffff7894d2c in _int_free (av=<optimized out>, p=<optimized out>, have_lock=have_lock@entry=0) at ./malloc/malloc.c:4434
#8  0x00007ffff789789f in __GI___libc_free (mem=<optimized out>) at ./malloc/malloc.c:3386
#9  0x000055555595931f in PyMem_RawFree (ptr=<optimized out>) at Objects/obmalloc.c:1614
#10 _PyObject_Free (ctx=<optimized out>, p=<optimized out>) at Objects/obmalloc.c:1614
#11 _PyObject_Free (ctx=<optimized out>, p=<optimized out>) at Objects/obmalloc.c:1605
#12 0x000055555594081d in Py_DECREF (op=<optimized out>) at ./Include/object.h:561
#13 Py_XDECREF (op=<optimized out>) at ./Include/object.h:625
#14 free_keys_object (keys=0x555555f674e0) at Objects/dictobject.c:658
#15 0x0000555555942fbc in dictkeys_decref (dk=<optimized out>) at Objects/dictobject.c:319
#16 dict_dealloc (mp=0x7ffff76bb7c0) at Objects/dictobject.c:2328
#17 0x0000555555a083c5 in Py_DECREF (op=<optimized out>) at ./Include/object.h:561
#18 _PyImport_FixupExtensionObject (mod=mod@entry=0x7ffff6d33880, name=name@entry=0x555555e1c2a0 <const_str_sys>,
    filename=filename@entry=0x555555e1c2a0 <const_str_sys>, modules=modules@entry=0x7ffff6d0f440) at Python/import.c:524
#19 0x0000555555a0841f in _PyImport_FixupBuiltin (mod=mod@entry=0x7ffff6d33880, name=name@entry=0x555555b8625a "sys",
    modules=0x7ffff6d0f440) at Python/import.c:552
#20 0x0000555555a3661a in _PySys_Create (tstate=tstate@entry=0x555555e8e0e0 <_PyRuntime+450656>, sysmod_p=sysmod_p@entry=0x7fffffffbd98)
    at ./Python/sysmodule.c:3415
#21 0x0000555555a1d09a in pycore_interp_init (tstate=tstate@entry=0x555555e8e0e0 <_PyRuntime+450656>) at Python/pylifecycle.c:908
#22 0x0000555555a1f4aa in pyinit_config (runtime=0x555555e20080 <_PyRuntime>, config=0x7fffffffbeb0, tstate_p=0x7fffffffc0a8)
    at Python/pylifecycle.c:950
#23 pyinit_core (src_config=src_config@entry=0x7fffffffc258, tstate_p=tstate_p@entry=0x7fffffffc0a8, runtime=0x555555e20080 <_PyRuntime>)
    at Python/pylifecycle.c:1113
#24 0x0000555555a1faaf in Py_InitializeFromConfig (config=0x7fffffffc258) at Python/pylifecycle.c:1308
#25 0x0000555555812e98 in pybind11::initialize_interpreter (config=0x7fffffffc258, argc=0, argv=0x0, add_program_dir_to_path=true)
    at /usr/local/google/home/rwgk/forked/pybind11/include/pybind11/embed.h:157
#26 0x00005555558129c3 in pybind11::initialize_interpreter (init_signal_handlers=true, argc=0, argv=0x0, add_program_dir_to_path=true)
    at /usr/local/google/home/rwgk/forked/pybind11/include/pybind11/embed.h:205
#27 0x00005555557c8966 in C_A_T_C_H_T_E_S_T_10 () at /usr/local/google/home/rwgk/forked/pybind11/tests/test_embed/test_interpreter.cpp:184
#28 0x0000555555837413 in Catch::TestInvokerAsFunction::invoke (this=0x555555ea40a0)
    at /usr/local/google/home/rwgk/clone/Catch2/single_include/catch2/catch.hpp:14330
#29 0x0000555555831864 in Catch::TestCase::invoke (this=0x555555f1e808)
    at /usr/local/google/home/rwgk/clone/Catch2/single_include/catch2/catch.hpp:14169
#30 0x00005555558317b1 in Catch::RunContext::invokeActiveTestCase (this=0x7fffffffcf10)
    at /usr/local/google/home/rwgk/clone/Catch2/single_include/catch2/catch.hpp:13025
#31 0x000055555583043b in Catch::RunContext::runCurrentTest (this=0x7fffffffcf10, Python Exception <class 'gdb.error'> There is no member named _M_p.:
redirectedCout=, Python Exception <class 'gdb.error'> There is no member named _M_p.:
redirectedCerr=)
    at /usr/local/google/home/rwgk/clone/Catch2/single_include/catch2/catch.hpp:12998
#32 0x000055555582f8b4 in Catch::RunContext::runTest (this=0x7fffffffcf10, testCase=...)
    at /usr/local/google/home/rwgk/clone/Catch2/single_include/catch2/catch.hpp:12759
#33 0x00005555558340b3 in Catch::(anonymous namespace)::TestGroup::execute (this=0x7fffffffcf00)
    at /usr/local/google/home/rwgk/clone/Catch2/single_include/catch2/catch.hpp:13352
#34 0x00005555558336c2 in Catch::Session::runInternal (this=0x7fffffffd220)
    at /usr/local/google/home/rwgk/clone/Catch2/single_include/catch2/catch.hpp:13562
#35 0x0000555555833487 in Catch::Session::run (this=0x7fffffffd220)
    at /usr/local/google/home/rwgk/clone/Catch2/single_include/catch2/catch.hpp:13518
#36 0x000055555586580a in Catch::Session::run<char> (this=0x7fffffffd220, argc=1, argv=0x7fffffffd528)
    at /usr/local/google/home/rwgk/clone/Catch2/single_include/catch2/catch.hpp:13236
#37 0x0000555555849751 in main (argc=1, argv=0x7fffffffd528) at /usr/local/google/home/rwgk/forked/pybind11/tests/test_embed/catch.cpp:40
(gdb) quit

[henryiii]

I don't like the name change. Why change the name once a year when you can just have a simpler name that doesn't change every year? Also, we've used upstream for other upstream checks, like dev versions of PyPy.

I'll probably implement something like pypa/build#543 soon, and might even drop upstream.yml entirely.

[rwgk]

I don't like the name change. Why change the name once a year when you can just have a simpler name that doesn't change every year?

Because then it's very obvious what it does. "upstream" is very opaque, it took me a while to realize (back then).

Also, we've used upstream for other upstream checks, like dev versions of PyPy.

That's very confusing. Meaningful naming has a lot of value.

I'll probably implement something like pypa/build#543 soon, and might even drop upstream.yml entirely.

Sounds good, but could we please have this change until then? It's important for my work right now.

I could rename it back, if it gets replaced soon anyway it doesn't matter one way or another.

[henryiii]

If you do rename a file, it needs to be a separate commit then changes, so git can track the difference. As it is, it's just a brand new file. I would recommend renaming it back, we can review and merge based on this, then if you really want the name changed, maybe bring that up separately. Though I'm against the name change.

[rwgk]

If you do rename a file, it needs to be a separate commit then changes, so git can track the difference. As it is, it's just a brand new file. I would recommend renaming it back, we can review and merge based on this, then if you really want the name changed, maybe bring that up separately. Though I'm against the name change.

Done. I'm OK to keep the name if that's preferred. Just adding my thoughts for completeness:

There are several things that are highly specific to Python 3.12 in its current state, and are likely to change as 3.12 evolves.

By the time 3.13 comes around, likely there are a bunch of other things here and there they I learned or changed in the environment. Usually I start over ~fresh in such a situation, bringing in everything I know at that time. Doing that under a new name makes it easier to follow for others; it's immediately obvious that there was some sort of discontinuity, and why.

[henryiii]

Why remove VERBOSE just to add it to every line below?

[henryiii]

These generally will only work after RC1 is out, and by that time, we should be including the next release in our regular, required CI. I'd say just keep them out of this.

[henryiii]

This should get printed in our regular build? If not, we could add it (only if pybind11 is the master project).

[henryiii]

This is the only 3.12 specific part. I'd rather record logs of things happening outside the repo, certainly not in the CI files. I'd just make an issue and dump this in there. One issue per Python release would make much more sense than renaming a CI file every year. It's easier to update an issue than commit a change to the files.

[henryiii]

Why spend time deleting things if this is the last step?

[henryiii]

Why is this false? If you push a new commit, you probably don't care about the old one finishing, regardless of the fact this is a development build?

[henryiii]

I'm not sure what the benefit of not running the tests to make the not-required and not-default tests pass is.

[rwgk]

This PR was transferred to google/pybind11clif#30006.

Note: At the moment, the http://github.com/google/pywrapcc repo is effectively a branch of smart_holder, to support the PyCLIF-pybind11 integration work. Currently pywrapcc is not recommended for other uses.