[BUG] Failures with current CPython main branch

[tacaswell]

Describe the bug

Trying to build numpy with current CPython main branch (2022-04-10) fails (see traceback below). I have bisected this back to:

$ git bisect bad
884eba3c76916889fd6bff3b37b8552bfb4f9566 is the first bad commit
commit 884eba3c76916889fd6bff3b37b8552bfb4f9566
Author: Serhiy Storchaka <storchaka@gmail.com>
Date:   Wed Apr 6 20:00:14 2022 +0300

    bpo-26579: Add object.__getstate__(). (GH-2821)
    
    Copying and pickling instances of subclasses of builtin types
    bytearray, set, frozenset, collections.OrderedDict, collections.deque,
    weakref.WeakSet, and datetime.tzinfo now copies and pickles instance attributes
    implemented as slots.

Which came in via python/cpython#2821 .

full traceback

Details

      Traceback (most recent call last):
        File "<frozen runpy>", line 198, in _run_module_as_main
        File "<frozen runpy>", line 88, in _run_code
        File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/cython.py", line 17, in <module>
          main(command_line = 1)
          ^^^^^^^^^^^^^^^^^^^^^^
        File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/Cython/Compiler/Main.py", line 731, in main
          result = compile(sources, options)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/Cython/Compiler/Main.py", line 629, in compile
          return compile_multiple(source, options)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/Cython/Compiler/Main.py", line 606, in compile_multiple
          result = run_pipeline(source, options, context=context)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/Cython/Compiler/Main.py", line 504, in run_pipeline
          err, enddata = Pipeline.run_pipeline(pipeline, source)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/Cython/Compiler/Pipeline.py", line 394, in run_pipeline
          data = run(phase, data)
                 ^^^^^^^^^^^^^^^^
        File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/Cython/Compiler/Pipeline.py", line 371, in run
          return phase(data)
                 ^^^^^^^^^^^
        File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/Cython/Compiler/Pipeline.py", line 52, in generate_pyx_code_stage
          module_node.process_implementation(options, result)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/Cython/Compiler/ModuleNode.py", line 206, in process_implementation
          self.generate_c_code(env, options, result)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/Cython/Compiler/ModuleNode.py", line 501, in generate_c_code
          self.body.generate_function_definitions(env, code)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/Cython/Compiler/Nodes.py", line 403, in generate_function_definitions
          stat.generate_function_definitions(env, code)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/Cython/Compiler/Nodes.py", line 403, in generate_function_definitions
          stat.generate_function_definitions(env, code)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/Cython/Compiler/Nodes.py", line 2107, in generate_function_definitions
          self.generate_function_body(env, code)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/Cython/Compiler/Nodes.py", line 1867, in generate_function_body
          self.body.generate_execution_code(code)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/Cython/Compiler/Nodes.py", line 409, in generate_execution_code
          stat.generate_execution_code(code)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/Cython/Compiler/Nodes.py", line 7689, in generate_execution_code
          self.body.generate_execution_code(code)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/Cython/Compiler/Nodes.py", line 8185, in generate_execution_code
          fresh_finally_clause().generate_execution_code(code)
          ^^^^^^^^^^^^^^^^^^^^^^
        File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.11/site-packages/Cython/Compiler/Nodes.py", line 8173, in fresh_finally_clause
          node_copy = copy.deepcopy(node)
                      ^^^^^^^^^^^^^^^^^^^
        File "/home/tcaswell/.pybuild/bleeding/lib/python3.11/copy.py", line 172, in deepcopy
          y = _reconstruct(x, memo, *rv)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/tcaswell/.pybuild/bleeding/lib/python3.11/copy.py", line 271, in _reconstruct
          state = deepcopy(state, memo)
                  ^^^^^^^^^^^^^^^^^^^^^
        File "/home/tcaswell/.pybuild/bleeding/lib/python3.11/copy.py", line 146, in deepcopy
          y = copier(x, memo)
              ^^^^^^^^^^^^^^^
        File "/home/tcaswell/.pybuild/bleeding/lib/python3.11/copy.py", line 231, in _deepcopy_dict
          y[deepcopy(key, memo)] = deepcopy(value, memo)
                                   ^^^^^^^^^^^^^^^^^^^^^
        File "/home/tcaswell/.pybuild/bleeding/lib/python3.11/copy.py", line 172, in deepcopy
          y = _reconstruct(x, memo, *rv)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/tcaswell/.pybuild/bleeding/lib/python3.11/copy.py", line 271, in _reconstruct
          state = deepcopy(state, memo)
                  ^^^^^^^^^^^^^^^^^^^^^
        File "/home/tcaswell/.pybuild/bleeding/lib/python3.11/copy.py", line 146, in deepcopy
          y = copier(x, memo)
              ^^^^^^^^^^^^^^^
        File "/home/tcaswell/.pybuild/bleeding/lib/python3.11/copy.py", line 231, in _deepcopy_dict
          y[deepcopy(key, memo)] = deepcopy(value, memo)
                                   ^^^^^^^^^^^^^^^^^^^^^
        File "/home/tcaswell/.pybuild/bleeding/lib/python3.11/copy.py", line 172, in deepcopy
          y = _reconstruct(x, memo, *rv)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/tcaswell/.pybuild/bleeding/lib/python3.11/copy.py", line 271, in _reconstruct
          state = deepcopy(state, memo)
                  ^^^^^^^^^^^^^^^^^^^^^
        File "/home/tcaswell/.pybuild/bleeding/lib/python3.11/copy.py", line 146, in deepcopy
          y = copier(x, memo)
              ^^^^^^^^^^^^^^^
        File "/home/tcaswell/.pybuild/bleeding/lib/python3.11/copy.py", line 231, in _deepcopy_dict
          y[deepcopy(key, memo)] = deepcopy(value, memo)
                                   ^^^^^^^^^^^^^^^^^^^^^
        File "/home/tcaswell/.pybuild/bleeding/lib/python3.11/copy.py", line 172, in deepcopy
          y = _reconstruct(x, memo, *rv)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/tcaswell/.pybuild/bleeding/lib/python3.11/copy.py", line 271, in _reconstruct
          state = deepcopy(state, memo)
                  ^^^^^^^^^^^^^^^^^^^^^
        File "/home/tcaswell/.pybuild/bleeding/lib/python3.11/copy.py", line 146, in deepcopy
          y = copier(x, memo)
              ^^^^^^^^^^^^^^^
        File "/home/tcaswell/.pybuild/bleeding/lib/python3.11/copy.py", line 231, in _deepcopy_dict
          y[deepcopy(key, memo)] = deepcopy(value, memo)
                                   ^^^^^^^^^^^^^^^^^^^^^
        File "/home/tcaswell/.pybuild/bleeding/lib/python3.11/copy.py", line 172, in deepcopy
          y = _reconstruct(x, memo, *rv)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/tcaswell/.pybuild/bleeding/lib/python3.11/copy.py", line 288, in _reconstruct
          item = deepcopy(item, memo)
                 ^^^^^^^^^^^^^^^^^^^^
        File "/home/tcaswell/.pybuild/bleeding/lib/python3.11/copy.py", line 161, in deepcopy
          rv = reductor(4)
               ^^^^^^^^^^^
      TypeError: cannot pickle 'Argument' object

The error it is failing with is

TypeError: cannot pickle 'Argument' object

If you get into a debugger the object that is failing to pickle is

> /home/tcaswell/.pybuild/bleeding/lib/python3.11/copy.py(161)deepcopy()
-> rv = reductor(4)
(Pdb) p x
Argument(entry=Entry(<7f2d8b1199d0>, name=lock, type=Python object))
(Pdb) p type(x)
<class 'Cython.Compiler.FlowControl.Argument'>
(Pdb) 

To Reproduce

Try to install numpy. I have not tried to isolate the part of the file that triggers the problem though.

Expected behavior
A clear and concise description of what you expected to happen.

compile

Environment (please complete the following information):

• OS: [e.g. Linux, Windows, macOS] Linux
• Python version [e.g. 3.8.4] main branch (post python/cpython@884eba3)
• Cython version [e.g. 0.29.18] master branch

Additional context

None.

[da-woods]

I'd guess (but I haven't tested) that it'll work if you install Cython without compiling it.

pip install Cython --install-option="--no-cython-compile"

It looks like Argument is a regular class (but compiled in Cython) that inherits from a cdef class. auto_pickle is true for FlowControl.py (so the base cdef class should be pickleable). That's the extent of what I know right now...

[da-woods]

So my best guess at what I think is going on (all completely untested...).

When Cython generates code to pickle a cdef class it does so tentatively: it generates code for __reduce_cython__ (which obviously isn't a "real" special method), and then at runtime when the class is initialized it looks for __getstate__, __reduce__, __reduce_ex__ etc and only overrides with its own pickling code if they don't exist:

cython/Cython/Utility/ExtensionTypes.c

Lines 409 to 430 in 12628a6

	static int __Pyx_setup_reduce(PyObject* type_obj) {
	int ret = 0;
	PyObject *object_reduce = NULL;
	PyObject *object_reduce_ex = NULL;
	PyObject *reduce = NULL;
	PyObject *reduce_ex = NULL;
	PyObject *reduce_cython = NULL;
	PyObject *setstate = NULL;
	PyObject *setstate_cython = NULL;
	#if CYTHON_USE_PYTYPE_LOOKUP
	if (_PyType_Lookup((PyTypeObject*)type_obj, PYIDENT("__getstate__"))) goto __PYX_GOOD;
	#else
	if (PyObject_HasAttr(type_obj, PYIDENT("__getstate__"))) goto __PYX_GOOD;
	#endif
	#if CYTHON_USE_PYTYPE_LOOKUP
	object_reduce_ex = _PyType_Lookup(&PyBaseObject_Type, PYIDENT("__reduce_ex__")); if (!object_reduce_ex) goto __PYX_BAD;
	#else
	object_reduce_ex = __Pyx_PyObject_GetAttrStr((PyObject*)&PyBaseObject_Type, PYIDENT("__reduce_ex__")); if (!object_reduce_ex) goto __PYX_BAD;
	#endif

I think Cython's seeing the new object.__getstate__ and deciding not to override it. It probably just needs that check to be modified slightly to something like if not hasattr("__getstate__") or not self.__getstate__ == object.__getstate__.

That's probably a fairly simple fix I think.

[tacaswell]

I'd guess (but I haven't tested) that it'll work if you install Cython without compiling it.

I can confirm that gets numpy building again.