[BUG] Build failure with Python 3.11

[opoplawski]

Welcome! You should write in your Bug Report:

OS version (https://www.google.com/search?channel=fs&q=check+os+version&ie=utf-8&oe=utf-8):
Fedora Rawhide

Python3 version (python3 -V -V):
Python 3.11.0b3 (main, Jun 24 2022, 00:00:00) [GCC 12.1.1 20220507 (Red Hat 12.1.1-1)]

Actual result (with the python stack trace if present):

+ /usr/bin/python3 setup.py build '--executable=/usr/bin/python3 -s'
running build
running build_py
creating build
creating build/lib.linux-x86_64-cpython-311
creating build/lib.linux-x86_64-cpython-311/frozendict
copying frozendict/__init__.py -> build/lib.linux-x86_64-cpython-311/frozendict
copying frozendict/core.py -> build/lib.linux-x86_64-cpython-311/frozendict
copying frozendict/VERSION -> build/lib.linux-x86_64-cpython-311/frozendict
copying frozendict/py.typed -> build/lib.linux-x86_64-cpython-311/frozendict
running build_ext
building 'frozendict._frozendict' extension
creating build/temp.linux-x86_64-cpython-311
creating build/temp.linux-x86_64-cpython-311/frozendict
creating build/temp.linux-x86_64-cpython-311/frozendict/src
creating build/temp.linux-x86_64-cpython-311/frozendict/src/3_11
gcc -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fPIC -I/home/orion/fedora/python-frozendict/frozendict-2.3.2/frozendict/src/3_11/Include -I/home/orion/fedora/python-frozendict/frozendict-2.3.2/frozendict/src/3_11/cpython_src/Objects -I/home/orion/fedora/python-frozendict/frozendict-2.3.2/frozendict/src/3_11/cpython_src/Objects/stringlib -I/home/orion/fedora/python-frozendict/frozendict-2.3.2/frozendict/src/3_11/cpython_src/Objects/clinic -I/home/orion/fedora/python-frozendict/frozendict-2.3.2/frozendict/src/3_11/cpython_src -I/usr/include/python3.11 -c frozendict/src/3_11/frozendictobject.c -o build/temp.linux-x86_64-cpython-311/frozendict/src/3_11/frozendictobject.o -DPY_SSIZE_T_CLEAN
cc1: fatal error: frozendict/src/3_11/frozendictobject.c: No such file or directory
compilation terminated.

[opoplawski]

If I copy 3_10 to 3_11:

In file included from frozendict/src/3_11/frozendictobject.c:6:
/home/orion/fedora/python-frozendict/frozendict-2.3.2/frozendict/src/3_11/cpython_src/Objects/dictobject.c: In function ‘_PyDict_CheckConsistency’:
/home/orion/fedora/python-frozendict/frozendict-2.3.2/frozendict/src/3_11/cpython_src/Objects/dictobject.c:491:36: error: invalid use of undefined type ‘struct _dictvalues’
  491 |                 CHECK(mp->ma_values[i] != NULL);
      |                                    ^
/home/orion/fedora/python-frozendict/frozendict-2.3.2/frozendict/src/3_11/cpython_src/Objects/dictobject.c:434:16: note: in definition of macro ‘CHECK’
  434 |     do { if (!(expr)) { _PyObject_ASSERT_FAILED_MSG(op, Py_STRINGIFY(expr)); } } while (0)
      |                ^~~~
/home/orion/fedora/python-frozendict/frozendict-2.3.2/frozendict/src/3_11/cpython_src/Objects/dictobject.c:491:23: error: invalid use of incomplete typedef ‘PyDictValues’ {aka ‘struct _dictvalues’}
  491 |                 CHECK(mp->ma_values[i] != NULL);
      |                       ^~
/home/orion/fedora/python-frozendict/frozendict-2.3.2/frozendict/src/3_11/cpython_src/Objects/dictobject.c:434:16: note: in definition of macro ‘CHECK’
  434 |     do { if (!(expr)) { _PyObject_ASSERT_FAILED_MSG(op, Py_STRINGIFY(expr)); } } while (0)
      |                ^~~~
/home/orion/fedora/python-frozendict/frozendict-2.3.2/frozendict/src/3_11/cpython_src/Objects/dictobject.c: In function ‘_d_PyDict_Next’:
/home/orion/fedora/python-frozendict/frozendict-2.3.2/frozendict/src/3_11/cpython_src/Objects/dictobject.c:799:30: error: invalid use of undefined type ‘struct _dictvalues’
  799 |         value = mp->ma_values[i];
      |                              ^
/home/orion/fedora/python-frozendict/frozendict-2.3.2/frozendict/src/3_11/cpython_src/Objects/dictobject.c:799:17: error: invalid use of incomplete typedef ‘PyDictValues’ {aka ‘struct _dictvalues’}
  799 |         value = mp->ma_values[i];
      |                 ^~
/home/orion/fedora/python-frozendict/frozendict-2.3.2/frozendict/src/3_11/cpython_src/Objects/dictobject.c: In function ‘dict_dealloc’:
/home/orion/fedora/python-frozendict/frozendict-2.3.2/frozendict/src/3_11/cpython_src/Objects/dictobject.c:830:25: warning: initialization of ‘PyObject **’ {aka ‘struct _object **’} from incompatible pointer type ‘PyDictValues *’ {aka ‘struct _dictvalues *’} [-Wincompatible-pointer-types]
  830 |     PyObject **values = mp->ma_values;
      |                         ^~
In file included from /usr/include/python3.11/Python.h:45,
                 from frozendict/src/3_11/frozendictobject.c:1:
/home/orion/fedora/python-frozendict/frozendict-2.3.2/frozendict/src/3_11/cpython_src/Objects/dictobject.c: In function ‘dict_traverse’:
/home/orion/fedora/python-frozendict/frozendict-2.3.2/frozendict/src/3_11/cpython_src/Objects/dictobject.c:1090:39: error: invalid use of undefined type ‘struct _dictvalues’
 1090 |                 Py_VISIT(mp->ma_values[i]);
      |                                       ^
/usr/include/python3.11/objimpl.h:199:13: note: in definition of macro ‘Py_VISIT’
  199 |         if (op) {                                                       \
      |             ^~
/home/orion/fedora/python-frozendict/frozendict-2.3.2/frozendict/src/3_11/cpython_src/Objects/dictobject.c:1090:26: error: invalid use of incomplete typedef ‘PyDictValues’ {aka ‘struct _dictvalues’}
 1090 |                 Py_VISIT(mp->ma_values[i]);
      |                          ^~
/usr/include/python3.11/objimpl.h:199:13: note: in definition of macro ‘Py_VISIT’
  199 |         if (op) {                                                       \
      |             ^~
In file included from /usr/include/python3.11/Python.h:38:
/home/orion/fedora/python-frozendict/frozendict-2.3.2/frozendict/src/3_11/cpython_src/Objects/dictobject.c:1090:39: error: invalid use of undefined type ‘struct _dictvalues’
 1090 |                 Py_VISIT(mp->ma_values[i]);
      |                                       ^
/usr/include/python3.11/pyport.h:69:40: note: in definition of macro ‘_Py_CAST’
   69 | #  define _Py_CAST(type, expr) ((type)(expr))
      |                                        ^~~~
/usr/include/python3.11/objimpl.h:200:30: note: in expansion of macro ‘_PyObject_CAST’
  200 |             int vret = visit(_PyObject_CAST(op), arg);                  \
      |                              ^~~~~~~~~~~~~~
/home/orion/fedora/python-frozendict/frozendict-2.3.2/frozendict/src/3_11/cpython_src/Objects/dictobject.c:1090:17: note: in expansion of macro ‘Py_VISIT’
 1090 |                 Py_VISIT(mp->ma_values[i]);
      |                 ^~~~~~~~
/usr/include/python3.11/pyport.h:69:39: error: invalid use of incomplete typedef ‘PyDictValues’ {aka ‘struct _dictvalues’}
   69 | #  define _Py_CAST(type, expr) ((type)(expr))
      |                                       ^
/usr/include/python3.11/object.h:107:28: note: in expansion of macro ‘_Py_CAST’
  107 | #define _PyObject_CAST(op) _Py_CAST(PyObject*, (op))
      |                            ^~~~~~~~
/usr/include/python3.11/objimpl.h:200:30: note: in expansion of macro ‘_PyObject_CAST’
  200 |             int vret = visit(_PyObject_CAST(op), arg);                  \
      |                              ^~~~~~~~~~~~~~
/home/orion/fedora/python-frozendict/frozendict-2.3.2/frozendict/src/3_11/cpython_src/Objects/dictobject.c:1090:17: note: in expansion of macro ‘Py_VISIT’
 1090 |                 Py_VISIT(mp->ma_values[i]);
      |                 ^~~~~~~~
frozendict/src/3_11/frozendictobject.c: In function ‘frozendict_merge’:
frozendict/src/3_11/frozendictobject.c:447:41: error: invalid use of undefined type ‘struct _dictvalues’
  447 |                 value = other->ma_values[i];
      |                                         ^
frozendict/src/3_11/frozendictobject.c:447:25: error: invalid use of incomplete typedef ‘PyDictValues’ {aka ‘struct _dictvalues’}
  447 |                 value = other->ma_values[i];
      |                         ^~~~~
/home/orion/fedora/python-frozendict/frozendict-2.3.2/frozendict/src/3_11/cpython_src/Objects/dictobject.c: At top level:
/home/orion/fedora/python-frozendict/frozendict-2.3.2/frozendict/src/3_11/cpython_src/Objects/dictobject.c:431:1: warning: ‘_PyDict_CheckConsistency’ defined but not used [-Wunused-function]
  431 | _PyDict_CheckConsistency(PyObject *op, int check_content)
      | ^~~~~~~~~~~~~~~~~~~~~~~~

[Marco-Sulla]

Hello,

beta, alpha and non-release versions of Python and PyPy are not supported. You can just use the pyre python package.

[opoplawski]

FWIW - I'm am packaging frozendict for Fedora and it (rawhide specifically) is moving to 3.11 to be ready for when it is released. So the pyre python package is not relevant here.

[Marco-Sulla]

Well, I'm quite sure this is exciting for much people.Anyway, two considerations.

1. Rawhide, for what I know, is the current dev branch of Fedora. I think it's normal for a dev branch to use beta packages, in particular for Fedora
2. Python 3.11 seems to have changed the code of dict a lot internally. Inada and the other core devs did a great job. Since the C Extension of frozendict uses a bunch of heavily modified dict source code, the adaptation is not trivial this time. Furthermore the dict code of Python 3.11, IMHO, it's quite better, and that means that probably I'll rebase the code of all the other versions to 3.11. This is to make it easier modification of the C code for all the supported version (currently, the base code is the 3.10 one).

Shortly, the effort of making a frozendict C Extension for Python 3.11 seems to be harder than the previous versions, and I don't want to risk to make the extension and see the CPython 3.11 dict code change again under my nose.

Is it too much for you waiting 5-6 months for the official stable release?

[opoplawski]

It appears that would be fine - Fedora is actually still on frozendict 1.2 which seems to build fine with Python 3.11.

[Marco-Sulla]

So I don't see why you shouldn't use the py version of frozendict 2 in the meanwhile. Frozendict 1.2 is pure python too, requires patches to work because is unmantained, it's more slow, less immutable and with a naif hash algorithm.

The py version of frozendict 2 can be packaged using

python3 setup.py py bdist_wheel

Don't know how you can specify it in you rpm.