Pip install failing on python 3.11 #58

wujiaqi · 2023-10-17T22:10:45Z

I tried doing the pip install ./ on a fresh raspberry pi OS install and noticed that this is throwing errors. I believe it's due to the API changes in python 3.11, the generated c code is accessing members no longer available https://docs.python.org/3/whatsnew/3.11.html#pyframeobject-3-11-hiding.

Installing collected packages: IT8951
  Running setup.py install for IT8951 ... error
  error: subprocess-exited-with-error
  
  × Running setup.py install for IT8951 did not run successfully.
  │ exit code: 1
  ╰─> [54 lines of output]
      running install
      /home/wujiaqi/git/webepd/eink2/lib/python3.11/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
        warnings.warn(
      running build
      running build_py
      creating build
      creating build/lib.linux-aarch64-cpython-311
      creating build/lib.linux-aarch64-cpython-311/IT8951
      copying IT8951/interface.py -> build/lib.linux-aarch64-cpython-311/IT8951
      copying IT8951/constants.py -> build/lib.linux-aarch64-cpython-311/IT8951
      copying IT8951/display.py -> build/lib.linux-aarch64-cpython-311/IT8951
      copying IT8951/__init__.py -> build/lib.linux-aarch64-cpython-311/IT8951
      running build_ext
      building 'IT8951.spi' extension
      creating build/temp.linux-aarch64-cpython-311
      creating build/temp.linux-aarch64-cpython-311/IT8951
      aarch64-linux-gnu-gcc -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -mbranch-protection=standard -g -fwrapv -O2 -fPIC -I/home/wujiaqi/git/webepd/eink2/include -I/usr/include/python3.11 -c IT8951/spi.c -o build/temp.linux-aarch64-cpython-311/IT8951/spi.o
      IT8951/spi.c: In function ‘__Pyx_TraceSetupAndCall’:
      IT8951/spi.c:22082:37: error: invalid use of incomplete typedef ‘PyFrameObject’ {aka ‘struct _frame’}
      22082 |         if (CYTHON_TRACE && (*frame)->f_trace == NULL) {
            |                                     ^~
      IT8951/spi.c:22084:21: error: invalid use of incomplete typedef ‘PyFrameObject’ {aka ‘struct _frame’}
      22084 |             (*frame)->f_trace = Py_None;
            |                     ^~
      IT8951/spi.c:438:62: error: invalid use of incomplete typedef ‘PyFrameObject’ {aka ‘struct _frame’}
        438 |   #define __Pyx_PyFrame_SetLineNumber(frame, lineno)  (frame)->f_lineno = (lineno)
            |                                                              ^~
      IT8951/spi.c:22091:5: note: in expansion of macro ‘__Pyx_PyFrame_SetLineNumber’
      22091 |     __Pyx_PyFrame_SetLineNumber(*frame, firstlineno);
            |     ^~~~~~~~~~~~~~~~~~~~~~~~~~~
      IT8951/spi.c: In function ‘__Pyx_PyBytes_Equals’:
      IT8951/spi.c:23312:13: warning: ‘ob_shash’ is deprecated [-Wdeprecated-declarations]
      23312 |             hash1 = ((PyBytesObject*)s1)->ob_shash;
            |             ^~~~~
      In file included from /usr/include/python3.11/bytesobject.h:62,
                       from /usr/include/python3.11/Python.h:50,
                       from IT8951/spi.c:6:
      /usr/include/python3.11/cpython/bytesobject.h:7:35: note: declared here
          7 |     Py_DEPRECATED(3.11) Py_hash_t ob_shash;
            |                                   ^~~~~~~~
      IT8951/spi.c:23313:13: warning: ‘ob_shash’ is deprecated [-Wdeprecated-declarations]
      23313 |             hash2 = ((PyBytesObject*)s2)->ob_shash;
            |             ^~~~~
      /usr/include/python3.11/cpython/bytesobject.h:7:35: note: declared here
          7 |     Py_DEPRECATED(3.11) Py_hash_t ob_shash;
            |                                   ^~~~~~~~
      IT8951/spi.c: In function ‘__Pyx_AddTraceback’:
      IT8951/spi.c:438:62: error: invalid use of incomplete typedef ‘PyFrameObject’ {aka ‘struct _frame’}
        438 |   #define __Pyx_PyFrame_SetLineNumber(frame, lineno)  (frame)->f_lineno = (lineno)
            |                                                              ^~
      IT8951/spi.c:24384:5: note: in expansion of macro ‘__Pyx_PyFrame_SetLineNumber’
      24384 |     __Pyx_PyFrame_SetLineNumber(py_frame, py_line);
            |     ^~~~~~~~~~~~~~~~~~~~~~~~~~~
      error: command '/usr/bin/aarch64-linux-gnu-gcc' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> IT8951

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.

Update:
When I run this with USE_CYTHON='USE_CYTHON' pip install ./ to recompile the c code it installs fine

The text was updated successfully, but these errors were encountered:

txoof · 2023-10-18T05:25:41Z

I was having similar problems yesterday. I'll give your solution a try today.

Have you tried the more recent versions of pillow with this module? There's a PR that pins pillow at 9.2.

txoof · 2023-10-18T18:36:11Z

@wujiaqi
I've been trying to get this to work for hours, but I'm running into something similar. I'm specifically trying to install IT8951 in a pipenv (virtual environment). I've gotten it to work once and verified that everything was working, but now I can't replicate it. I also found that there's no issues with the latest version of Pillow.

I can't figure out what I've done differently between this venv and the successful ones. I've started fresh with a blank venv and only installed cython~=0.29 into the venv.

I'm trying the following from within the venv:
$ USE_CYTHON='USE_CYTHON'; pip install ../IT8951/

If you have any ideas, they'd be greatly appreciated!

The output:

Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Processing /home/pi/src/IT8951
  Preparing metadata (setup.py) ... done
Building wheels for collected packages: IT8951
  Building wheel for IT8951 (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [54 lines of output]
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.linux-aarch64-cpython-311
      creating build/lib.linux-aarch64-cpython-311/IT8951
      copying IT8951/interface.py -> build/lib.linux-aarch64-cpython-311/IT8951
      copying IT8951/__init__.py -> build/lib.linux-aarch64-cpython-311/IT8951
      copying IT8951/constants.py -> build/lib.linux-aarch64-cpython-311/IT8951
      copying IT8951/display.py -> build/lib.linux-aarch64-cpython-311/IT8951
      warning: build_py: byte-compiling is disabled, skipping.

      running build_ext
      building 'IT8951.spi' extension
      creating build/temp.linux-aarch64-cpython-311
      creating build/temp.linux-aarch64-cpython-311/IT8951
      arm-linux-gnueabihf-gcc -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -I/home/pi/.local/share/virtualenvs/PaperPi-VBShxqF-/include -I/usr/include/python3.11 -c IT8951/spi.c -o build/temp.linux-aarch64-cpython-311/IT8951/spi.o
      IT8951/spi.c: In function ‘__Pyx_TraceSetupAndCall’:
      IT8951/spi.c:22082:37: error: invalid use of incomplete typedef ‘PyFrameObject’ {aka ‘struct _frame’}
      22082 |         if (CYTHON_TRACE && (*frame)->f_trace == NULL) {
            |                                     ^~
      IT8951/spi.c:22084:21: error: invalid use of incomplete typedef ‘PyFrameObject’ {aka ‘struct _frame’}
      22084 |             (*frame)->f_trace = Py_None;
            |                     ^~
      IT8951/spi.c:438:62: error: invalid use of incomplete typedef ‘PyFrameObject’ {aka ‘struct _frame’}
        438 |   #define __Pyx_PyFrame_SetLineNumber(frame, lineno)  (frame)->f_lineno = (lineno)
            |                                                              ^~
      IT8951/spi.c:22091:5: note: in expansion of macro ‘__Pyx_PyFrame_SetLineNumber’
      22091 |     __Pyx_PyFrame_SetLineNumber(*frame, firstlineno);
            |     ^~~~~~~~~~~~~~~~~~~~~~~~~~~
      IT8951/spi.c: In function ‘__Pyx_PyBytes_Equals’:
      IT8951/spi.c:23312:13: warning: ‘ob_shash’ is deprecated [-Wdeprecated-declarations]
      23312 |             hash1 = ((PyBytesObject*)s1)->ob_shash;
            |             ^~~~~
      In file included from /usr/include/python3.11/bytesobject.h:62,
                       from /usr/include/python3.11/Python.h:50,
                       from IT8951/spi.c:6:
      /usr/include/python3.11/cpython/bytesobject.h:7:35: note: declared here
          7 |     Py_DEPRECATED(3.11) Py_hash_t ob_shash;
            |                                   ^~~~~~~~
      IT8951/spi.c:23313:13: warning: ‘ob_shash’ is deprecated [-Wdeprecated-declarations]
      23313 |             hash2 = ((PyBytesObject*)s2)->ob_shash;
            |             ^~~~~
      /usr/include/python3.11/cpython/bytesobject.h:7:35: note: declared here
          7 |     Py_DEPRECATED(3.11) Py_hash_t ob_shash;
            |                                   ^~~~~~~~
      IT8951/spi.c: In function ‘__Pyx_AddTraceback’:
      IT8951/spi.c:438:62: error: invalid use of incomplete typedef ‘PyFrameObject’ {aka ‘struct _frame’}
        438 |   #define __Pyx_PyFrame_SetLineNumber(frame, lineno)  (frame)->f_lineno = (lineno)
            |                                                              ^~
      IT8951/spi.c:24384:5: note: in expansion of macro ‘__Pyx_PyFrame_SetLineNumber’
      24384 |     __Pyx_PyFrame_SetLineNumber(py_frame, py_line);
            |     ^~~~~~~~~~~~~~~~~~~~~~~~~~~
      error: command '/usr/bin/arm-linux-gnueabihf-gcc' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for IT8951
  Running setup.py clean for IT8951
Failed to build IT8951
ERROR: Could not build wheels for IT8951, which is required to install pyproject.toml-based projects
(PaperPi) 20:30:12

wujiaqi · 2023-10-18T18:43:46Z

@txoof I'm also using a venv. I have the following Cython installed, from pip list
Cython 3.0.4

And also, when you pass the USE_CYTHON var you have to inline it with the pip execution rather than execute it with a separate command...either that or export it, so omit the semicolon
From the root directory of the repo:
$ USE_CYTHON='USE_CYTHON' pip install ./

txoof · 2023-10-18T19:04:59Z

@wujiaqi
I finally had some success! Thanks!

I think the problem was cython 0.29.

For anyone else that stumbles onto this and needs some help with pipenv here are the basic steps:

Create pipenv: pipenv --python 3.11
Install cython 3: pipenv install cython
Install IT8951 from source: USE_CYTHON=1 pipenv install path/to/IT8951

Alternatively: Install IT8951 from source: USE_CYTHON=1 pipenv run pip install path/to/IT8951

wujiaqi · 2023-10-18T19:27:39Z

and I just realized that there's a PR out to fix this already #55

txoof · 2023-10-19T07:26:31Z

The PR is quite old and insists that pillow is pinned at v9. I tested it against pillow 10.1 and had no problems.

Thanks again for the help. This was driving me mad.

robweber · 2023-11-01T16:04:02Z

The fix of using USE_CYTHON=1 definitely works when installing directly but if you need to install as part of a project using a requirements.txt or setup.cfg file how would you go about getting that working? I've tried a few things and fell short. Ultimately an actual fix to update the code would be ideal. If the Pillow dependency from #55 is outdated the rest of that fix would probably be a good thing to get merged in.

txoof · 2023-11-01T17:04:33Z

@robweber
I'm solving this with an install script that detects the current python version and then downloads source and installs in a venv when necessary. Here's a script I cooked up for building a development environment reliably

robweber · 2023-11-01T17:48:30Z

Thanks for sharing that. Where I'm stumbling is when a downstream project is trying to include something as a dependency - like in a requirements or setup.py type file. Trying to keep it all "pure python" since the downstream project won't necessarily be cloning from source and be able to run a bash script.

Right now I have IT8951 as a dependency in omni-epd, which is in turn used by other projects. Would be nice if this just built properly. Wonder if anyone is maintaining a fork?

txoof · 2023-11-03T11:38:55Z

I'm stumbling similarly. My downstream project (PaperPi) requires this and I have to create a bunch of bash to manage the install.

GregDMeyer · 2023-11-04T02:01:44Z

Hi all,
I finally finished my PhD and moved to a new house and unpacked my moving boxes, and found my e-paper display to make some updates and test :-)
All the above issues should be fixed by 777d347. I have switched to pyproject.toml for the build, enabling a few nice things:

Cython is automatically installed just for the build environment, and always used to rebuild the C files instead of just when USE_CYTHON environment variable is set. (That env variable has no effect now). This should prevent this type of problem where upgrades to Python break the C files
Dependencies are automatically handled by pip (or whatever build system) now, so there shouldn't be any need for weird workarounds to get things built correctly.

The only difference in the build process is that now you should install on Raspberry Pi with pip install ./[rpi] which ensures that RPi.GPIO dependency is present.

Hopefully this weekend I will also upload the package to PyPI so you don't have to manually clone the git repository.

Thanks for everyone's patience while my life was a bit in chaos ;-) Please let me know if you have any issues with the new build setup!

wujiaqi mentioned this issue Oct 18, 2023

update cython generated code with latest cython for python 3.11 #59

Closed

robweber mentioned this issue Oct 30, 2023

add code to handle python venv TomWhitwell/SlowMovie#151

Merged

JonCellini mentioned this issue Oct 30, 2023

Install broken due to python changes in Raspbian Bookworm TomWhitwell/SlowMovie#150

Closed

GregDMeyer closed this as completed Nov 4, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pip install failing on python 3.11 #58

Pip install failing on python 3.11 #58

wujiaqi commented Oct 17, 2023 •

edited

Loading

txoof commented Oct 18, 2023

txoof commented Oct 18, 2023

wujiaqi commented Oct 18, 2023

txoof commented Oct 18, 2023 •

edited

Loading

wujiaqi commented Oct 18, 2023

txoof commented Oct 19, 2023

robweber commented Nov 1, 2023

txoof commented Nov 1, 2023

robweber commented Nov 1, 2023

txoof commented Nov 3, 2023

GregDMeyer commented Nov 4, 2023

Pip install failing on python 3.11 #58

Pip install failing on python 3.11 #58

Comments

wujiaqi commented Oct 17, 2023 • edited Loading

txoof commented Oct 18, 2023

txoof commented Oct 18, 2023

wujiaqi commented Oct 18, 2023

txoof commented Oct 18, 2023 • edited Loading

wujiaqi commented Oct 18, 2023

txoof commented Oct 19, 2023

robweber commented Nov 1, 2023

txoof commented Nov 1, 2023

robweber commented Nov 1, 2023

txoof commented Nov 3, 2023

GregDMeyer commented Nov 4, 2023

wujiaqi commented Oct 17, 2023 •

edited

Loading

txoof commented Oct 18, 2023 •

edited

Loading