Pip install is failing to build due to av/logging.pyx error #1140

nicholas-presien · 2023-07-18T06:54:00Z

Overview

I'm currently using the stable version of PyAV (v9.0.2) in a docker container, and building in a docker image for a while with no issue. Recently, my docker image failed due to build errors in pyav when installing pyav using pip.

I'm building on a TX2NX dev kit, running Jetpack 4.6 with an arm64 compiled version of FFMPEG 4.3.2(https://launchpad.net/~jonathonf/+archive/ubuntu/ffmpeg-4), on python v3.6.9

Expected behavior

It should build without error

Actual behavior

It's now building with error, when nothing in the environment has changed due to it being in a docker container.

Build report:

python3 setup.py build_ext --inplace                                                                                                                                                                           
Compiling av/logging.pyx because it changed.                                                                                                                                                                                                  
[1/1] Cythonizing av/logging.pyx                                                                                                                                                                                                              
                                                                                                                                                                                                                                              
Error compiling Cython file:                                                                                                                                                                                                                  
------------------------------------------------------------                                                                                                                                                                                  
...                                                                                                                                                                                                                                           
cdef const char *log_context_name(void *ptr) nogil:                                                                                                                                                                                           
    cdef log_context *obj = <log_context*>ptr                                                                                                                                                                                                 
    return obj.name                                                                                                                                                                                                                           
                                                                                                                                                                                                                                              
cdef lib.AVClass log_class                                                                                                                                                                                                                    
log_class.item_name = log_context_name                                                                                                                                                                                                        
                      ^                                                                                                                                                                                                                       
------------------------------------------------------------                                                                                                                                                                                  
                                                                                                                                                                                                                                              
av/logging.pyx:216:22: Cannot assign type 'const char *(void *) except? NULL nogil' to 'const char *(*)(void *) noexcept nogil'                                                                                                               
                                                                                                                                                                                                                                              
Error compiling Cython file:                                                                                                                                                                                                                  
------------------------------------------------------------                                                                                                                                                                                  
...                                                                                                                                                                                                                                           
                                                                                                                                                                                                                                              
# Start the magic!                                                                                                                                                                                                                            
# We allow the user to fully disable the logging system as it will not play                                                                                                                                                                   
# nicely with subinterpreters due to FFmpeg-created threads.                                                                                                                                                                                  
if os.environ.get('PYAV_LOGGING') != 'off':                                                                                                                                                                                                   
    lib.av_log_set_callback(log_callback)                                                                                                                                                                                                     
                            ^                                                                                                                                                                                                                 
------------------------------------------------------------                                                                                                                                                                                  
                                                                                                                                                                                                                                              
av/logging.pyx:351:28: Cannot assign type 'void (void *, int, const char *, va_list) except * nogil' to 'av_log_callback'                                                                                                                     
Traceback (most recent call last):                                                                                                                                                                                                            
  File "setup.py", line 163, in <module>                                                                                                                                                                                                      
    include_path=["include"],                                                                                                                                                                                                                 
  File "/home/presien/.local/lib/python3.6/site-packages/Cython/Build/Dependencies.py", line 1134, in cythonize                                                                                                                               
    cythonize_one(*args)                                                                                                                                                                                                                      
  File "/home/presien/.local/lib/python3.6/site-packages/Cython/Build/Dependencies.py", line 1301, in cythonize_one                                                                                                                           
    raise CompileError(None, pyx_file)                                                                                                                                                                                                        
Cython.Compiler.Errors.CompileError: av/logging.pyx

Investigation

I've tried to run the build natively on the host using the python3 setup.py method, at branch v9.0.2, and I'm getting the same issue

Reproduction

This might be hard to reproduce since its ran on a Nvidia TX2NX Hardware

Jetpack OS version 4.6.1
Install ffmpeg 4.3.2 from https://launchpad.net/~jonathonf/+archive/ubuntu/ffmpeg-4, along with all of the dev libs shown below

libavformat-dev 
libavcodec-dev 
libavdevice-dev 
libavutil-dev 
libavfilter-dev 
libswscale-dev 
libswresample-dev

clone pyav
checkout tag v9.0.2
follow installation guide in readme

Versions

OS: Nvidia Jetpack 4.6.1
PyAV runtime:

n/a

PyAV build:

python3 setup.py config --verbose
Compiling av/logging.pyx because it changed.
[1/1] Cythonizing av/logging.pyx

Error compiling Cython file:
------------------------------------------------------------
...
cdef const char *log_context_name(void *ptr) nogil:
    cdef log_context *obj = <log_context*>ptr
    return obj.name

cdef lib.AVClass log_class
log_class.item_name = log_context_name
                      ^
------------------------------------------------------------

av/logging.pyx:216:22: Cannot assign type 'const char *(void *) except? NULL nogil' to 'const char *(*)(void *) noexcept nogil'

Error compiling Cython file:
------------------------------------------------------------
...

# Start the magic!
# We allow the user to fully disable the logging system as it will not play
# nicely with subinterpreters due to FFmpeg-created threads.
if os.environ.get('PYAV_LOGGING') != 'off':
    lib.av_log_set_callback(log_callback)
                            ^
------------------------------------------------------------

av/logging.pyx:351:28: Cannot assign type 'void (void *, int, const char *, va_list) except * nogil' to 'av_log_callback'
Traceback (most recent call last):
  File "setup.py", line 163, in <module>
    include_path=["include"],
  File "/home/presien/.local/lib/python3.6/site-packages/Cython/Build/Dependencies.py", line 1134, in cythonize
    cythonize_one(*args)
  File "/home/presien/.local/lib/python3.6/site-packages/Cython/Build/Dependencies.py", line 1301, in cythonize_one
    raise CompileError(None, pyx_file)
Cython.Compiler.Errors.CompileError: av/logging.pyx

FFmpeg:

ffmpeg -version
ffmpeg version 4.3.2-0york0~18.04 Copyright (c) 2000-2021 the FFmpeg developers
built with gcc 7 (Ubuntu/Linaro 7.5.0-3ubuntu1~18.04)
configuration: --prefix=/usr --extra-version='0york0~18.04' --toolchain=hardened --libdir=/usr/lib/aarch64-linux-gnu --incdir=/usr/include/aarch64-linux-gnu --arch=arm64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librabbitmq --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-libzimg --enable-pocketsphinx --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
libavutil      56. 51.100 / 56. 51.100
libavcodec     58. 91.100 / 58. 91.100
libavformat    58. 45.100 / 58. 45.100
libavdevice    58. 10.100 / 58. 10.100
libavfilter     7. 85.100 /  7. 85.100
libavresample   4.  0.  0 /  4.  0.  0
libswscale      5.  7.100 /  5.  7.100
libswresample   3.  7.100 /  3.  7.100
libpostproc    55.  7.100 / 55.  7.100

Research

I have done the following:

The text was updated successfully, but these errors were encountered:

christianglodt · 2023-07-18T07:21:25Z

I get the same issue on x86_64 ubuntu 23.04, and in a docker container based on python:3.11-slim (which is based on debian:bookworm-slim). I can't use a binary wheel because the bundled ffmpeg is compiled without Pulseaudio support, which I need.

samdbmg · 2023-07-18T08:22:11Z

Same here. I'm wondering if this is due to the release of Cython 3.0.0 - my failing CI shows it's being picked up as part of the PyAV build, and it worked yesterday morning 😄

dhopsonlb · 2023-07-18T16:48:23Z

I can confirm I'm having the same issue. Likewise I would just use the binaries, but there's neither alsa nor pulse support built into the binary distribution. I tried fixing the errors manually and got Cython to build the Cython source to C, and then it explodes in the generated C source like so:

src/av/video/format.c:3307:40: error: assignment of read-only member ‘__pyx_genexpr_arg_0’

pal-sanich · 2023-07-19T10:00:47Z

Same here on python:alpine3.16, python:alpine3.17, python:alpine3.18,
Tried:
wheel-0.38.4, wheel-0.37.1,
cython-0.29.24, cython-0.29.36

araczkowski · 2023-07-19T15:58:09Z

I have the same problem:

~/PyAV $ python3 setup.py build_ext --inplace                                                                                                                                                                           
Compiling av/frame.pyx because it changed.
[1/1] Cythonizing av/frame.pyx
Compiling av/buffer.pyx because it changed.
[1/1] Cythonizing av/buffer.pyx
Compiling av/dictionary.pyx because it changed.
[1/1] Cythonizing av/dictionary.pyx
Compiling av/format.pyx because it changed.
[1/1] Cythonizing av/format.pyx
Compiling av/packet.pyx because it changed.
[1/1] Cythonizing av/packet.pyx
Compiling av/descriptor.pyx because it changed.
[1/1] Cythonizing av/descriptor.pyx
Compiling av/stream.pyx because it changed.
[1/1] Cythonizing av/stream.pyx
Compiling av/plane.pyx because it changed.
[1/1] Cythonizing av/plane.pyx
Compiling av/option.pyx because it changed.
[1/1] Cythonizing av/option.pyx
Compiling av/error.pyx because it changed.
[1/1] Cythonizing av/error.pyx
Compiling av/_core.pyx because it changed.
[1/1] Cythonizing av/_core.pyx
Compiling av/logging.pyx because it changed.
[1/1] Cythonizing av/logging.pyx

Error compiling Cython file:
------------------------------------------------------------
...
cdef const char *log_context_name(void *ptr) nogil:
    cdef log_context *obj = <log_context*>ptr
    return obj.name

cdef lib.AVClass log_class
log_class.item_name = log_context_name
                      ^
------------------------------------------------------------

av/logging.pyx:216:22: Cannot assign type 'const char *(void *) except? NULL nogil' to 'const char *(*)(void *) noexcept nogil'

Error compiling Cython file:
------------------------------------------------------------
...

# Start the magic!
# We allow the user to fully disable the logging system as it will not play
# nicely with subinterpreters due to FFmpeg-created threads.
if os.environ.get('PYAV_LOGGING') != 'off':
    lib.av_log_set_callback(log_callback)
                            ^
------------------------------------------------------------

av/logging.pyx:351:28: Cannot assign type 'void (void *, int, const char *, va_list) except * nogil' to 'av_log_callback'
Traceback (most recent call last):
  File "/data/data/com.termux/files/home/PyAV/setup.py", line 157, in <module>
    ext_modules += cythonize(
                   ^^^^^^^^^^
  File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/Cython/Build/Dependencies.py", line 1134, in cythonize
    cythonize_one(*args)
  File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/Cython/Build/Dependencies.py", line 1301, in cythonize_one
    raise CompileError(None, pyx_file)
Cython.Compiler.Errors.CompileError: av/logging.pyx

abhinavsingh · 2023-07-19T16:41:07Z

We tried pinning to Cython==0.29.36 but that doesn't seem to help either. Is there a recommended workaround here?

dhopsonlb · 2023-07-19T16:49:19Z

Cython seems to misgenerate the C, so even if you edit the .pyx files with errors and change the complaining function prototypes to "noexcept nogil" like I did, that doesn't fully fix the issue. I had to patch the machine-generated C code myself, wait until Cython got past that file in the build process, and quickly copy the patched version in. I ran pip install . from the PyAV git repo.

What's interesting is it seems every Cython version I tried, as far back as 2019, misgenerated the C.

Something unusual is broken.

cdce8p · 2023-07-19T17:14:26Z

It's an issue with cython==3.0 which was released a few days ago. Since PyAV doesn't use an upper limit in it's build dependencies, it's always used.

PyAV/pyproject.toml

Line 2 in 6982d81

requires = ["setuptools", "wheel", "cython"]

A workaround is to add a constraint. However, flags are not passed when creating the isolated build environments. For that it's necessary to use an environment variable. Something like this works

echo "cython<3.0" >> c.txt
PIP_CONSTRAINT=c.txt pip install av==10.0.0

nicholas-presien · 2023-07-20T00:50:47Z

It's an issue with cython==3.0 which was released a few days ago. Since PyAV doesn't use an upper limit in it's build dependencies, it's always used.

PyAV/pyproject.toml

Line 2 in 6982d81

requires = ["setuptools", "wheel", "cython"]

A workaround is to add a constraint. However, flags are not passed when creating the isolated build environments. For that it's necessary to use an environment variable. Something like this works
echo "cython<3.0" >> c.txt
PIP_CONSTRAINT=c.txt pip install av==10.0.0

Can confirm this fixes the build issue. Thanks for the help @cdce8p!

cdce8p · 2023-07-20T04:22:01Z

Can confirm this fixes the build issue. Thanks for the help @cdce8p!

@nicholas-presien Would you mind reopening this again? My solution is only a workaround, this still needs a real / committed fix.

leidix · 2023-07-21T15:37:14Z

hope this gets officially fixed soon :)

Mostly it seems that cython wants you to declare things as `noexcept` Closes PyAV-Org#1140 I'm going to see if I can report the bug to cython, but it is hard for me to create a minimum reproducible example

dreamsavior · 2023-08-09T02:25:12Z

The problem is a lot of dependencies are not precompiled for Python 32 bit.
I switched to Python 64 bit and the usual python.exe -m pip install av run smoothly.

yubin-tech · 2023-09-11T08:06:16Z

It's an issue with cython==3.0 which was released a few days ago. Since PyAV doesn't use an upper limit in it's build dependencies, it's always used.

PyAV/pyproject.toml

Line 2 in 6982d81

requires = ["setuptools", "wheel", "cython"]

A workaround is to add a constraint. However, flags are not passed when creating the isolated build environments. For that it's necessary to use an environment variable. Something like this works
echo "cython<3.0" >> c.txt
PIP_CONSTRAINT=c.txt 

I am trying to trying to build pyav using the pip install av --no-binary av. The use of the environment variable did not seems to resolve the compile error. Is there something which I am missing?

ENate · 2023-09-22T22:17:53Z

Is there any fix to this issue? I am facing the same issue and cannot get it to work with the suggestion by @yubin-tech

abrvkh · 2023-10-06T18:02:32Z

Have there been any updates? I am facing the same issue. After trying the echo "cython<3.0" >> c.txt
PIP_CONSTRAINT=c.txt I am now getting error: command '/usr/bin/gcc' failed with exit code 1

ENate · 2023-10-06T19:03:22Z

I dropped to python 3.9 (seems to work) but another did come up when building the doc

dschneider-l5 · 2023-10-06T22:29:30Z

I was able to get it to install on Ubuntu 22.04 / python3.10 with the following (in a venv):

echo 'cython<3.0' > constraint.txt
export PIP_CONSTRAINT=constraint.txt
pip install -v av --no-binary av

(Note the use of "export" when setting the environment variable.)

Furthermore, I confirmed that I was able to use the hevc_cuvid codec for decoding an h.265 video stream from python and making the frames available for online image processing (which was not available in the binar ffmpeg installed by av).

tim-win · 2023-10-06T23:28:00Z

I'm using alpine 3.17 and the PIP_CONSTRAINT=c.txt workaround did the trick.

ENate · 2023-10-07T23:05:04Z

@dschneider-l5 I tried it before deciding to drop the Python version because it did not work in my case.

ENate · 2023-10-18T09:29:49Z

I added

[build-system]
requires = ["setuptools", "wheel", "cython<3.0"]

to the pyproject.toml but it doesn't solve the issue.

ENate · 2023-10-18T12:05:01Z

@dschneider-l5 did you install the ffmpeg libs seperately on Ubuntu 22.04? I will imagine these libs will installed during the build process, right?. Used your solution but got the following error on Ubuntu 22.04:

/home/miniconda3/envs/transformers/include/python3.11/object.h:581:29: note: in expansion of macro ‘_PyObject_CAST’
        581 |         PyObject *_py_tmp = _PyObject_CAST(op); \
            |                             ^~~~~~~~~~~~~~
      src/av/enum.c:15197:9: note: in expansion of macro ‘Py_CLEAR’
      15197 |         Py_CLEAR(f->f_back);
            |         ^~~~~~~~
      In file included from /home/miniconda3/envs/transformers/include/python3.11/Python.h:44,
                       from src/av/enum.c:33:
      src/av/enum.c:15197:19: error: invalid use of incomplete typedef ‘PyFrameObject’ {aka ‘struct _frame’}
      15197 |         Py_CLEAR(f->f_back);
            |                   ^~
      /home/miniconda3/envs/transformers/include/python3.11/object.h:583:14: note: in definition of macro ‘Py_CLEAR’
        583 |             (op) = NULL;                        \
            |              ^~
      error: command '/usr/bin/gcc' failed with exit code 1

 Building editable for transformers (pyproject.toml) ... done
  Created wheel for transformers: filename=transformers-4.34.0.dev0-0.editable-py3-none-any.whl size=39367 sha256=7e2198c71df124de3f59b62cb5ebaf86cdecbbdf5b7b3ffc8f190445b18b8b50
  Stored in directory: /tmp/pip-ephem-wheel-cache-vif9lk97/wheels/d3/9a/6d/b8fb54dc1ab15d2cf5a608abd895e4fa8b861133f942d602d4
Successfully built transformers

Though it seems it built the wheel but I cant see anything in the /tmp location. Any ideas?

dschneider-l5 · 2023-10-18T15:12:29Z

@ENate
On my system, ffmpeg and its libraries (e.g. libavcodec-dev) are installed at the system level via apt and not during the build process.

yubin-tech · 2023-10-19T01:34:35Z

How I got around the build issue temporarily is to build it directly from the source instead, and copy the binaries into the dist-package folded of the Python installation.

ENate · 2023-10-20T18:13:02Z

@yubin-tech can you explain how you did it? Thanks

gregfalx · 2023-12-17T13:52:03Z

Hello everyone, I had the exact same issue as @nicholas-presien . In my case, downgrading my Python version from 3.12 to 3.11.5 resolved the problem.

nicholas-presien added the build label Jul 18, 2023

cdce8p mentioned this issue Jul 18, 2023

TEST: Add 3.12-dev [ci] cdce8p/ha-core#53

Closed

32 tasks

nicholas-presien closed this as completed Jul 20, 2023

nicholas-presien reopened this Jul 20, 2023

hmaarrfk mentioned this issue Jul 23, 2023

Small updates to enable Cython 3 #1145

Merged

cdce8p mentioned this issue Jul 23, 2023

Fix wheel build - pin cython<3.0 #1146

Closed

BrenekH mentioned this issue Aug 5, 2023

Update auto-editor to 23w31a BrenekH/automated-aur#89

Merged

moi90 mentioned this issue Sep 3, 2023

CompileError: av/logging.pyx on main branch #1160

Closed

6 tasks

aruiplex mentioned this issue Sep 21, 2023

av 9.2 installation issue nerfstudio-project/nerfstudio#2409

Open

aliebald mentioned this issue Sep 23, 2023

Updated backend dependencies to fix installation error TUMFARSynchrony/experimental-hub#143

Merged

daveisfera mentioned this issue Oct 12, 2023

Limit cython version #1174

Closed

JohanLindvall mentioned this issue Oct 20, 2023

Error installing av #1177

Closed

WyattBlue closed this as completed in #1145 Oct 28, 2023

vagelim mentioned this issue Nov 15, 2023

Compilation Error When Installing faster-whisper on Fedora: Cython Compilation Issues with av/logging.pyx SYSTRAN/faster-whisper#560

Closed

PaulleDemon mentioned this issue Jan 21, 2024

Error on install PaulleDemon/tkVideoPlayer#46

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pip install is failing to build due to av/logging.pyx error #1140

Pip install is failing to build due to av/logging.pyx error #1140

nicholas-presien commented Jul 18, 2023

christianglodt commented Jul 18, 2023

samdbmg commented Jul 18, 2023

dhopsonlb commented Jul 18, 2023 •

edited

pal-sanich commented Jul 19, 2023

araczkowski commented Jul 19, 2023

abhinavsingh commented Jul 19, 2023

dhopsonlb commented Jul 19, 2023

cdce8p commented Jul 19, 2023

nicholas-presien commented Jul 20, 2023

cdce8p commented Jul 20, 2023

leidix commented Jul 21, 2023

dreamsavior commented Aug 9, 2023

yubin-tech commented Sep 11, 2023

ENate commented Sep 22, 2023

abrvkh commented Oct 6, 2023 •

edited

ENate commented Oct 6, 2023

dschneider-l5 commented Oct 6, 2023 •

edited

tim-win commented Oct 6, 2023

ENate commented Oct 7, 2023 •

edited

ENate commented Oct 18, 2023

ENate commented Oct 18, 2023 •

edited

dschneider-l5 commented Oct 18, 2023 •

edited

yubin-tech commented Oct 19, 2023

ENate commented Oct 20, 2023

gregfalx commented Dec 17, 2023

Pip install is failing to build due to av/logging.pyx error #1140

Pip install is failing to build due to av/logging.pyx error #1140

Comments

nicholas-presien commented Jul 18, 2023

Overview

Expected behavior

Actual behavior

Investigation

Reproduction

Versions

Research

christianglodt commented Jul 18, 2023

samdbmg commented Jul 18, 2023

dhopsonlb commented Jul 18, 2023 • edited

pal-sanich commented Jul 19, 2023

araczkowski commented Jul 19, 2023

abhinavsingh commented Jul 19, 2023

dhopsonlb commented Jul 19, 2023

cdce8p commented Jul 19, 2023

nicholas-presien commented Jul 20, 2023

cdce8p commented Jul 20, 2023

leidix commented Jul 21, 2023

dreamsavior commented Aug 9, 2023

yubin-tech commented Sep 11, 2023

ENate commented Sep 22, 2023

abrvkh commented Oct 6, 2023 • edited

ENate commented Oct 6, 2023

dschneider-l5 commented Oct 6, 2023 • edited

tim-win commented Oct 6, 2023

ENate commented Oct 7, 2023 • edited

ENate commented Oct 18, 2023

ENate commented Oct 18, 2023 • edited

dschneider-l5 commented Oct 18, 2023 • edited

yubin-tech commented Oct 19, 2023

ENate commented Oct 20, 2023

gregfalx commented Dec 17, 2023

dhopsonlb commented Jul 18, 2023 •

edited

abrvkh commented Oct 6, 2023 •

edited

dschneider-l5 commented Oct 6, 2023 •

edited

ENate commented Oct 7, 2023 •

edited

ENate commented Oct 18, 2023 •

edited

dschneider-l5 commented Oct 18, 2023 •

edited