`av==9.1.1` wheels seem to be be not packaged properly for Windows and Python 3.8/9 #952

pmeier · 2022-04-01T17:21:07Z

Overview

av==9.1.1 wheel released a couple of hours ago seems to be not packaged properly for Windows and Python 3.8 and 3.9.

Expected behavior

Import without issues.

Actual behavior

Traceback (most recent call last):
  File "C:\Users\circleci\project\test\datasets_utils.py", line 90, in _import
    module = importlib.import_module(package)
  File "C:\Users\circleci\project\env\lib\importlib\__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "C:\Users\circleci\project\env\lib\site-packages\av\__init__.py", line 44, in <module>
    from av._core import time_base, library_versions
ImportError: DLL load failed while importing _core: The specified module could not be found.

Investigation

CI runs:

✔️ Python 3.7
✖️ Python 3.8
✖️ Python 3.9
✔️ Python 3.10

Reproduction

import av

Versions

OS: Windows
PyAV runtime: 9.1.1

The text was updated successfully, but these errors were encountered:

pmeier · 2022-04-01T18:09:43Z

For reference, here are the two CI runs after we pinned av != 9.1.1:

✔️ Python 3.8
✔️ Python 3.9

jlaine · 2022-04-02T08:14:37Z

There was a significant change in this release which is that the wheels bow use an FFmpeg built from source (same as the other platforms), sorry for the breakage.

What intrigues me:

is that the failure is Python-version-specific
that we didn't get a CI failure

It sounds like there is a DLL missing, or clashing with another.. as I don't have access to a Windows machine this is going to be tricky to investigate.

jlaine · 2022-04-02T08:29:12Z

I just checked and CI definitely tested the 3.8 and 3.9 wheels on Windows:

https://github.com/PyAV-Org/PyAV/runs/5783121554?check_suite_focus=true

This leaves me wondering whether the wheels are actually broken or if we have a conflict with pytorch?

pmeier · 2022-04-04T06:12:52Z

is that the failure is Python-version-specific

Not only version specific, but also the "middle" ones. 3.7 and 3.10 are fine, which is especially peculiar.

This leaves me wondering whether the wheels are actually broken or if we have a conflict with pytorch?

I'll try to investigate on our side. @bjuncek are you aware of anything on our side that could cause this?

pmeier · 2022-04-04T06:40:33Z

Error persists even if we only setup the environment and not install anything form the PyTorch ecosystem. Thus, if there is a DLL conflict, it is not coming from us. Here are the envs:

3.7

# Name                    Version                   Build  Channel
atomicwrites              1.4.0                      py_0  
attrs                     21.4.0             pyhd3eb1b0_0  
av                        9.1.1                    pypi_0    pypi
ca-certificates           2022.3.29            haa95532_0  
cached-property           1.5.2                    pypi_0    pypi
certifi                   2021.10.8        py37haa95532_2  
colorama                  0.4.4              pyhd3eb1b0_0  
coverage                  6.3.2            py37h2bbff1b_0  
dataclasses               0.6                      pypi_0    pypi
future                    0.18.2                   pypi_0    pypi
h5py                      3.6.0                    pypi_0    pypi
hdf5                      1.12.0               h1756f20_0  
icc_rt                    2019.0.0             h0cc432a_1  
importlib-metadata        4.11.3           py37haa95532_0  
importlib_metadata        4.11.3               hd3eb1b0_0  
iniconfig                 1.1.1              pyhd3eb1b0_0  
jpeg                      9d                   h2bbff1b_0  
libpng                    1.6.37               h2a8f88b_0  
numpy                     1.21.5                   pypi_0    pypi
openssl                   1.1.1n               h2bbff1b_0  
packaging                 21.3               pyhd3eb1b0_0  
pillow                    9.1.0                    pypi_0    pypi
pip                       21.2.4           py37haa95532_0  
pluggy                    1.0.0            py37haa95532_1  
py                        1.11.0             pyhd3eb1b0_0  
pyparsing                 3.0.4              pyhd3eb1b0_0  
pytest                    6.2.5            py37haa95532_2  
pytest-cov                3.0.0              pyhd3eb1b0_0  
pytest-mock               3.6.1              pyhd3eb1b0_0  
python                    3.7.13               h6244533_0  
scipy                     1.7.3                    pypi_0    pypi
setuptools                58.0.4           py37haa95532_0  
sqlite                    3.38.2               h2bbff1b_0  
toml                      0.10.2             pyhd3eb1b0_0  
typing_extensions         4.1.1              pyh06a4308_0  
vc                        14.2                 h21ff451_1  
vs2015_runtime            14.27.29016          h5e58377_2  
wheel                     0.37.1             pyhd3eb1b0_0  
wincertstore              0.2              py37haa95532_2  
zipp                      3.7.0              pyhd3eb1b0_0  
zlib                      1.2.11               hbd8134f_5

3.8

# Name                    Version                   Build  Channel
atomicwrites              1.4.0                      py_0  
attrs                     21.4.0             pyhd3eb1b0_0  
av                        9.1.1                    pypi_0    pypi
ca-certificates           2022.3.29            haa95532_0  
certifi                   2021.10.8        py38haa95532_2  
colorama                  0.4.4              pyhd3eb1b0_0  
coverage                  6.3.2            py38h2bbff1b_0  
dataclasses               0.6                      pypi_0    pypi
future                    0.18.2                   pypi_0    pypi
h5py                      3.6.0                    pypi_0    pypi
hdf5                      1.12.0               h1756f20_0  
icc_rt                    2019.0.0             h0cc432a_1  
iniconfig                 1.1.1              pyhd3eb1b0_0  
jpeg                      9d                   h2bbff1b_0  
libpng                    1.6.37               h2a8f88b_0  
numpy                     1.22.3                   pypi_0    pypi
openssl                   1.1.1n               h2bbff1b_0  
packaging                 21.3               pyhd3eb1b0_0  
pillow                    9.1.0                    pypi_0    pypi
pip                       21.2.2           py38haa95532_0  
pluggy                    1.0.0            py38haa95532_1  
py                        1.11.0             pyhd3eb1b0_0  
pyparsing                 3.0.4              pyhd3eb1b0_0  
pytest                    6.2.5            py38haa95532_2  
pytest-cov                3.0.0              pyhd3eb1b0_0  
pytest-mock               3.6.1              pyhd3eb1b0_0  
python                    3.8.13               h6244533_0  
scipy                     1.8.0                    pypi_0    pypi
setuptools                58.0.4           py38haa95532_0  
sqlite                    3.38.2               h2bbff1b_0  
toml                      0.10.2             pyhd3eb1b0_0  
vc                        14.2                 h21ff451_1  
vs2015_runtime            14.27.29016          h5e58377_2  
wheel                     0.37.1             pyhd3eb1b0_0  
wincertstore              0.2              py38haa95532_2  
zlib                      1.2.11               hbd8134f_5

3.9

# Name                    Version                   Build  Channel
atomicwrites              1.4.0                      py_0  
attrs                     21.4.0             pyhd3eb1b0_0  
av                        9.1.1                    pypi_0    pypi
ca-certificates           2022.3.29            haa95532_0  
certifi                   2021.10.8        py39haa95532_2  
colorama                  0.4.4              pyhd3eb1b0_0  
coverage                  6.3.2            py39h2bbff1b_0  
dataclasses               0.6                      pypi_0    pypi
future                    0.18.2                   pypi_0    pypi
h5py                      3.6.0                    pypi_0    pypi
hdf5                      1.12.0               h1756f20_0  
icc_rt                    2019.0.0             h0cc432a_1  
iniconfig                 1.1.1              pyhd3eb1b0_0  
jpeg                      9d                   h2bbff1b_0  
libpng                    1.6.37               h2a8f88b_0  
numpy                     1.22.3                   pypi_0    pypi
openssl                   1.1.1n               h2bbff1b_0  
packaging                 21.3               pyhd3eb1b0_0  
pillow                    9.1.0                    pypi_0    pypi
pip                       21.2.4           py39haa95532_0  
pluggy                    1.0.0            py39haa95532_1  
py                        1.11.0             pyhd3eb1b0_0  
pyparsing                 3.0.4              pyhd3eb1b0_0  
pytest                    6.2.5            py39haa95532_2  
pytest-cov                3.0.0              pyhd3eb1b0_0  
pytest-mock               3.6.1              pyhd3eb1b0_0  
python                    3.9.11               h6244533_2  
scipy                     1.8.0                    pypi_0    pypi
setuptools                58.0.4           py39haa95532_0  
sqlite                    3.38.2               h2bbff1b_0  
toml                      0.10.2             pyhd3eb1b0_0  
tzdata                    2022a                hda174b7_0  
vc                        14.2                 h21ff451_1  
vs2015_runtime            14.27.29016          h5e58377_2  
wheel                     0.37.1             pyhd3eb1b0_0  
wincertstore              0.2              py39haa95532_2  
zlib                      1.2.11               hbd8134f_5

3.10

# Name                    Version                   Build  Channel
atomicwrites              1.4.0                      py_0  
attrs                     21.4.0             pyhd3eb1b0_0  
av                        9.1.1                    pypi_0    pypi
bzip2                     1.0.8                he774522_0  
ca-certificates           2022.3.29            haa95532_0  
certifi                   2021.5.30       py310haa95532_0  
colorama                  0.4.4              pyhd3eb1b0_0  
coverage                  6.3.2           py310h2bbff1b_0  
dataclasses               0.6                      pypi_0    pypi
future                    0.18.2                   pypi_0    pypi
h5py                      3.6.0                    pypi_0    pypi
hdf5                      1.12.0               h1756f20_0  
icc_rt                    2019.0.0             h0cc432a_1  
iniconfig                 1.1.1              pyhd3eb1b0_0  
jpeg                      9d                   h2bbff1b_0  
libffi                    3.4.2                h604cdb4_1  
libpng                    1.6.37               h2a8f88b_0  
more-itertools            8.12.0             pyhd3eb1b0_0  
numpy                     1.22.3                   pypi_0    pypi
openssl                   1.1.1n               h2bbff1b_0  
packaging                 21.3               pyhd3eb1b0_0  
pillow                    9.1.0                    pypi_0    pypi
pip                       21.2.4          py310haa95532_0  
pluggy                    0.13.1          py310haa95532_0  
py                        1.11.0             pyhd3eb1b0_0  
pyparsing                 3.0.4              pyhd3eb1b0_0  
pytest                    6.2.4           py310haa95532_2  
pytest-cov                3.0.0              pyhd3eb1b0_0  
pytest-mock               3.6.1              pyhd3eb1b0_0  
python                    3.10.4               hbb2ffb3_0  
scipy                     1.8.0                    pypi_0    pypi
setuptools                58.0.4          py310haa95532_0  
sqlite                    3.38.2               h2bbff1b_0  
tk                        8.6.11               h2bbff1b_0  
toml                      0.10.2             pyhd3eb1b0_0  
tzdata                    2022a                hda174b7_0  
vc                        14.2                 h21ff451_1  
vs2015_runtime            14.27.29016          h5e58377_2  
wheel                     0.37.1             pyhd3eb1b0_0  
wincertstore              0.2             py310haa95532_2  
xz                        5.2.5                h62dcd97_0  
zlib                      1.2.11               hbd8134f_5

Is there a "verbose mode" or the like to find which DLL breaks the import?

pmeier · 2022-04-04T06:58:45Z

Failure also happens if nothing but pyav is installed:

3.8

# Name                    Version                   Build  Channel
av                        9.1.1                    pypi_0    pypi
ca-certificates           2022.3.29            haa95532_0  
certifi                   2021.10.8        py38haa95532_2  
openssl                   1.1.1n               h2bbff1b_0  
pip                       21.2.2           py38haa95532_0  
python                    3.8.13               h6244533_0  
setuptools                58.0.4           py38haa95532_0  
sqlite                    3.38.2               h2bbff1b_0  
vc                        14.2                 h21ff451_1  
vs2015_runtime            14.27.29016          h5e58377_2  
wheel                     0.37.1             pyhd3eb1b0_0  
wincertstore              0.2              py38haa95532_2

3.9

# Name                    Version                   Build  Channel
av                        9.1.1                    pypi_0    pypi
ca-certificates           2022.3.29            haa95532_0  
certifi                   2021.10.8        py39haa95532_2  
openssl                   1.1.1n               h2bbff1b_0  
pip                       21.2.4           py39haa95532_0  
python                    3.9.11               h6244533_2  
setuptools                58.0.4           py39haa95532_0  
sqlite                    3.38.2               h2bbff1b_0  
tzdata                    2022a                hda174b7_0  
vc                        14.2                 h21ff451_1  
vs2015_runtime            14.27.29016          h5e58377_2  
wheel                     0.37.1             pyhd3eb1b0_0  
wincertstore              0.2              py39haa95532_2

jlaine · 2022-04-04T19:29:55Z

I had a closer look at the DLLs contained in the av 9.1.0 vs 9.1.1 wheels and:

the DLL names seem to be properly mangled so we shouldn't experience "DLL hell"
the only additional required library seems to be MFPlat.DLL (media foundation), which I understand may not be present on all Windows installations.

If I provide you with a wheel for a Python version of your choice would you be able to test it? Otherwise it means going through a full release process, potentially with no gain..

pmeier · 2022-04-05T06:03:23Z

Yes, we could test if we had a wheel available.

jlaine · 2022-04-05T10:34:00Z

OK, the FFmpeg rebuild has completed (11 hours to cover all platforms..), and I've put together PR #954 to produce updated wheels. I'll keep you posted.

jlaine · 2022-04-05T16:57:31Z

Here are the wheels for Windows/64bit.. As github won't allow me to upload .whl files, nor to put them all in one big zip, I've had to put each in its .zip, sorry for the inconvenience!

av-9.1.1-cp37-cp37m-win_amd64.whl.zip
av-9.1.1-cp38-cp38-win_amd64.whl.zip
av-9.1.1-cp39-cp39-win_amd64.whl.zip
av-9.1.1-cp310-cp310-win_amd64.whl.zip

jlaine · 2022-04-07T09:43:13Z

Hi @pmeier have you had a chance to check the wheels? The reason I'm asking is I'd like to cut a new release and would like to make sure the issue is fixed for you.

pmeier · 2022-04-07T09:50:35Z

I'll have some time in a couple of hours. Sorry for the delay.

pmeier · 2022-04-07T12:35:00Z

The issue persists:

✔️ Python 3.7
✖️ Python 3.8
✖️ Python 3.9
✔️ Python 3.10

jlaine · 2022-04-08T12:17:33Z

Thanks for your feedback @pmeier so we're not quite done here.. It looks as though I'm going to need to find a way to spin up a Windows VM somehow, because the ImportError really doesn't tell us anything. I'm probably not going to be very online much for the coming week, but I'll get to the bottom of this before our next release - which I'll therefore postpone.

pmeier · 2022-04-08T12:21:13Z

No rush on our side. We pinned av != 9.1.1 for now so unless you publish a new release with the same issues, there is no extra work on our side.

jlaine · 2022-04-08T12:23:56Z

No rush on our side. We pinned av != 9.1.1 for now so unless you publish a new release with the same issues, there is no extra work on our side.

Good to know but it's still pretty annoying I introduced a regression! I really wanted to make sure we're not depending on any third-party binaries, so we can ensure all our platforms have consistent behaviour and security fixes.

uvjustin · 2022-04-12T04:28:29Z

Something something underscore?
pypa/cibuildwheel#1076
https://stackoverflow.com/questions/66586674/python-3-8-dll-issue-importerror-dll-load-failed-while-importing-libprojecta
https://stackoverflow.com/questions/66814716/issue-in-importing-awkward-array-package-dll-load-failed-while-importing-ext

jlaine · 2022-04-17T14:07:07Z

@uvjustin I really have no idea how to debug this as I don't have a Windows box, and even if I did I don't know what tool (equivalent to strace) I should use to figure out where the DLL loading breaks. Do you have any ideas here, as I could really use some help?

uvjustin · 2022-04-17T18:21:15Z

Sure, I'll look into this

jlaine · 2022-04-19T10:39:11Z

OK, so after quite a bit of trouble I managed to install a Windows VM, using the image provided by Microsoft:

https://developer.microsoft.com/en-us/windows/downloads/virtual-machines/

I then installed Python 3.10, created a virtualenv, installed PyAV 9.1.1 from PyPI.. and it loads just fine?

So it looks as though we're going to need more information about your environment, because so far I am unable to reproduce.

pmeier · 2022-04-19T11:38:41Z

Python 3.10 as well as 3.7 are fine. The wheels for Python 3.8 and 3.9 are not working. As for the environments, I posted our conda environments in #952 (comment). Otherwise we are using a default Windows CI runner.

jlaine · 2022-04-19T12:55:42Z

Python 3.9 works for me too:

pmeier · 2022-04-19T13:36:16Z

We are using a Windows Server 2019 image. Any idea how I could help debug this? Is there a way to find which DLL is causing this?

jlaine · 2022-04-19T14:31:14Z

I'm really not familiar enough with Windows to give you an answer, I was hoping I could trigger the error on a Windows VM so I could try and debug it, but unfortunately I'm not reproducing the error. Additionally I've put together a test which does:

pip install av
python -c "import av; print(av.__version__)"

.. across (Windows Server 2019, Windows Server 2022) x (Python 3.7, 3.8, 3.9, 3.10) and I haven't managed to get it to fail:

https://github.com/jlaine/pyav-test/actions/runs/2190230680

pmeier · 2022-04-19T14:36:01Z

Hmm, if GitHub Actions cleanly installs and imports the wheel, I'm back wondering if there is actually something wrong with our env. I'll think about it and get back to you.

jlaine · 2022-04-19T14:45:50Z

At this stage I see two main sources of potential differences:

do CircleCI and GitHub Actions have the same definition of what "Windows Server 2019" mean? Is one or the other installing additional packages?
is conda interfering?

uvjustin · 2022-04-19T14:56:57Z

At this stage I see two main sources of potential differences:

do CircleCI and GitHub Actions have the same definition of what "Windows Server 2019" mean? Is one or the other installing additional packages?

is conda interfering?

When googling this the other day I found evidence of the latter. I can actually reproduce this on a conda env.

pmeier · 2022-04-19T15:46:21Z

I'll investigate on our side.

jlaine · 2022-04-19T16:53:37Z

OK I'm learning more than I care to here, which involves messing with gflags.exe and windbg.exe but here goes.. I've set up conda and I'm indeed able to trigger an error when using Python 3.9.

AFAICT I can tell, it's not a deep dependency that's causing a problem, DLL loading seems to fail as soon as avformat-58-xxx.dll. Examining the output, the av.libs directory which contains the DLLs is not even being examined. This leads me to believe that conda's Python has been somehow modified, which interferes with the DLL search path!

uvjustin · 2022-04-19T17:31:39Z

See adang1345/delvewheel#14
Actually that issue seems to have been resolved. Sure seems similar though...

jlaine · 2022-04-19T18:03:34Z

Interestingly enough, PyAV's __init__.py has some code of its own which possibly worked around this issue by manipulating os.environ["PATH"]. I'm not sure why this code is there, added in 2015, way before we shipped binary wheels to my knowledge.

What changed in 9.1.1 is that we no longer vendor the DLLs inside the av directory, but in av.libs. If I modify this code to look for DLLs in av.libs .. it works?!

…-Org#952)

jlaine · 2022-04-20T06:09:03Z

Here is a new round of wheels:

av-9.1.1-cp37-cp37m-win_amd64.whl.zip
av-9.1.1-cp38-cp38-win_amd64.whl.zip
av-9.1.1-cp39-cp39-win_amd64.whl.zip
av-9.1.1-cp310-cp310-win_amd64.whl.zip

jlaine · 2022-04-20T06:19:12Z

I've just tested all 4 wheels using conda, and they all work, so I'm merging the PR!

pmeier · 2022-04-20T06:40:45Z

Our smoke tests are also passing 🎉 I'll do a full run with the test suite to see if everything else is still working and get back to you.

pmeier · 2022-04-20T10:01:16Z

Can confirm, with the new wheels our test suite is passing fine. Thanks a lot for the investigation and fix!

jlaine · 2022-04-20T10:07:12Z

PyAV 9.2.0 is on PyPI, and I have just checked that the wheels work on 3.7/3.8/3.9/3.10 using Conda, so we're all good here!

pmeier added the bug label Apr 1, 2022

pmeier changed the title ~~The FOO does not BAR.~~ av==9.1.1 seems to be be not packaged properly for Windows and Python 3.8/9 Apr 1, 2022

pmeier mentioned this issue Apr 1, 2022

disable pyav==9.1.1 on windows pytorch/vision#5724

Merged

pmeier changed the title ~~av==9.1.1 seems to be be not packaged properly for Windows and Python 3.8/9~~ av==9.1.1 wheels seem to be be not packaged properly for Windows and Python 3.8/9 Apr 1, 2022

pmeier mentioned this issue Apr 4, 2022

[DEBUG] possible pyav DLL conflict pytorch/vision#5731

Closed

jlaine mentioned this issue Apr 17, 2022

Win10: ImportError: DLL load failed: The specified module could not be found #490

Closed

jlaine added the help wanted The maintainer could use help here. label Apr 17, 2022

jlaine added a commit to jlaine/PyAV that referenced this issue Apr 19, 2022

[package] work around Conda bug which breaks DLL loading (fixes: PyAV…

24d19d8

…-Org#952)

jlaine added a commit to jlaine/PyAV that referenced this issue Apr 19, 2022

[package] work around Conda bug which breaks DLL loading (fixes: PyAV…

a92ab78

…-Org#952)

jlaine closed this as completed in af1493c Apr 20, 2022

PaulleDemon mentioned this issue Dec 10, 2023

error install in macos m1 PaulleDemon/tkVideoPlayer#45

Closed

av==9.1.1 wheels seem to be be not packaged properly for Windows and Python 3.8/9 #952

av==9.1.1 wheels seem to be be not packaged properly for Windows and Python 3.8/9 #952

Comments

pmeier commented Apr 1, 2022 • edited

Overview

Expected behavior

Actual behavior

Investigation

Reproduction

Versions

pmeier commented Apr 1, 2022 • edited

jlaine commented Apr 2, 2022 • edited

jlaine commented Apr 2, 2022

pmeier commented Apr 4, 2022

pmeier commented Apr 4, 2022 • edited

pmeier commented Apr 4, 2022 • edited

jlaine commented Apr 4, 2022

pmeier commented Apr 5, 2022

jlaine commented Apr 5, 2022

jlaine commented Apr 5, 2022

jlaine commented Apr 7, 2022

pmeier commented Apr 7, 2022

pmeier commented Apr 7, 2022

jlaine commented Apr 8, 2022 • edited

pmeier commented Apr 8, 2022

jlaine commented Apr 8, 2022

uvjustin commented Apr 12, 2022

jlaine commented Apr 17, 2022

uvjustin commented Apr 17, 2022

jlaine commented Apr 19, 2022 • edited

pmeier commented Apr 19, 2022

jlaine commented Apr 19, 2022

pmeier commented Apr 19, 2022

jlaine commented Apr 19, 2022 • edited

pmeier commented Apr 19, 2022

jlaine commented Apr 19, 2022

uvjustin commented Apr 19, 2022

pmeier commented Apr 19, 2022

jlaine commented Apr 19, 2022

uvjustin commented Apr 19, 2022 • edited

jlaine commented Apr 19, 2022 • edited

jlaine commented Apr 20, 2022

jlaine commented Apr 20, 2022

pmeier commented Apr 20, 2022

pmeier commented Apr 20, 2022

jlaine commented Apr 20, 2022 • edited

`av==9.1.1` wheels seem to be be not packaged properly for Windows and Python 3.8/9 #952

`av==9.1.1` wheels seem to be be not packaged properly for Windows and Python 3.8/9 #952

pmeier commented Apr 1, 2022 •

edited

pmeier commented Apr 1, 2022 •

edited

jlaine commented Apr 2, 2022 •

edited

pmeier commented Apr 4, 2022 •

edited

pmeier commented Apr 4, 2022 •

edited

jlaine commented Apr 8, 2022 •

edited

jlaine commented Apr 19, 2022 •

edited

jlaine commented Apr 19, 2022 •

edited

uvjustin commented Apr 19, 2022 •

edited

jlaine commented Apr 19, 2022 •

edited

jlaine commented Apr 20, 2022 •

edited