Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

imageio and imageio-ffmpeg Using isAlive() in Python3.9 #808

Closed
jmdelahanty opened this issue May 3, 2022 · 15 comments
Closed

imageio and imageio-ffmpeg Using isAlive() in Python3.9 #808

jmdelahanty opened this issue May 3, 2022 · 15 comments

Comments

@jmdelahanty
Copy link

Hello ImageIO Devs!

I'm currently using your reader with PIMS and Dask since it seems like ffmpeg backends are the way things are moving for video editing. It's for this project.

While running my code on a Linux machine in an environment that has Python 3.9.2, I received the following error message:

Traceback (most recent call last):
  File "/nadata/snlkt/data/facial_expression/specialk/MouseFacialExpressionAnalysis/dask_faces.py", line 635, in <module>
    hog_images.to_zarr("/scratch/snlkt_facial_expression/hogs/data.zarr", overwrite=True)
  File "/home/jdelahanty/miniconda3/envs/facial_expression2/lib/python3.9/site-packages/dask/array/core.py", line 2784, in to_zarr
    return to_zarr(self, *args, **kwargs)
  File "/home/jdelahanty/miniconda3/envs/facial_expression2/lib/python3.9/site-packages/dask/array/core.py", line 3547, in to_zarr
    return arr.store(z, lock=False, compute=compute, return_stored=return_stored)
  File "/home/jdelahanty/miniconda3/envs/facial_expression2/lib/python3.9/site-packages/dask/array/core.py", line 1710, in store
    r = store([self], [target], **kwargs)
  File "/home/jdelahanty/miniconda3/envs/facial_expression2/lib/python3.9/site-packages/dask/array/core.py", line 1184, in store
    compute_as_if_collection(Array, store_dsk, map_keys, **kwargs)
  File "/home/jdelahanty/miniconda3/envs/facial_expression2/lib/python3.9/site-packages/dask/base.py", line 319, in compute_as_if_collection
    return schedule(dsk2, keys, **kwargs)
  File "/home/jdelahanty/miniconda3/envs/facial_expression2/lib/python3.9/site-packages/dask/multiprocessing.py", line 220, in get
    result = get_async(
  File "/home/jdelahanty/miniconda3/envs/facial_expression2/lib/python3.9/site-packages/dask/local.py", line 508, in get_async
    raise_exception(exc, tb)
  File "/home/jdelahanty/miniconda3/envs/facial_expression2/lib/python3.9/site-packages/dask/local.py", line 316, in reraise
    raise exc
  File "/home/jdelahanty/miniconda3/envs/facial_expression2/lib/python3.9/site-packages/dask/local.py", line 221, in execute_task
    result = _execute_task(task, data)
  File "/home/jdelahanty/miniconda3/envs/facial_expression2/lib/python3.9/site-packages/dask/core.py", line 119, in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
  File "/home/jdelahanty/miniconda3/envs/facial_expression2/lib/python3.9/site-packages/dask/core.py", line 119, in <genexpr>
    return func(*(_execute_task(a, cache) for a in args))
  File "/home/jdelahanty/miniconda3/envs/facial_expression2/lib/python3.9/site-packages/dask/core.py", line 119, in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
  File "/home/jdelahanty/miniconda3/envs/facial_expression2/lib/python3.9/site-packages/dask/core.py", line 119, in <genexpr>
    return func(*(_execute_task(a, cache) for a in args))
  File "/home/jdelahanty/miniconda3/envs/facial_expression2/lib/python3.9/site-packages/dask/core.py", line 119, in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
  File "/home/jdelahanty/miniconda3/envs/facial_expression2/lib/python3.9/site-packages/dask/core.py", line 119, in <genexpr>
    return func(*(_execute_task(a, cache) for a in args))
  File "/home/jdelahanty/miniconda3/envs/facial_expression2/lib/python3.9/site-packages/dask/core.py", line 119, in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
  File "/home/jdelahanty/miniconda3/envs/facial_expression2/lib/python3.9/site-packages/dask/core.py", line 119, in <genexpr>
    return func(*(_execute_task(a, cache) for a in args))
  File "/home/jdelahanty/miniconda3/envs/facial_expression2/lib/python3.9/site-packages/dask/core.py", line 113, in _execute_task
    return [_execute_task(a, cache) for a in arg]
  File "/home/jdelahanty/miniconda3/envs/facial_expression2/lib/python3.9/site-packages/dask/core.py", line 113, in <listcomp>
    return [_execute_task(a, cache) for a in arg]
  File "/home/jdelahanty/miniconda3/envs/facial_expression2/lib/python3.9/site-packages/dask/core.py", line 119, in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
  File "/home/jdelahanty/miniconda3/envs/facial_expression2/lib/python3.9/site-packages/dask/optimization.py", line 990, in __call__
    return core.get(self.dsk, self.outkey, dict(zip(self.inkeys, args)))
  File "/home/jdelahanty/miniconda3/envs/facial_expression2/lib/python3.9/site-packages/dask/core.py", line 149, in get
    result = _execute_task(task, cache)
  File "/home/jdelahanty/miniconda3/envs/facial_expression2/lib/python3.9/site-packages/dask/core.py", line 119, in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
  File "/home/jdelahanty/miniconda3/envs/facial_expression2/lib/python3.9/site-packages/dask/utils.py", line 40, in apply
    return func(*args, **kwargs)
  File "/home/jdelahanty/miniconda3/envs/facial_expression2/lib/python3.9/site-packages/dask/array/core.py", line 514, in _pass_extra_kwargs
    return func(*args[len(keys) :], **kwargs)
  File "/home/jdelahanty/miniconda3/envs/facial_expression2/lib/python3.9/site-packages/dask_image/imread/__init__.py", line 99, in _map_read_frame
    return _utils._read_frame(fn=fn, i=slice(i, j), **kwargs)
  File "/home/jdelahanty/miniconda3/envs/facial_expression2/lib/python3.9/site-packages/dask_image/imread/_utils.py", line 8, in _read_frame
    return arrayfunc(imgs[i])
  File "/home/jdelahanty/miniconda3/envs/facial_expression2/lib/python3.9/site-packages/slicerator/__init__.py", line 226, in <genexpr>
    return (self._get(i) for i in self.indices)
  File "/home/jdelahanty/miniconda3/envs/facial_expression2/lib/python3.9/site-packages/slicerator/__init__.py", line 206, in _get
    return self._ancestor[key]
  File "/home/jdelahanty/miniconda3/envs/facial_expression2/lib/python3.9/site-packages/slicerator/__init__.py", line 187, in __getitem__
    return self._get(indices)
  File "/home/jdelahanty/miniconda3/envs/facial_expression2/lib/python3.9/site-packages/pims/base_frames.py", line 100, in __getitem__
    return self.get_frame(key)
  File "/home/jdelahanty/miniconda3/envs/facial_expression2/lib/python3.9/site-packages/pims/base_frames.py", line 598, in get_frame
    result = self._get_frame_wrapped(**coords)
  File "/home/jdelahanty/miniconda3/envs/facial_expression2/lib/python3.9/site-packages/pims/imageio_reader.py", line 123, in get_frame_2D
    frame = self.reader.get_data(i)
  File "/home/jdelahanty/miniconda3/envs/facial_expression2/lib/python3.9/site-packages/imageio/core/format.py", line 340, in get_data
    im, meta = self._get_data(index, **kwargs)
  File "/home/jdelahanty/miniconda3/envs/facial_expression2/lib/python3.9/site-packages/imageio/plugins/ffmpeg.py", line 396, in _get_data
    result, is_new = self._read_frame()
  File "/home/jdelahanty/miniconda3/envs/facial_expression2/lib/python3.9/site-packages/imageio/plugins/ffmpeg.py", line 585, in _read_frame
    s, is_new = self._read_frame_data()
  File "/home/jdelahanty/miniconda3/envs/facial_expression2/lib/python3.9/site-packages/imageio/plugins/ffmpeg.py", line 569, in _read_frame_data
    err2 = self._stderr_catcher.get_text(0.4)
  File "/home/jdelahanty/miniconda3/envs/facial_expression2/lib/python3.9/site-packages/imageio/plugins/ffmpeg.py", line 901, in get_text
    while self.isAlive() and time.time() < etime:  # pragma: no cover
AttributeError: 'StreamCatcher' object has no attribute 'isAlive'

After googling around a little, I found a few other issues that talk about how isAlive was deprecated at some point. I'm not really sure why other than it seems like they wanted to move to snake_case instead of camelCase?

When I looked at the source code in my Conda environment as the Traceback describes, I indeed found the use of isAlive! Changing this to is_alive lets things work again!

Below is my full environment for reference.

# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main
_openmp_mutex             4.5                       1_gnu
appdirs                   1.4.4              pyh9f0ad1d_0    conda-forge
asciitree                 0.3.3                      py_2    conda-forge
blas                      1.0                         mkl
blosc                     1.21.0               h8c45485_0
bokeh                     2.4.2            py39hf3d152e_1    conda-forge
bottleneck                1.3.4            py39hce1f21e_0
brotli                    1.0.9                h7f98852_5    conda-forge
brotli-bin                1.0.9                h7f98852_5    conda-forge
brotlipy                  0.7.0           py39h3811e60_1001    conda-forge
brunsli                   0.1                  h9c3ff4c_0    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
c-ares                    1.18.1               h7f8727e_0
ca-certificates           2021.10.8            ha878542_0    conda-forge
cached-property           1.5.2                hd8ed1ab_1    conda-forge
cached_property           1.5.2              pyha770c72_1    conda-forge
certifi                   2021.10.8        py39hf3d152e_2    conda-forge
cffi                      1.15.0           py39hd667e15_1
cfitsio                   3.470                h2e3daa1_7    conda-forge
charls                    2.2.0                h2531618_0
charset-normalizer        2.0.12             pyhd8ed1ab_0    conda-forge
click                     8.1.3            py39hf3d152e_0    conda-forge
cloudpickle               2.0.0              pyhd8ed1ab_0    conda-forge
cryptography              35.0.0           py39hbca0aa6_0    conda-forge
cycler                    0.11.0             pyhd8ed1ab_0    conda-forge
cytoolz                   0.11.0           py39h27cfd23_0
dask                      2022.4.2           pyhd8ed1ab_0    conda-forge
dask-core                 2022.4.2           pyhd8ed1ab_0    conda-forge
dask-image                2021.12.0          pyhd8ed1ab_0    conda-forge
decorator                 5.1.1              pyhd8ed1ab_0    conda-forge
distributed               2022.4.2           pyhd8ed1ab_0    conda-forge
fasteners                 0.17.3             pyhd8ed1ab_0    conda-forge
ffmpeg                    4.3.2                hca11adc_0    conda-forge
freetype                  2.10.4               h0708190_1    conda-forge
fsspec                    2022.3.0           pyhd8ed1ab_0    conda-forge
giflib                    5.2.1                h36c2ea0_2    conda-forge
gmp                       6.2.1                h58526e2_0    conda-forge
gnutls                    3.6.13               h85f3911_1    conda-forge
h5py                      3.2.1           nompi_py39h98ba4bc_100    conda-forge
hdf5                      1.10.6          nompi_h7c3c948_1111    conda-forge
heapdict                  1.0.1                      py_0    conda-forge
idna                      3.3                pyhd8ed1ab_0    conda-forge
imagecodecs               2021.8.26        py39h4cda21f_0
imageio                   2.3.0                      py_1    conda-forge
imageio-ffmpeg            0.4.7              pyhd8ed1ab_0    conda-forge
intel-openmp              2021.4.0          h06a4308_3561
jinja2                    3.1.1              pyhd8ed1ab_0    conda-forge
jpeg                      9e                   h7f8727e_0
jxrlib                    1.1                  h7f98852_2    conda-forge
kiwisolver                1.3.2            py39h295c915_0
krb5                      1.19.2               hac12032_0
lame                      3.100             h7f98852_1001    conda-forge
lcms2                     2.12                 hddcbb42_0    conda-forge
ld_impl_linux-64          2.35.1               h7274673_9
lerc                      3.0                  h295c915_0
libaec                    1.0.4                he6710b0_1
libbrotlicommon           1.0.9                h7f98852_5    conda-forge
libbrotlidec              1.0.9                h7f98852_5    conda-forge
libbrotlienc              1.0.9                h7f98852_5    conda-forge
libcurl                   7.82.0               h0b77cf5_0
libdeflate                1.8                  h7f98852_0    conda-forge
libedit                   3.1.20210910         h7f8727e_0
libev                     4.33                 h516909a_1    conda-forge
libffi                    3.3                  he6710b0_2
libgcc-ng                 9.3.0               h5101ec6_17
libgfortran-ng            7.5.0               h14aa051_20    conda-forge
libgfortran4              7.5.0               h14aa051_20    conda-forge
libgomp                   9.3.0               h5101ec6_17
libnghttp2                1.46.0               hce63b2e_0
libpng                    1.6.37               h21135ba_2    conda-forge
libssh2                   1.10.0               h8f2d780_0
libstdcxx-ng              9.3.0               hd4cf53a_17
libtiff                   4.2.0                h85742a9_0
libwebp                   1.2.2                h55f646e_0
libwebp-base              1.2.2                h7f8727e_0
libzopfli                 1.0.3                h9c3ff4c_0    conda-forge
locket                    1.0.0              pyhd8ed1ab_0    conda-forge
lz4                       3.1.3            py39h27cfd23_0
lz4-c                     1.9.3                h9c3ff4c_1    conda-forge
markupsafe                2.0.1            py39h3811e60_0    conda-forge
matplotlib-base           3.3.4            py39h2fa2bec_0    conda-forge
mkl                       2021.4.0           h06a4308_640
mkl-service               2.4.0            py39h3811e60_0    conda-forge
mkl_fft                   1.3.1            py39hd3c417c_0
mkl_random                1.2.2            py39hde0f152_0    conda-forge
msgpack-python            1.0.2            py39hff7bd54_1
ncurses                   6.3                  h7f8727e_2
nettle                    3.6                  he412f7d_0    conda-forge
networkx                  2.3                        py_0    conda-forge
numcodecs                 0.9.1            py39h295c915_0
numexpr                   2.8.1            py39h6abb31d_0
numpy                     1.21.5           py39he7a7128_1
numpy-base                1.21.5           py39hf524024_1
olefile                   0.46               pyh9f0ad1d_1    conda-forge
openh264                  2.1.1                h780b84a_0    conda-forge
openjpeg                  2.4.0                hb52868f_1    conda-forge
openssl                   1.1.1n               h7f8727e_0
packaging                 21.3               pyhd8ed1ab_0    conda-forge
pandas                    1.4.2            py39h295c915_0
partd                     1.2.0              pyhd8ed1ab_0    conda-forge
pillow                    7.2.0            py39h6f3857e_2    conda-forge
pims                      0.6.0              pyhd8ed1ab_0    conda-forge
pip                       21.2.4           py39h06a4308_0
pooch                     1.6.0              pyhd8ed1ab_0    conda-forge
psutil                    5.8.0            py39h27cfd23_1
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pyopenssl                 22.0.0             pyhd8ed1ab_0    conda-forge
pyparsing                 3.0.8              pyhd8ed1ab_0    conda-forge
pysocks                   1.7.1            py39hf3d152e_5    conda-forge
python                    3.9.2                hdb3f193_0
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python_abi                3.9                      2_cp39    conda-forge
pytz                      2022.1             pyhd8ed1ab_0    conda-forge
pywavelets                1.3.0            py39h7f8727e_0
pyyaml                    5.4.1            py39h3811e60_0    conda-forge
readline                  8.1.2                h7f8727e_1
requests                  2.27.1             pyhd8ed1ab_0    conda-forge
scikit-image              0.18.1           py39hde0f152_0    conda-forge
scipy                     1.7.3            py39hc147768_0
setuptools                61.2.0           py39h06a4308_0
six                       1.16.0             pyh6c4a22f_0    conda-forge
slicerator                1.1.0              pyhd8ed1ab_0    conda-forge
snappy                    1.1.8                he1b5a44_3    conda-forge
sortedcontainers          2.4.0              pyhd8ed1ab_0    conda-forge
sqlite                    3.38.2               hc218d9a_0
tblib                     1.7.0              pyhd8ed1ab_0    conda-forge
tifffile                  2021.7.2           pyhd8ed1ab_0    conda-forge
tk                        8.6.11               h1ccaba5_0
toolz                     0.11.2             pyhd8ed1ab_0    conda-forge
tornado                   6.1              py39h3811e60_1    conda-forge
typing_extensions         4.2.0              pyha770c72_1    conda-forge
tzdata                    2022a                hda174b7_0
urllib3                   1.26.9             pyhd8ed1ab_0    conda-forge
wheel                     0.37.1             pyhd3eb1b0_0
x264                      1!161.3030           h7f98852_1    conda-forge
xz                        5.2.5                h7b6447c_0
yaml                      0.2.5                h516909a_0    conda-forge
zarr                      2.11.3             pyhd8ed1ab_0    conda-forge
zfp                       0.5.5                h9c3ff4c_5    conda-forge
zict                      2.2.0              pyhd8ed1ab_0    conda-forge
zlib                      1.2.12               h7f8727e_2
zstd                      1.4.9                ha95c52a_0    conda-forge

I'm not sure if this is actually more related to Dask or PIMS, but since the traceback came from the ffmpeg.py module I figured I'd post it here.

@FirefoxMetzger
Copy link
Contributor

Hej @jmdelahanty 👋

A quick note on your setup. According to the exception, the ffmpeg plugin crashes in line 901, which is strange because the current version only has 700-ish lines of code. So I checked your version and noticed that you are still running on ImageIO 2.3.0:

imageio                   2.3.0                      py_1    conda-forge
imageio-ffmpeg            0.4.7              pyhd8ed1ab_0    conda-forge

Is there a specific reason for this? If not, a first step could be to try and upgrade ImageIO to something more recent (at the time of this writing we are on 2.19.0).

@jmdelahanty
Copy link
Author

jmdelahanty commented May 3, 2022 via email

@FirefoxMetzger
Copy link
Contributor

@jmdelahanty That sounds like a good start. v2.3 may have been installed because of some dependency; if that is the case, and you happen to figure out which package depends on this older version, please let me know. I will then go and check with the maintainers of that package if there is any bug we are unaware of, or if that version can be bumped.

@jmdelahanty
Copy link
Author

Doing a simple conda upgrade imageio did the trick it seems! I'm not sure why it installed such a low version to begin with.

imageio                   2.9.0              pyhd3eb1b0_0
imageio-ffmpeg            0.4.7              pyhd8ed1ab_0    conda-forge

Do you think that the fact that it's still 10 minor versions behind is a problem at all? I can try and see if there's other packages holding it up, but I'm not really sure how to check...

@FirefoxMetzger
Copy link
Contributor

Doing a simple conda upgrade imageio did the trick it seems! I'm not sure why it installed such a low version to begin with.

If you set up a fresh environment then one or more of your packages have pinned the version to 2.3.0 or below.

Do you think that the fact that it's still 10 minor versions behind is a problem at all?

In general, you should always try to upgrade to the latest version-major release, unless you have reasons not to. For a specific case, it depends. If you are using this as part of a public-facing service then yes, you should upgrade to avoid bugs and security vulnerabilities; otherwise, it's not too big an issue unless there is a bug that affects your use case.

I can try and see if there's other packages holding it up, but I'm not really sure how to check...

Unfortunately, I'm not sure if and how conda resolves dependencies since I almost exclusively use pip. I think @anjos might have an idea, since he maintains our conda feedstock.

@anjos Do you know if it is possible to visualize dependencies when installing with conda? More specifically, we have a downstream package that has pinned the ImageIO version to 2.3.0 and another that seems to pin it to 2.9.0 and we were wondering if it is possible to discover which package does so.

@anjos
Copy link
Contributor

anjos commented May 4, 2022

The current run requirements for imageio are defined on its recipe (https://github.com/conda-forge/imageio-feedstock/blob/main/recipe/meta.yaml):

  • python >=3 #for conda-forge, we are currently at python=3.8, so that does not matter that much
  • numpy >=1.20.0 #this comes from an old pin, that is now gone on setup.py
  • pillow >=8.3.2 #this comes from imageio's own setup.py

So, if the environment is set in such a way to conflict with those pins, then I guess a lower version will be tried until one is found that seems compatible.

The command conda info <pkg> can show direct dependence information of packages installed. From there, you'd have to iterate and find which packages are conflicting. Otherwise, you can try to do a conda install imageio=2.19 directly on the target environment, and see what would change or what conflicts if that is not possible.

@anjos
Copy link
Contributor

anjos commented May 4, 2022

@jmdelahanty -- PS.: I find it curious that your environment has a mixture of both conda-forge and defaults packages. IMO, you should avoid this, generally speaking. Use only conda-forge when conda-forge is required. Instead of miniconda, use mini-forge as base installer, which by default uses conda-forge instead of defaults. Or even better, use mamba-forge, available on the same page.

@FirefoxMetzger
Copy link
Contributor

Thanks for chiming in @anjos ! So conda does enforce dependencies / does dependency resolution. That's good to know!

@jmdelahanty On the same topic, I found a python package that can do a reverse dependency lookup, which is called pipdeptree. If you don't mind installing it, @jmdelahanty , would you mind running pipdeptree -p imageio -r in your environment and check which packages require ImageIO?

numpy >=1.20.0 #this comes from an old pin, that is now gone on setup.py

@anjos Yes, we no longer need numpy>=1.20 as of Imageio v2.17.0 because I refactored type annotations into separate files and created a fallback in case numpy.typing.ArrayLike doesn't exist. This means ImageIO>=2.17 should work nicely with every common version of numpy (I tested as far back as numpy 1.15.0).

@anjos
Copy link
Contributor

anjos commented May 4, 2022

Good to know, we should then remove it from the conda-forge recipe. I'll do this later today and issue a new build. This may fix observed issues, or at least relax the environment a bit further.

anjos added a commit to conda-forge/imageio-feedstock that referenced this issue May 4, 2022
@anjos
Copy link
Contributor

anjos commented May 4, 2022

OK, build 1 is available at the conda-forge channel, with the relaxed numpy pinning. You may want to try again and see if you still hit the same issue before starting to look elsewhere.

@jmdelahanty
Copy link
Author

Would you like me to try this new build @anjos? I can make a new environment, install the stuff I need, and see what happens? I'll also switch over to mini-forge and try out mamba-forge too! Thank you for the advice! I've heard of mamba/mamba-forge before but have never used it. I can update here what happens, will definitely do it in the next couple of days.

@anjos
Copy link
Contributor

anjos commented May 5, 2022

@jmdelahanty: It was just meant as debugging tips, in case you're looking for possible causes of issues. This is not the right thread for debugging conda setups. If you have issues with the conda package itself, I'd advise you open an issue at the repository for the package at https://github.com/conda-forge/imageio-feedstock/.

@jmdelahanty
Copy link
Author

Got it, thanks @anjos!

And for you @FirefoxMetzger, I ran the pipdeptree command as you asked and all it pointed out was this:

(facial_expression2) jdelahanty@cheetos:/scratch/snlkt_facial_expression$ pipdeptree -p imageio -r
imageio==2.9.0
  - scikit-image==0.18.1 [requires: imageio>=2.3.0]

@FirefoxMetzger
Copy link
Contributor

FirefoxMetzger commented May 6, 2022

@jmdelahanty Okay, in that case, it was probably just an outdated library on your system and nothing major is preventing you from using a more recent version of ImageIO.

To sum up all the discussion above, the solution to this issue seems to be "update ImageIO, because the problem is fixed in more recent versions".

@jmdelahanty Does that solve your problem? If so, I will go ahead and close this issue to stay organized, otherwise let me know what is missing.

@jmdelahanty
Copy link
Author

That's all correct! Thank you for your help and teaching me how to check these things.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants