regression: ResolutionImpossible with tables 3.9.0 when installing on Ubuntu 20.04 Python 3.8

[astraw]

Hi,

With the release of tables 3.9.0 on Python 3.8 in Ubuntu 20.04, a scheduled CI build of flydra_core is failing with ResolutionImpossible. The ResolutionImpossible is that oldest-supported-numpy 0.1 depends on numpy==1.17.3 and blosc2 2.2.9 depends on numpy>=1.20.3. I am also seeing this on in-house CI builds, so it's not a single isolated incident. The fundamental issue may not be in tables but the proximal change introducing breakage does seem to be the new tables release. I have not dug further into the issue.

2023-10-05T23:29:06.5690421Z ##[group]Run python -m pip install ./flydra_core
2023-10-05T23:29:06.5690926Z �[36;1mpython -m pip install ./flydra_core�[0m
2023-10-05T23:29:06.5749984Z shell: /usr/bin/bash -e {0}
2023-10-05T23:29:06.5750303Z env:
2023-10-05T23:29:06.5750693Z   pythonLocation: /opt/hostedtoolcache/Python/3.8.18/x64
2023-10-05T23:29:06.5751176Z   PKG_CONFIG_PATH: /opt/hostedtoolcache/Python/3.8.18/x64/lib/pkgconfig
2023-10-05T23:29:06.5751619Z   Python_ROOT_DIR: /opt/hostedtoolcache/Python/3.8.18/x64
2023-10-05T23:29:06.5751992Z   Python2_ROOT_DIR: /opt/hostedtoolcache/Python/3.8.18/x64
2023-10-05T23:29:06.5752430Z   Python3_ROOT_DIR: /opt/hostedtoolcache/Python/3.8.18/x64
2023-10-05T23:29:06.5752884Z   LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.8.18/x64/lib
2023-10-05T23:29:06.5866861Z ##[endgroup]
2023-10-05T23:29:07.0893369Z Processing ./flydra_core
2023-10-05T23:29:07.0937038Z   Installing build dependencies: started
2023-10-05T23:29:12.9899257Z   Installing build dependencies: finished with status 'done'
2023-10-05T23:29:12.9912107Z   Getting requirements to build wheel: started
2023-10-05T23:29:15.3571177Z   Getting requirements to build wheel: finished with status 'done'
2023-10-05T23:29:15.3589456Z   Installing backend dependencies: started
2023-10-05T23:29:16.2784864Z   Installing backend dependencies: finished with status 'done'
2023-10-05T23:29:16.2801555Z   Preparing metadata (pyproject.toml): started
2023-10-05T23:29:16.8675779Z   Preparing metadata (pyproject.toml): finished with status 'done'
2023-10-05T23:29:17.3360628Z Collecting numpy>=1.19 (from flydra-core==0.7.16)
2023-10-05T23:29:17.3362590Z   Obtaining dependency information for numpy>=1.19 from https://files.pythonhosted.org/packages/98/5d/5738903efe0ecb73e51eb44feafba32bdba2081263d40c5043568ff60faf/numpy-1.24.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata
2023-10-05T23:29:17.3389220Z   Using cached numpy-1.24.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (5.6 kB)
2023-10-05T23:29:17.4142640Z Collecting tables (from flydra-core==0.7.16)
2023-10-05T23:29:17.4688332Z   Downloading tables-3.9.0.tar.gz (4.7 MB)
2023-10-05T23:29:17.5884397Z      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.7/4.7 MB 40.5 MB/s eta 0:00:00
2023-10-05T23:29:17.9094177Z   Installing build dependencies: started
2023-10-05T23:29:32.0566764Z   Installing build dependencies: finished with status 'error'
2023-10-05T23:29:32.0701004Z   error: subprocess-exited-with-error
2023-10-05T23:29:32.0701529Z   
2023-10-05T23:29:32.0702739Z   × pip subprocess to install build dependencies did not run successfully.
2023-10-05T23:29:32.0703207Z   │ exit code: 1
2023-10-05T23:29:32.0703515Z   ╰─> [67 lines of output]
2023-10-05T23:29:32.0703790Z       Collecting setuptools>=61.0.0
2023-10-05T23:29:32.0704654Z         Obtaining dependency information for setuptools>=61.0.0 from https://files.pythonhosted.org/packages/bb/26/7945080113158354380a12ce26873dd6c1ebd88d47f5bc24e2c5bb38c16a/setuptools-68.2.2-py3-none-any.whl.metadata
2023-10-05T23:29:32.0705413Z         Using cached setuptools-68.2.2-py3-none-any.whl.metadata (6.3 kB)
2023-10-05T23:29:32.0705761Z       Collecting wheel
2023-10-05T23:29:32.0706542Z         Obtaining dependency information for wheel from https://files.pythonhosted.org/packages/b8/8b/31273bf66016be6ad22bb7345c37ff350276cfd46e389a0c2ac5da9d9073/wheel-0.41.2-py3-none-any.whl.metadata
2023-10-05T23:29:32.0707258Z         Using cached wheel-0.41.2-py3-none-any.whl.metadata (2.2 kB)
2023-10-05T23:29:32.0707673Z       Collecting oldest-supported-numpy
2023-10-05T23:29:32.0709062Z         Obtaining dependency information for oldest-supported-numpy from https://files.pythonhosted.org/packages/94/9a/756fef9346e5ca2289cb70d73990b4c9f25446a885c1186cfb93a85e7da0/oldest_supported_numpy-2023.8.3-py3-none-any.whl.metadata
2023-10-05T23:29:32.0709900Z         Downloading oldest_supported_numpy-2023.8.3-py3-none-any.whl.metadata (9.5 kB)
2023-10-05T23:29:32.0710247Z       Collecting packaging
2023-10-05T23:29:32.0711284Z         Obtaining dependency information for packaging from https://files.pythonhosted.org/packages/ec/1a/610693ac4ee14fcdf2d9bf3c493370e4f2ef7ae2e19217d7a237ff42367d/packaging-23.2-py3-none-any.whl.metadata
2023-10-05T23:29:32.0712051Z         Downloading packaging-23.2-py3-none-any.whl.metadata (3.2 kB)
2023-10-05T23:29:32.0712450Z       Collecting py-cpuinfo
2023-10-05T23:29:32.0712848Z         Downloading py_cpuinfo-9.0.0-py3-none-any.whl (22 kB)
2023-10-05T23:29:32.0713169Z       Collecting Cython>=0.29.32
2023-10-05T23:29:32.0714058Z         Obtaining dependency information for Cython>=0.29.32 from https://files.pythonhosted.org/packages/e1/c7/35f368d0e9140939df898e3befb2931bbe6d7b0ca69598b0ac20ee38e978/Cython-3.0.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata
2023-10-05T23:29:32.0714894Z         Using cached Cython-3.0.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.2 kB)
2023-10-05T23:29:32.0715247Z       Collecting blosc2>=2.2.8
2023-10-05T23:29:32.0715615Z         Downloading blosc2-2.2.9.tar.gz (4.7 MB)
2023-10-05T23:29:32.0716114Z            ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.7/4.7 MB 42.9 MB/s eta 0:00:00
2023-10-05T23:29:32.0716445Z         Installing build dependencies: started
2023-10-05T23:29:32.0716889Z         Installing build dependencies: finished with status 'done'
2023-10-05T23:29:32.0717252Z         Getting requirements to build wheel: started
2023-10-05T23:29:32.0717720Z         Getting requirements to build wheel: finished with status 'done'
2023-10-05T23:29:32.0718085Z         Preparing metadata (pyproject.toml): started
2023-10-05T23:29:32.0718827Z         Preparing metadata (pyproject.toml): finished with status 'done'
2023-10-05T23:29:32.0719496Z       Collecting numpy==1.17.3 (from oldest-supported-numpy)
2023-10-05T23:29:32.0720060Z         Using cached numpy-1.17.3-cp38-cp38-manylinux1_x86_64.whl (20.5 MB)
2023-10-05T23:29:32.0720693Z       INFO: pip is looking at multiple versions of blosc2 to determine which version is compatible with other requirements. This could take a while.
2023-10-05T23:29:32.0721304Z       Collecting oldest-supported-numpy
2023-10-05T23:29:32.0721862Z         Downloading oldest_supported_numpy-2022.11.19-py3-none-any.whl (4.9 kB)
2023-10-05T23:29:32.0722948Z         Downloading oldest_supported_numpy-2022.8.16-py3-none-any.whl (3.9 kB)
2023-10-05T23:29:32.0724329Z         Downloading oldest_supported_numpy-2022.5.28-py3-none-any.whl (3.9 kB)
2023-10-05T23:29:32.0724857Z         Downloading oldest_supported_numpy-2022.5.27-py3-none-any.whl (3.9 kB)
2023-10-05T23:29:32.0725386Z         Downloading oldest_supported_numpy-2022.4.18-py3-none-any.whl (3.9 kB)
2023-10-05T23:29:32.0725897Z         Downloading oldest_supported_numpy-2022.4.10-py3-none-any.whl (3.9 kB)
2023-10-05T23:29:32.0726405Z         Downloading oldest_supported_numpy-2022.4.8-py3-none-any.whl (3.9 kB)
2023-10-05T23:29:32.0726962Z       INFO: pip is still looking at multiple versions of blosc2 to determine which version is compatible with other requirements. This could take a while.
2023-10-05T23:29:32.0727597Z         Downloading oldest_supported_numpy-2022.3.27-py3-none-any.whl (3.9 kB)
2023-10-05T23:29:32.0728098Z         Downloading oldest_supported_numpy-2022.1.30-py3-none-any.whl (3.9 kB)
2023-10-05T23:29:32.0728610Z         Downloading oldest_supported_numpy-0.15-py3-none-any.whl (3.8 kB)
2023-10-05T23:29:32.0729547Z         Downloading oldest_supported_numpy-0.14-py3-none-any.whl (3.8 kB)
2023-10-05T23:29:32.0730047Z         Downloading oldest_supported_numpy-0.13-py3-none-any.whl (3.8 kB)
2023-10-05T23:29:32.0730971Z       INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. See https://pip.pypa.io/warnings/backtracking for guidance. If you want to abort this run, press Ctrl + C.
2023-10-05T23:29:32.0731762Z         Downloading oldest_supported_numpy-0.12-py3-none-any.whl (3.8 kB)
2023-10-05T23:29:32.0732397Z         Downloading oldest_supported_numpy-0.11-py3-none-any.whl (3.8 kB)
2023-10-05T23:29:32.0732908Z         Downloading oldest_supported_numpy-0.10-py3-none-any.whl (3.8 kB)
2023-10-05T23:29:32.0733400Z         Downloading oldest_supported_numpy-0.9-py3-none-any.whl (3.8 kB)
2023-10-05T23:29:32.0733908Z         Downloading oldest_supported_numpy-0.8-py3-none-any.whl (3.7 kB)
2023-10-05T23:29:32.0734408Z         Downloading oldest_supported_numpy-0.7-py3-none-any.whl (3.6 kB)
2023-10-05T23:29:32.0734899Z         Downloading oldest_supported_numpy-0.6-py3-none-any.whl (3.5 kB)
2023-10-05T23:29:32.0735399Z         Downloading oldest_supported_numpy-0.5-py3-none-any.whl (3.5 kB)
2023-10-05T23:29:32.0735906Z         Downloading oldest_supported_numpy-0.4-py3-none-any.whl (3.5 kB)
2023-10-05T23:29:32.0736415Z         Downloading oldest_supported_numpy-0.3-py3-none-any.whl (3.5 kB)
2023-10-05T23:29:32.0736920Z         Downloading oldest_supported_numpy-0.2-py2.py3-none-any.whl (2.5 kB)
2023-10-05T23:29:32.0737453Z         Downloading oldest_supported_numpy-0.1-py2.py3-none-any.whl (2.2 kB)
2023-10-05T23:29:32.0738151Z       ERROR: Cannot install blosc2==2.2.9 and oldest-supported-numpy==0.1 because these package versions have conflicting dependencies.
2023-10-05T23:29:32.0738539Z       
2023-10-05T23:29:32.0738784Z       The conflict is caused by:
2023-10-05T23:29:32.0739345Z           oldest-supported-numpy 0.1 depends on numpy==1.17.3; python_version >= "3.8" and platform_system != "AIX"
2023-10-05T23:29:32.0739766Z           blosc2 2.2.9 depends on numpy>=1.20.3
2023-10-05T23:29:32.0740010Z       
2023-10-05T23:29:32.0740262Z       To fix this you could try to:
2023-10-05T23:29:32.0740705Z       1. loosen the range of package versions you've specified
2023-10-05T23:29:32.0741127Z       2. remove package versions to allow pip attempt to solve the dependency conflict
2023-10-05T23:29:32.0741462Z       
2023-10-05T23:29:32.0742101Z       ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts
2023-10-05T23:29:32.0742575Z       [end of output]
2023-10-05T23:29:32.0742793Z   
2023-10-05T23:29:32.0743132Z   note: This error originates from a subprocess, and is likely not a problem with pip.
2023-10-05T23:29:32.0767708Z error: subprocess-exited-with-error
2023-10-05T23:29:32.0767934Z 
2023-10-05T23:29:32.0768282Z × pip subprocess to install build dependencies did not run successfully.
2023-10-05T23:29:32.0768678Z │ exit code: 1
2023-10-05T23:29:32.0769158Z ╰─> See above for output.
2023-10-05T23:29:32.0769323Z 
2023-10-05T23:29:32.0769534Z note: This error originates from a subprocess, and is likely not a problem with pip.
2023-10-05T23:29:32.2942389Z ##[error]Process completed with exit code 1.

[avalentino]

python-blosc2 has been recently updated to support Python 3.8 (again), but, apparently, we still have problems with dependencies.

@astraw a workaround could be to replace oldest-supported-numpy with numpy in pyproject.toml build-system.requires.

[astraw]

numpy is already in pyproject.toml build-system.requires: https://github.com/strawlab/flydra/blob/24ba56ad47b312c3edba3e43cea96e7c169257b8/flydra_core/pyproject.toml#L32

[danodonovan]

PyTables CI doesn't seem to be testing python 38 - should we expect 38 to be unsupported? @astraw Though suboptimal, we've started pinning the max value of PyTables to get around - for this issue tables<3.9.0

[astraw]

The readme banner says python 3.8.

[avalentino]

numpy is already in pyproject.toml build-system.requires: https://github.com/strawlab/flydra/blob/24ba56ad47b312c3edba3e43cea96e7c169257b8/flydra_core/pyproject.toml#L32

Sorry, I was referring to the PyTables's one

[FrancescAlted]

It is true that we are not actively testing Python 3.8 anymore because it is too heavy for the CI workers (with 4 versions we are near the limit of 6 hours for building wheels for aarch64 on Linux). Having said that, we did not want to remove the Python 3.8 label on the supported versions because we are not using any specific feature of Python>=3.9, so Python 3.8 support should still be ok.

Having said that, you should still be able to use Python 3.8 to compile PyTables as long as you use pip install oldest-supported-numpy (which should install numpy 1.21.0) instead of pip install numpy. Please try that and tell us how it goes.

[FrancescAlted]

@astraw for your particular setup, I think you should replace numpy by oldest-supported-numpy in https://github.com/strawlab/flydra/blob/main/flydra_core/pyproject.toml#L32 . See why it is recommended to use oldest-supported-numpy when building packages.

[astraw]

@FrancescAlted thanks for the tip and nice to "see" you again after many(!) years. Unfortunately this failed in the same way. https://github.com/strawlab/flydra/actions/runs/6430836096/job/17462553426

[FrancescAlted]

It is true that we are not actively testing Python 3.8 anymore because it is too heavy for the CI workers (with 4 versions we are near the limit of 6 hours for building wheels for aarch64 on Linux). Having said that, we did not want to remove the Python 3.8 label on the supported versions because we are not using any specific feature of Python>=3.9, so Python 3.8 support should still be ok.

Ugh, we actually introduced a Python 3.9 feature, so unfortunately the test suite needs Python 3.9 to run. This is unfortunate, and my vote would be to remove support for Python 3.8 in a future PyTables 3.9.1.

[FrancescAlted]

@FrancescAlted thanks for the tip and nice to "see" you again after many(!) years. Unfortunately this failed in the same way. https://github.com/strawlab/flydra/actions/runs/6430836096/job/17462553426

Yeah, nice to talk to you after such a long time!

Uh, I still see that flydra is still trying to install numpy, not oldest-supported-numpy:

Collecting numpy>=1.19 (from flydra-core==0.7.16)
  Obtaining dependency information for numpy>=1.19 from https://files.pythonhosted.org/packages/98/5d/5738903efe0ecb73e51eb44feafba32bdba2081263d40c5043568ff60faf/numpy-1.24.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata

At any rate, as per my previous comment, I don't think we can go back and provide support for Python 3.8 in PyTables>=3.9. Better use tables<3.9 in your setup if you need Python 3.8 (at least for now).

[astraw]

OK, yes, using tables < 3.9 will be what we have to do as long as we support Ubuntu 20.04, which uses Python 3.8. I'm happy if you close this issue now, but I will let you do it in case there are action points from your side.

[FrancescAlted]

A new hot fix release 3.9.1 is out just to convey that Python 3.8 is not supported anymore.

[ivilata]

Ugh, we actually introduced a Python 3.9 feature, so unfortunately the test suite needs Python 3.9 to run. This is unfortunate, and my vote would be to remove support for Python 3.8 in a future PyTables 3.9.1.

My bad, sorry! ☹️ I used importlib.resources.files to avoid an extra runtime dependency on setuptools, but I oversaw the Python >= 3.9 dependency of that function (instead of the module's dependency on Python >= 3.7).

[ivilata]

For anybody following this issue, as a summary: since PyTables 3.9.1 has been published, which explicitly depends on Python >= 3.9, and since PyTables 3.9.0 has been yanked from PyPI (see #1065), there should be no need to depend on tables<3.9 in Python 3.8 environments, and you may go back to depend on plain tables (pip will figure out the right version to install).