h5py allows hdf5 dependencies that do not work together

[hugobuddel]

• I read the conda-forge documentation and could not find the solution for my problem there.

Issue:

The h5py dependency specification allows h5py-hdf5 combinations to be selected that do not work together. This causes problems when using mamba.

Mamba is a drop-in replacement for conda with a more powerful dependency solver. However, mamba (or any other solver) can only do its magic when packages conflict if they do not work together.

Mamba is a bit unpredictable, and I've not yet been able to reliably reproduce the problem. But it is technically possible to get the same problem with conda. For me these two packages were selected by mamba:

  h5py               conda-forge/linux-64::h5py-3.2.1-nompi_py39h98ba4bc_100
  hdf5               conda-forge/linux-64::hdf5-1.10.6-nompi_h3c11f04_101

Conda also happily installs these if explicitly asked (so conda also thinks they aren't conflicting), but the combination doesn't work:

$ conda create -n h5pytest "h5py=3.2.1=nompi_py39h98ba4bc_100" "hdf5=1.10.6=nompi_h3c11f04_101"
$ conda activate h5pytest
$ python -c "import h5py"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "[..]/miniconda3/envs/h5pytest/lib/python3.9/site-packages/h5py/__init__.py", line 33, in <module>
    from . import version
  File "[..]/miniconda3/envs/h5pytest/lib/python3.9/site-packages/h5py/version.py", line 15, in <module>
    from . import h5 as _h5
  File "h5py/h5.pyx", line 1, in init h5py.h5
ImportError: [..]/miniconda3/envs/h5pytest/lib/python3.9/site-packages/h5py/defs.cpython-39-x86_64-linux-gnu.so: undefined symbol: H5Pset_fapl_ros3

Is it possible to somehow make the dependencies of h5py more narrow such that it will conflict with any hdf5 package that it is not compatible with?

E.g. maybe it is possible to allow only exactly one version of hdf5 as a dependency for each h5py package? (A different one for each h5py package: one that works for that build.)

Or maybe the build string can be used to distinguish different feature sets? That is, expanding the mechanism that is currently used to distinguish mpi and nompi builds, to also prevent other conflicts.

(Note that this particular conflict will always be possible until one or both of the offending packages are actually removed from the repository.)

Vice-versa, is there perhaps a mechanism that solvers like mamba can use to distinguish compatible versions in another way? As in, is there maybe something mamba might do to prevent such collisions without relying on package specific workarounds? (I'm not involved with mamba; I like it because it is so fast.)

[hugobuddel]

It seems that ros3 support is handled in an identical way as mpi support, so maybe the build string can indeed be used to distinguish ros3 support as well.

(Note that a simple workaround for this problem is to hardcode specific versions in my own dependencies. So for me this is not urgent.)

[sejjbia]

(Note that a simple workaround for this problem is to hardcode specific versions in my own dependencies. So for me this is not urgent.)

Thank you hugobuddel.
Can you please share what you have done exactly for beginners like myself?

[hugobuddel]

The easiest you can do is probably put h5py<3.2 as a dependency, if that version is sufficient for you. However, you should also be able to explicitly list a good version of hdf5, e.g. hdf5-1.10.6-nompi_h6a2412b_1114.

But to honestly describe what I did exactly: for now I could get away with not using h5py at all, so I've personally postponed the problem.

I believe there might be only one specific problematic hdf5 package, hdf5-1.10.6-nompi_h3c11f04_101. If that is indeed the case, then the best solution would be to get the hdf5 conda package maintainers to remove that specific package.

[tschaka1904]

I've the same issue.

[tschaka1904]

17:01:20 h5py conda-forge/linux-64::h5py-3.2.1-nompi_py38h9915d05_100
17:01:20 hdf5 conda-forge/linux-64::hdf5-1.10.6-nompi_h3c11f04_101

[nvaytet]

Hi, any update on this? Thanks!