pyBigWig dependency update

[joachimwolff]

pyBigWig lists as dependency numpy version 1.12, however, numpy version 1.15 is the newest one and 1.14 is missing too.
Furthermore, a version for Python 3.7 is missing. Is there any way to enforce an update to the mentioned versions (assuming nothing will break and would need a fix of pybigwig from @dpryan79)?

Thanks a lot.

[dpryan79]

pyBigWig doesn't depend on any particular version of numpy, it's just that when you build it you need to then not upgrade to a non-ABI compatible library. bioconda and conda-forge build against a couple different versions of numpy and nothing else, so that's what pyBigWig and other packages that need pinning get pinned against.

I don't think anything in bioconda is building against python 3.7 yet. It's in the conda-forge-pinning repository, but requires an environment variable be set appropriately to be used. Presumably this change is then waiting for everything in conda-forge to be built, one would need to ask /core.

[joachimwolff]

Thanks for your feedback Devon. Good to hear that no changes for a newer numpy version in the code need to be done.
Can anyone tell me what I need to do to build for a newer numpy version in bioconda?

[bgruening]

pyBigWig lists as dependency numpy version 1.12

Can you explain this a little bit more? Where is this listed?

conda create -n pbw pybigwig gives me numpy 1.14 and I can get 1.15 with conda create -n pbw pybigwig numpy=1.15

[joachimwolff]

Yesterday I did:
conda install --file requirements.txt -c bioconda -c conda-forge

and got

UnsatisfiableError: The following specifications were found to be in conflict:
  - numpy  1.14.*
  - pybigwig  0.3.* -> numpy 1.12*

Requirements.txt:

numpy = 1.14.*
scipy = 1.1.*
matplotlib = 2.2*
pysam = 0.14.*
intervaltree = 2.1.*
biopython = 1.72.*
pytables = 3.4.*
pandas = 0.23.*
pybigwig = 0.3.*
cooler = 0.7.10
jinja2 = 2.10
six = 1.11.*
future = 0.16.*
unidecode = 1.0.*
hic2cool = 0.4.*
scikit-learn = 0.19.*

Today I am doing the same, and there is no conflict and it can be installed. I am a bit confused about what is going on.

[dpryan79]

Sounds like the universe is trying to remind you what weekends are :)

[dpryan79]

@joachimwolff Can this be closed or is there still a problem?

[joachimwolff]

Hi,

there is still some issue. If you do conda create --name foo hicexplorer -c bioconda -c numpy a run of HiCExplorer quits with

RuntimeError: module compiled against API version 0xc but this version of numpy is 0xb
Traceback (most recent call last):
  File "/home/wolffj/miniconda3/envs/foo_bug/bin/hicPlotMatrix", line 4, in <module>
    from hicexplorer.hicPlotMatrix import main
  File "/home/wolffj/miniconda3/envs/foo_bug/lib/python2.7/site-packages/hicexplorer/hicPlotMatrix.py", line 16, in <module>
    import pyBigWig
ImportError: numpy.core.multiarray failed to import

Something to resolve the dependencies is not working as it was in the past.

[dpryan79]

@bgruening This points to an issue with pin_compatible('numpy'). Currently recipes are getting built against numpy 1.15 but listed in conda info with dependencies numpy >=1.9.3,<2.0a0. But since numpy 1.13 and 1.15 aren't compatible (see above) that dependency is wrong. Presumably this is something that needs to be fixed on conda-forge-pinning. Alternatively, there needs to be a different mechanism for pinning things that use the numpy C API (we can go back to numpy 1.15.*, but that's overly restrictive), since perhaps this doesn't affect pure python packages (then again, what's the point of pinning them then?).

FYI, current packages pinned to numpy 1.15 are only compatible with versions back to 1.14 (1.13 is only API compatible with itself): https://github.com/numpy/numpy/blob/464f79eb1d05bf938d16b49da1c39a4e02506fa3/numpy/core/setup_common.py#L35-L44

[dpryan79]

@joachimwolff The immediate solution to your problem is to not specify 1.13.* in the hicexplorer recipe. You have a pure python package, that's completely unnecessary.

[joachimwolff]

Thanks Devon, this is fixing it.

The reason I am pinning the version numbers is that some packages (especially numpy, matplotlib and cooler) have the "talent" to change their behaviour from version to version (numpy: slightly different results for floating numbers, matplotlib: different number of pixel for the same images and colours, cooler: long story). If they are fixed to a specific version can guarantee it is working with these versions and I have less support effort (at least in theory :) )

[dpryan79]

I think theory and practice will differ significantly. In practice I expect you'll have more issues from downgrading dependencies from other packages and thereby breaking them.