Defaults and conda-forge incompatibilities

[isuruf]

This issue is created to remind that conda-forge and defaults have incompatibilities. To fix this, there are two ways.

1. conda-forge channel must have higher priority than defaults channel. To check, run conda config --get channels which gives you the channel priority order. conda config --add channels conda-forge would make conda-forge the highest priority channel.

2. When creating conda-forge packages, make sure the dependencies come from conda-forge. (There are few like libgcc that conda-forge is relying on defaults). If there's a dependency missing in conda-forge, add them as well because this missing dependency pulled in from defaults might be relying on the behaviour of another package in defaults which would conflict with conda-forge.

[jakirkham]

Thanks @isuruf. Maybe we should be making the docs more explicit on this point as well. What do you think?

[msarahan]

If you use conda-build 3's pin_compatible functionality, a lot of this inter-channel incompatibility goes away, because the more intelligent constraints mean that you won't actually be able to install incompatible software. Not that this is an immediate option, but more of a reminder that channel priority is not the end-all solution here. The better solution is for packages to capture enough compatibility information that mixing channels with confidence is actually possible.

[jakirkham]

Should we move this to the docs? Admittedly it would be great if this wasn't an issue, but it seems we are still a long ways off. Not to mention it remains unclear how we would actually test all combinations of packages between the two channels in any reasonable way.

[dhirschfeld]

FWIW I install a fairly large environment from both channels with channel_priority: false and other than conda-forge/zeromq-feedstock#32 & conda-forge/openmp-feedstock#8 I'm only seeing an issue with:

conda-forge::boost-cpp-1.67.0-vc14_0, defaults::libboost-1.65.1-he51fdeb_4

There are other packaging issues (ClobberErrors) with hdf4/5, mro-base/r-base and curl but I don't think the particular channel plays a part in those.

[dhirschfeld]

Actually, it seems there are pretty big problems mixing R from defaults and conda-forge

In conda/conda#7459 (comment) @mbargull suggested conda-forge might need to implement the _rmutex scheme to ensure compatibility. I'll raise an issue against r-base-feedstock to discuss this particular point.

[guyer]

Even when explicit about channels, I still have problems. In the following, on Mac OS X, numpy is downloaded from defaults and so pysparse is broken:

conda create -n pysparsereqs --channel conda-forge pysparse python=2.7
Solving environment: done

## Package Plan ##

  environment location: /Users/guyer/anaconda/envs/pysparsereqs

  added / updated specs: 
    - pysparse
    - python=2.7


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    numpy-1.15.2               |   py27h6a91979_1          48 KB
    setuptools-40.4.3          |           py27_0         586 KB  conda-forge
    numpy-base-1.15.2          |   py27h8a80b8c_1         4.0 MB
    ------------------------------------------------------------
                                           Total:         4.6 MB

The following NEW packages will be INSTALLED:

    blas:            1.0-mkl                           
    ca-certificates: 2018.8.24-ha4d7672_0   conda-forge
    certifi:         2018.8.24-py27_1001    conda-forge
    intel-openmp:    2019.0-118                        
    libgfortran:     3.0.1-h93005f0_2                  
    mkl:             2019.0-118                        
    mkl_fft:         1.0.6-py27_0           conda-forge
    mkl_random:      1.0.1-py27_0           conda-forge
    ncurses:         6.1-hfc679d8_1         conda-forge
    numpy:           1.15.2-py27h6a91979_1             
    numpy-base:      1.15.2-py27h8a80b8c_1             
    openssl:         1.0.2p-h470a237_0      conda-forge
    pip:             18.0-py27_1001         conda-forge
    pysparse:        1.3.dev-py27h7eb728f_0 conda-forge
    python:          2.7.15-h43f7c74_0      conda-forge
    readline:        7.0-haf1bffa_1         conda-forge
    setuptools:      40.4.3-py27_0          conda-forge
    sqlite:          3.25.2-hb1c47c0_0      conda-forge
    tk:              8.6.8-ha92aebf_0       conda-forge
    wheel:           0.32.0-py27_1000       conda-forge
    zlib:            1.2.11-h470a237_3      conda-forge

Proceed ([y]/n)? 


Downloading and Extracting Packages
numpy-1.15.2         | 48 KB     | ############################################################ | 100% 
setuptools-40.4.3    | 586 KB    | ############################################################ | 100% 
numpy-base-1.15.2    | 4.0 MB    | ############################################################ | 100% 
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use:
# > source activate pysparsereqs
#
# To deactivate an active environment, use:
# > source deactivate
#

$ source activate pysparsereqs
$ python -c "from pysparse.sparse import spmatrix"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/guyer/anaconda/envs/pysparsereqs/lib/python2.7/site-packages/pysparse/__init__.py", line 13, in <module>
    from sparse import spmatrix
  File "/Users/guyer/anaconda/envs/pysparsereqs/lib/python2.7/site-packages/pysparse/sparse/__init__.py", line 6, in <module>
    from pysparseMatrix import *
  File "/Users/guyer/anaconda/envs/pysparsereqs/lib/python2.7/site-packages/pysparse/sparse/pysparseMatrix.py", line 57, in <module>
    from pysparse.sparse import spmatrix
ImportError: dlopen(/Users/guyer/anaconda/envs/pysparsereqs/lib/python2.7/site-packages/pysparse/sparse/spmatrix.so, 2): Library not loaded: @rpath/libopenblas.dylib
  Referenced from: /Users/guyer/anaconda/envs/pysparsereqs/lib/python2.7/site-packages/pysparse/sparse/spmatrix.so
  Reason: image not found

In contrast, the Travis build and test get numpy from conda-forge:

 numpy:           1.9.3-py27_blas_openblashd3ea46f_206 conda-forge [blas_openblas]

I (think I) can pass --override-channels, but until recently this didn't work due to things like libgcc, as mentioned in the OP.

How do I ensure that the same, compatible, version of numpy is fetched as when built? I'm using {{ pin_compatible('numpy') }}.

[msarahan]

@guyer conda really does respect channel priority. Here you have features taking priority, though. Conda tries to minimize the number of features in an env. If you instead force a specific blas implementation (which is associated with a feature), you can get conda to avoid the MKL builds. Those are being chosen because they do not have a track_feature, whereas the openblas numpy stuff does.