Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

channel priority sometimes messed up #6065

Closed
MaxBo opened this issue Oct 3, 2017 · 11 comments
Closed

channel priority sometimes messed up #6065

MaxBo opened this issue Oct 3, 2017 · 11 comments
Labels
locked [bot] locked due to inactivity stale::closed [bot] closed after being marked as stale stale [bot] marked as stale due to inactivity

Comments

@MaxBo
Copy link

MaxBo commented Oct 3, 2017

Installing

cythonarrays
with

(t35) C:\>conda --version
conda 4.3.27

from channel MaxBo

failes on my local computer and on the ci-server for python 3.5 on windows and linux.
Installing for python 3.6 works fine.
There seems to be a problem with the conda-channels on python 3.5, but not on python 3.6.

My channels are in the following priority:

MaxBo
conda-force
defaults

set by

conda config --remove channels defaults
conda config --add channels defaults
conda config --add channels conda-forge
conda config --add channels MaxBo

I figured out, that on python 3.5,
conda install cythonarrays installes the followning dependencies:

(t35) C:\>conda install cythonarrays
Fetching package metadata .................
Solving package specifications: .

Package plan for installation in environment C:\Anaconda\envs\t35:

The following NEW packages will be INSTALLED:

    colorama:             0.3.9-py35_0         conda-forge
    cython:               0.27.1-py35_0        conda-forge
    cythonarrays:         1.3.5-py35hb580c6e_0 MaxBo
    cythoninstallhelpers: 1.1.3-py35h3a5bacc_0 MaxBo
    mkl:                  2017.0.3-0
    numpy:                1.13.1-py35_0
    pandas:               0.20.3-py35_1        conda-forge
    py:                   1.4.34-py35_0        conda-forge
    pytest:               3.2.2-py35_1         conda-forge
    pytest-runner:        2.11.1-py35_0        conda-forge
    python-dateutil:      2.6.1-py35_0         conda-forge
    pytz:                 2017.2-py35_0        conda-forge
    six:                  1.11.0-py35_1        conda-forge
    xarray:               0.9.6-py35_0

Proceed ([y]/n)?

So the package xarray is insalled from the defaults-channel and not from conda-forge.
The problem is, that xarray from the defaults-channel does not install several dependencies. I first guessed, that the xarray-package is missing on conda-forge for python 3.5.
But when i try to install xarray directly, it is taken from conda-forge and installes the required dependencies:

(t35) C:\>conda install xarray
Fetching package metadata .................
Solving package specifications: .

Package plan for installation in environment C:\Anaconda\envs\t35:

The following NEW packages will be INSTALLED:

    bokeh:            0.12.9-py35_0      conda-forge
    bottleneck:       1.2.1-py35_1       conda-forge
    click:            6.7-py35_0         conda-forge
    cloudpickle:      0.4.0-py35_0       conda-forge
    curl:             7.49.1-vc14_2      conda-forge [vc14]
    dask:             0.15.3-py_0        conda-forge
    dask-core:        0.15.3-py_0        conda-forge
    distributed:      1.19.1-py35_0      conda-forge
    h5netcdf:         0.4.2-py_0         conda-forge
    h5py:             2.7.1-py35_1       conda-forge
    hdf4:             4.2.12-vc14_0      conda-forge [vc14]
    hdf5:             1.8.18-vc14_0      conda-forge [vc14]
    heapdict:         1.0.0-py35_0       conda-forge
    jinja2:           2.9.6-py35_0       conda-forge
    jpeg:             9b-vc14_1          conda-forge [vc14]
    libnetcdf:        4.4.1.1-vc14_5     conda-forge [vc14]
    locket:           0.2.0-py35_1       conda-forge
    markupsafe:       1.0-py35_0         conda-forge
    mkl:              2017.0.3-0
    msgpack-python:   0.4.8-py35_0       conda-forge
    netcdf4:          1.3.0-py35_0       conda-forge
    numpy:            1.13.1-py35_0
    pandas:           0.20.3-py35_1      conda-forge
    partd:            0.3.8-py35_0       conda-forge
    psutil:           5.3.1-py35_0       conda-forge
    python-dateutil:  2.6.1-py35_0       conda-forge
    pytz:             2017.2-py35_0      conda-forge
    pyyaml:           3.12-py35_1        conda-forge
    scipy:            0.19.1-np113py35_0
    six:              1.11.0-py35_1      conda-forge
    sortedcontainers: 1.5.7-py35_0       conda-forge
    tblib:            1.3.2-py35_0       conda-forge
    toolz:            0.8.2-py_2         conda-forge
    tornado:          4.5.2-py35_0       conda-forge
    vc:               14-0               conda-forge
    xarray:           0.9.6-py35_0       conda-forge
    yaml:             0.1.6-vc14_0       conda-forge [vc14]
    zict:             0.1.3-py_0         conda-forge
    zlib:             1.2.11-vc14_0      conda-forge [vc14]

Proceed ([y]/n)?

In python 3.6, installing cythonarrays works fine, because xarray is taken from the right channel:

(t36) C:\>conda install cythonarrays
Fetching package metadata .................
Solving package specifications: .

Package plan for installation in environment C:\Anaconda\envs\t36:

The following NEW packages will be INSTALLED:

    bokeh:                0.12.9-py36_0        conda-forge
    bottleneck:           1.2.1-py36_1         conda-forge
    click:                6.7-py36_0           conda-forge
    cloudpickle:          0.4.0-py36_0         conda-forge
    colorama:             0.3.9-py36_0         conda-forge
    curl:                 7.49.1-vc14_2        conda-forge [vc14]
    cython:               0.27.1-py36_0        conda-forge
    cythonarrays:         1.3.5-py36he83401b_0 MaxBo
    cythoninstallhelpers: 1.1.3-py36hddc3171_0 MaxBo
    dask:                 0.15.3-py_0          conda-forge
    dask-core:            0.15.3-py_0          conda-forge
    distributed:          1.19.1-py36_0        conda-forge
    h5netcdf:             0.4.2-py_0           conda-forge
    h5py:                 2.7.1-py36_1         conda-forge
    hdf4:                 4.2.12-vc14_0        conda-forge [vc14]
    hdf5:                 1.8.18-vc14_0        conda-forge [vc14]
    heapdict:             1.0.0-py36_0         conda-forge
    jinja2:               2.9.6-py36_0         conda-forge
    jpeg:                 9b-vc14_1            conda-forge [vc14]
    libnetcdf:            4.4.1.1-vc14_5       conda-forge [vc14]
    locket:               0.2.0-py36_1         conda-forge
    markupsafe:           1.0-py36_0           conda-forge
    mkl:                  2017.0.3-0
    msgpack-python:       0.4.8-py36_0         conda-forge
    netcdf4:              1.3.0-py36_0         conda-forge
    numpy:                1.13.1-py36_0
    pandas:               0.20.3-py36_1        conda-forge
    partd:                0.3.8-py36_0         conda-forge
    psutil:               5.3.1-py36_0         conda-forge
    py:                   1.4.34-py36_0        conda-forge
    pytest:               3.2.2-py36_1         conda-forge
    pytest-runner:        2.11.1-py36_0        conda-forge
    python-dateutil:      2.6.1-py36_0         conda-forge
    pytz:                 2017.2-py36_0        conda-forge
    pyyaml:               3.12-py36_1          conda-forge
    scipy:                0.19.1-np113py36_0
    six:                  1.11.0-py36_1        conda-forge
    sortedcontainers:     1.5.7-py36_0         conda-forge
    tblib:                1.3.2-py36_0         conda-forge
    toolz:                0.8.2-py_2           conda-forge
    tornado:              4.5.2-py36_0         conda-forge
    vc:                   14-0                 conda-forge
    xarray:               0.9.6-py36_0         conda-forge
    yaml:                 0.1.6-vc14_0         conda-forge [vc14]
    zict:                 0.1.3-py_0           conda-forge
    zlib:                 1.2.11-vc14_0        conda-forge [vc14]

Proceed ([y]/n)?

I checked the channel priority, ensuring that the channel priority should take place with

conda config --set channel_priority true

I have no idea, why xarray from the defaults-channel is preferred over conda-forge on python 3.5, but not on 3.6. Am i making a mistake or what could be the reason for this behaviour?

Thanks for any support,
Max

@kalefranz
Copy link
Contributor

kalefranz commented Oct 5, 2017

What do you get for

conda create --dry-run -n t35-2 python=3.5 cythonarrays

@MaxBo
Copy link
Author

MaxBo commented Oct 5, 2017

For python 3.5 i get:

C:\>conda create --dry-run -n t35a python=3.5 cythonarrays
Fetching package metadata ...................
Solving package specifications: .

Package plan for installation in environment C:\Anaconda\envs\t35a:

The following NEW packages will be INSTALLED:

    certifi:              2017.7.27.1-py35_0    conda-forge
    colorama:             0.3.9-py35_0          conda-forge
    cython:               0.27.1-py35_0         conda-forge
    cythonarrays:         1.3.5-py35hb580c6e_0  MaxBo
    cythoninstallhelpers: 1.1.3-py35h3a5bacc_0  MaxBo
    icc_rt:               2017.0.4-h97af966_0
    intel-openmp:         2018.0.0-hcd89f80_7
    mkl:                  2018.0.0-h36b65af_4
    numpy:                1.13.1-py35h9801081_2
    pandas:               0.20.3-py35_1         conda-forge
    pip:                  9.0.1-py35_0          conda-forge
    py:                   1.4.34-py35_0         conda-forge
    pytest:               3.2.3-py35_0          conda-forge
    pytest-runner:        2.11.1-py35_0         conda-forge
    python:               3.5.4-0               conda-forge
    python-dateutil:      2.6.1-py35_0          conda-forge
    pytz:                 2017.2-py35_0         conda-forge
    setuptools:           36.3.0-py35_0         conda-forge
    six:                  1.11.0-py35_1         conda-forge
    vc:                   14-0                  conda-forge
    vs2015_runtime:       14.0.25420-0          conda-forge
    wheel:                0.30.0-py_1           conda-forge
    wincertstore:         0.2-py35_0            conda-forge
    xarray:               0.9.6-py35_0

For python 3.6 its correct:

C:\>conda create --dry-run -n t35a python=3.6 cythonarrays
Fetching package metadata ...................
Solving package specifications: .

Package plan for installation in environment C:\Anaconda\envs\t35a:

The following NEW packages will be INSTALLED:

    bokeh:                0.12.9-py36_0         conda-forge
    bottleneck:           1.2.1-py36_1          conda-forge
    certifi:              2017.7.27.1-py36_0    conda-forge
    click:                6.7-py36_0            conda-forge
    cloudpickle:          0.4.0-py36_0          conda-forge
    colorama:             0.3.9-py36_0          conda-forge
    curl:                 7.49.1-vc14_2         conda-forge [vc14]
    cython:               0.27.1-py36_0         conda-forge
    cythonarrays:         1.3.5-py36he83401b_0  MaxBo
    cythoninstallhelpers: 1.1.3-py36hddc3171_0  MaxBo
    dask:                 0.15.3-py_0           conda-forge
    dask-core:            0.15.3-py_0           conda-forge
    distributed:          1.19.1-py36_0         conda-forge
    h5netcdf:             0.4.2-py_0            conda-forge
    h5py:                 2.7.1-py36_1          conda-forge
    hdf4:                 4.2.12-vc14_0         conda-forge [vc14]
    hdf5:                 1.8.18-vc14_0         conda-forge [vc14]
    heapdict:             1.0.0-py36_0          conda-forge
    icc_rt:               2017.0.4-h97af966_0
    intel-openmp:         2018.0.0-hcd89f80_7
    jinja2:               2.9.6-py36_0          conda-forge
    jpeg:                 9b-vc14_1             conda-forge [vc14]
    libnetcdf:            4.4.1.1-vc14_5        conda-forge [vc14]
    locket:               0.2.0-py36_1          conda-forge
    markupsafe:           1.0-py36_0            conda-forge
    mkl:                  2018.0.0-h36b65af_4
    msgpack-python:       0.4.8-py36_0          conda-forge
    netcdf4:              1.3.0-py36_0          conda-forge
    numpy:                1.13.1-py36haf1bc54_2
    pandas:               0.20.3-py36_1         conda-forge
    partd:                0.3.8-py36_0          conda-forge
    pip:                  9.0.1-py36_0          conda-forge
    psutil:               5.3.1-py36_0          conda-forge
    py:                   1.4.34-py36_0         conda-forge
    pytest:               3.2.3-py36_0          conda-forge
    pytest-runner:        2.11.1-py36_0         conda-forge
    python:               3.6.3-0               conda-forge
    python-dateutil:      2.6.1-py36_0          conda-forge
    pytz:                 2017.2-py36_0         conda-forge
    pyyaml:               3.12-py36_1           conda-forge
    scipy:                0.19.1-py36h7565378_3
    setuptools:           36.3.0-py36_0         conda-forge
    six:                  1.11.0-py36_1         conda-forge
    sortedcontainers:     1.5.7-py36_0          conda-forge
    tblib:                1.3.2-py36_0          conda-forge
    toolz:                0.8.2-py_2            conda-forge
    tornado:              4.5.2-py36_0          conda-forge
    vc:                   14-0                  conda-forge
    vs2015_runtime:       14.0.25420-0          conda-forge
    wheel:                0.30.0-py_1           conda-forge
    wincertstore:         0.2-py36_0            conda-forge
    xarray:               0.9.6-py36_0          conda-forge
    yaml:                 0.1.6-vc14_0          conda-forge [vc14]
    zict:                 0.1.3-py_0            conda-forge
    zlib:                 1.2.11-vc14_0         conda-forge [vc14]

@kalefranz
Copy link
Contributor

Ok that's interesting. I'll try to help track this down. Can you also give the output of conda info?

@MaxBo
Copy link
Author

MaxBo commented Oct 5, 2017

Voila


C:\>conda info
Current conda install:

               platform : win-64
          conda version : 4.3.27
       conda is private : False
      conda-env version : 4.3.27
    conda-build version : 3.0.12
         python version : 3.6.0.final.0
       requests version : 2.12.4
       root environment : C:\Anaconda  (writable)
    default environment : C:\Anaconda
       envs directories : C:\Anaconda\envs
                          C:\Users\MaxBohnet\AppData\Local\conda\conda\envs
                          C:\Users\MaxBohnet\.conda\envs
          package cache : C:\Anaconda\pkgs
                          C:\Users\MaxBohnet\AppData\Local\conda\conda\pkgs
           channel URLs : https://conda.anaconda.org/MaxBo/win-64
                          https://conda.anaconda.org/MaxBo/noarch
                          https://conda.anaconda.org/conda-forge/win-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://conda.anaconda.org/synthicity/win-64
                          https://conda.anaconda.org/synthicity/noarch
                          https://repo.continuum.io/pkgs/main/win-64
                          https://repo.continuum.io/pkgs/main/noarch
                          https://repo.continuum.io/pkgs/free/win-64
                          https://repo.continuum.io/pkgs/free/noarch
                          https://repo.continuum.io/pkgs/r/win-64
                          https://repo.continuum.io/pkgs/r/noarch
                          https://repo.continuum.io/pkgs/pro/win-64
                          https://repo.continuum.io/pkgs/pro/noarch
                          https://repo.continuum.io/pkgs/msys2/win-64
                          https://repo.continuum.io/pkgs/msys2/noarch
            config file : C:\Users\MaxBohnet\.condarc
             netrc file : None
           offline mode : False
             user-agent : conda/4.3.27 requests/2.12.4 CPython/3.6.0 Windows/10
Windows/10.0.14393
          administrator : False

@minrk
Copy link
Contributor

minrk commented Dec 15, 2017

I ran into this just now, where defaults is jumping priority for indirect dependencies. In this example, I used boost, but implicitly-installed numpy will always come from defaults instead of the higher-priority conda-forge channel unless requested on the command-line, or if one of the requested packages requires openblas.

Notably: version numbers are identical and build number for conda-forge is higher, so it seems that channel priority isn't tied, but rather inverted somehow. I could imagine that features are somehow relevant, since the conda-forge numpy has a blas_openblas feature, while the defaults channel numpy has no features.

Example using the miniconda docker image:

$ docker run --rm -it continuumio/miniconda3
root@c4ef39cea9c1:/# conda install conda
Fetching package metadata ...........
Solving package specifications: .

Package plan for installation in environment /opt/conda:

The following packages will be UPDATED:

    conda: 4.3.27-py36h2866c0b_0 --> 4.3.30-py36h5d9f9f4_0

root@c4ef39cea9c1:/# conda config --add channels conda-forge
root@c4ef39cea9c1:/# conda config --set show_channel_urls true
root@c4ef39cea9c1:/# conda create --dry-run -n ignore boost=1.65.1 python=3.6
Fetching package metadata .............
Solving package specifications: .

Package plan for installation in environment /opt/conda/envs/ignore:

The following NEW packages will be INSTALLED:

    boost:           1.65.1-py36_0         conda-forge
    boost-cpp:       1.65.1-1              conda-forge
    bzip2:           1.0.6-1               conda-forge
    ca-certificates: 2017.11.5-0           conda-forge
    certifi:         2017.11.5-py36_0      conda-forge
    icu:             58.2-0                conda-forge
    intel-openmp:    2018.0.0-hc7b2577_8   defaults
    mkl:             2018.0.1-h19d6760_4   defaults
    ncurses:         5.9-10                conda-forge
    numpy:           1.13.3-py36ha12f23b_0 defaults
    openssl:         1.0.2n-0              conda-forge
    pip:             9.0.1-py36_0          conda-forge
    python:          3.6.3-4               conda-forge
    readline:        7.0-0                 conda-forge
    setuptools:      38.2.4-py36_0         conda-forge
    sqlite:          3.20.1-0              conda-forge
    tk:              8.6.7-0               conda-forge
    wheel:           0.30.0-py_1           conda-forge
    xz:              5.2.3-0               conda-forge
    zlib:            1.2.11-0              conda-forge

root@c4ef39cea9c1:/# conda create --dry-run -n ignore boost=1.65.1 python=3.6 numpy
Fetching package metadata .............
Solving package specifications: .

Package plan for installation in environment /opt/conda/envs/ignore:

The following NEW packages will be INSTALLED:

    blas:            1.1-openblas                  conda-forge
    boost:           1.65.1-py36_0                 conda-forge
    boost-cpp:       1.65.1-1                      conda-forge
    bzip2:           1.0.6-1                       conda-forge
    ca-certificates: 2017.11.5-0                   conda-forge
    certifi:         2017.11.5-py36_0              conda-forge
    icu:             58.2-0                        conda-forge
    libgfortran:     3.0.0-1                       defaults
    ncurses:         5.9-10                        conda-forge
    numpy:           1.13.3-py36_blas_openblas_201 conda-forge [blas_openblas]
    openblas:        0.2.20-6                      conda-forge
    openssl:         1.0.2n-0                      conda-forge
    pip:             9.0.1-py36_0                  conda-forge
    python:          3.6.3-4                       conda-forge
    readline:        7.0-0                         conda-forge
    setuptools:      38.2.4-py36_0                 conda-forge
    sqlite:          3.20.1-0                      conda-forge
    tk:              8.6.7-0                       conda-forge
    wheel:           0.30.0-py_1                   conda-forge
    xz:              5.2.3-0                       conda-forge
    zlib:            1.2.11-0                      conda-forge

root@c4ef39cea9c1:/# conda info
Current conda install:

               platform : linux-64
          conda version : 4.3.30
       conda is private : False
      conda-env version : 4.3.30
    conda-build version : not installed
         python version : 3.6.2.final.0
       requests version : 2.18.4
       root environment : /opt/conda  (writable)
    default environment : /opt/conda
       envs directories : /opt/conda/envs
                          /root/.conda/envs
          package cache : /opt/conda/pkgs
                          /root/.conda/pkgs
           channel URLs : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.continuum.io/pkgs/main/linux-64
                          https://repo.continuum.io/pkgs/main/noarch
                          https://repo.continuum.io/pkgs/free/linux-64
                          https://repo.continuum.io/pkgs/free/noarch
                          https://repo.continuum.io/pkgs/r/linux-64
                          https://repo.continuum.io/pkgs/r/noarch
                          https://repo.continuum.io/pkgs/pro/linux-64
                          https://repo.continuum.io/pkgs/pro/noarch
            config file : /root/.condarc
             netrc file : None
           offline mode : False
             user-agent : conda/4.3.30 requests/2.18.4 CPython/3.6.2 Linux/4.9.60-linuxkit-aufs debian/8.9 glibc/2.19
                UID:GID : 0:0

@minrk
Copy link
Contributor

minrk commented Dec 15, 2017

Indeed it does appear to be feature-related. When I commented-out the feature-minimization here, it results in the expected (by me) solution. From my understanding of the solver, it would appear that satisfying channel priority either has equal or lower weight to eliminating a feature.

@minrk
Copy link
Contributor

minrk commented Dec 15, 2017

And the difference between specifying numpy at install time vs not appears to be that channel priority appears to take effect during pruning in get_reduced_index prior to feature minimization, whereas the same pruning doesn't get applied to dependencies.

@mbargull
Copy link
Member

get_reduced_index should not affect this. If you want more information on the topic, follow the discussion at #6296 and the related issues.

@minrk
Copy link
Contributor

minrk commented Dec 18, 2017

Good point, on reading more carefully, the bug is indeed that versions/channels are optimized first on requested packages, then features are optimized out, then versions/channels are optimized for dependency packages, precisely T3 in your PR.

@github-actions
Copy link

github-actions bot commented Aug 19, 2021

Hi there, thank you for your contribution to Conda!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed automatically if no further activity occurs.

If you would like this issue to remain open please:

  1. Verify that you can still reproduce the issue in the latest version of Conda
  2. Comment that the issue is still reproducible and include:
  • What version of Conda you reproduced the issue on
  • What OS and version you reproduced the issue on
  • What steps you followed to reproduce the issue
  1. It would also be helpful to have the output of the following commands available:
  • conda info
  • conda config --show-sources
  • conda list --show-channel-urls

NOTE: If this issue was closed prematurely, please leave a comment and we will gladly reopen the issue.

In case this issue was originally about a project that is covered by the Anaconda issue tracker (e.g. Anaconda, Miniconda, packages built by Anaconda, Inc. like Anaconda Navigator etc), please reopen the issue there again.

Thanks!

@github-actions github-actions bot added the stale [bot] marked as stale due to inactivity label Aug 19, 2021
@github-actions
Copy link

Hi again!

This issue has been closed since it has not had recent activity.
Please don't hesitate to leave a comment if that was done prematurely.

Thank you for your contribution.

@github-actions github-actions bot added the stale::closed [bot] closed after being marked as stale label Nov 20, 2021
@github-actions github-actions bot added the locked [bot] locked due to inactivity label Nov 22, 2022
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Nov 22, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
locked [bot] locked due to inactivity stale::closed [bot] closed after being marked as stale stale [bot] marked as stale due to inactivity
Projects
None yet
Development

No branches or pull requests

4 participants