Conda solver does not respect package specifications in the pinned file

[yhuai]

Current Behavior

pinned file in conda-meta/ folder does not pin packages listed their to the expected version.

Steps to Reproduce

1. Create an empty environment

conda create --name myenv

2. Put the pinned file pinned.txt to the conda-meta folder in myenv.

ls -lh /conda/envs/myenv/conda-meta/

-rw-r--r-- 1 root root  120 Jul 30 03:34 history
-r-xr-xr-x 1 root root 4.5K Jul 30 03:34 pinned

3. Run the following command

conda install --name myenv certifi=2019.6.16

This command will not fail like what Conda 4.6 does. Instead it will try to do the following

Solving environment: | WARNING conda.core.solve:_add_specs(583): pinned spec certifi==2019.3.9 conflicts with explicit specs.  Overriding pinned spec.
failed with current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: \ WARNING conda.core.solve:_add_specs(583): pinned spec certifi==2019.3.9 conflicts with explicit specs.  Overriding pinned spec.
done

## Package Plan ##

  environment location: /conda/envs/myenv

  added / updated specs:
    - certifi=2019.6.16


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    certifi-2019.6.16          |           py37_1         156 KB
    pip-19.0.3                 |           py37_0         1.6 MB
    setuptools-40.8.0          |           py37_0         496 KB
    wheel-0.33.1               |           py37_0          41 KB
    ------------------------------------------------------------
                                           Total:         2.2 MB

The following NEW packages will be INSTALLED:

  _libgcc_mutex      pkgs/main/linux-64::_libgcc_mutex-0.1-main
  ca-certificates    pkgs/main/linux-64::ca-certificates-2019.5.15-0
  certifi            pkgs/main/linux-64::certifi-2019.6.16-py37_1
  libedit            pkgs/main/linux-64::libedit-3.1.20181209-hc058e9b_0
  libffi             pkgs/main/linux-64::libffi-3.2.1-hd88cf55_4
  libgcc-ng          pkgs/main/linux-64::libgcc-ng-8.2.0-hdf63c60_1
  libstdcxx-ng       pkgs/main/linux-64::libstdcxx-ng-8.2.0-hdf63c60_1
  ncurses            pkgs/main/linux-64::ncurses-6.1-he6710b0_1
  openssl            pkgs/main/linux-64::openssl-1.1.1c-h7b6447c_1
  pip                pkgs/main/linux-64::pip-19.0.3-py37_0
  python             pkgs/main/linux-64::python-3.7.3-h0371630_0
  readline           pkgs/main/linux-64::readline-7.0-h7b6447c_5
  setuptools         pkgs/main/linux-64::setuptools-40.8.0-py37_0
  sqlite             pkgs/main/linux-64::sqlite-3.27.2-h7b6447c_0
  tk                 pkgs/main/linux-64::tk-8.6.8-hbc83047_0
  wheel              pkgs/main/linux-64::wheel-0.33.1-py37_0
  xz                 pkgs/main/linux-64::xz-5.2.4-h14c3975_4
  zlib               pkgs/main/linux-64::zlib-1.2.11-h7b6447c_3

If you compare this list with the pinned file, you find that ca-certificates's version is also not what gets defined in the pinned file (pinned file has 2019.1.23).

Expected Behavior

If the user-specified version of a package does not meet the spec defined in the pinned file, Conda should fail. Conda 4.6 actually fails with an error like

SpecsConfigurationConflictError: Requested specs conflict with configured specs.
  requested specs: 
    - certifi=2019.6.16
  pinned specs: 
    - certifi==2019.3.9

Also, Conda's solver needs to respect the version specs defined in the pinned file. For example, in the repro section, the version of ca-certificates should be 2019.1.23.

Environment Information

`conda info`

     active environment : None
       user config file : /root/.condarc
 populated config files : 
          conda version : 4.7.10
    conda-build version : not installed
         python version : 3.7.3.final.0
       virtual packages : 
       base environment : /conda  (writable)
           channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /conda/pkgs
                          /root/.conda/pkgs
       envs directories : /conda/envs
                          /root/.conda/envs
               platform : linux-64
             user-agent : conda/4.7.10 requests/2.22.0 CPython/3.7.3 Linux/4.9.125-linuxkit ubuntu/16.04.6 glibc/2.23
                UID:GID : 0:0
             netrc file : None
           offline mode : False

`conda config --show-sources`

This command does not return anything.

`conda list --show-channel-urls`

n/a

[msarahan]

In my humble opinion, it is annoying for conda to prevent things when you are explicitly specifying them. The purpose behind that change is to enforce pins such that conda will never accidentally change those pins, but if you explicitly say otherwise, conda will get out of your way.

I'm not really opposed to reverting this behavior, but I would like a lot more detail on your use case.

[jjhelmus]

I agree with @yhuai, pinning should be respected even if explicit versions are given on the command line. Putting a pinned file in an environment takes action by the user and is informing conda about a users request. Conda already has a command line argument, --no-pin, to ignore pinning. Only when that argument is given should a pinning file be ignored.

[yhuai]

Regarding my use case, we want to have a deterministic way to define Python environments for production workloads. Specifically, we want to make sure that environments built from the same spec are identical no matter when we build those environments. Because it is quite hard to control all of transient dependencies of Python packages, what we are doing is to make the yml file used to create the env only specify top level dependencies that we need (e.g. Python and Numpy) and use a long pinned files to fix the versions of all transient dependencies. We also want to use pinned file to lock down the versions of some critical libraries and fail the request if a user wants to change the version.

As @jjhelmus pointed out, because Conda already has --no-pin flag, current Conda 4.7 has two approaches to override versions specified in pinned file. But, it takes away the approach of making pinned file take precedence.

[msarahan]

This is really a lockfile/env spec use case more than anything. I do not see the two override behaviors as equivalent. --no-pin ignores ALL pins, whereas the new behavior only overrides ones that are explicit specs in the current operation and leaves anything else in place. That's much more fine-grained control. I understand they are probably functionally equivalent in your use case, though.

I'm not going to dedicate my team's time to changing this behavior. Our time is better spent on efforts to improve any kind of lockfile capability. That is on our roadmap for 4Q right now.

[yhuai]

@msarahan Thank you for point out the lock file capability. Is #7248 what you are referring to?

[msarahan]

@yhuai yes, that's a good issue. Further developments should be discussed there.

[yhuai]

@msarahan thank you

[eli-osherovich]

Related issue: #9995

[iSevenDays]

I also treat this as a bug.
I pinned matplotlib==3.2.2
But conda does not respect that requirement, instead, it wants to update it.

conda install matplotlib
Collecting package metadata (current_repodata.json): done
Solving environment: / WARNING conda.core.solve:_add_specs(611): pinned spec matplotlib==3.2.2 conflicts with explicit specs. Overriding pinned spec.
done
...
The following NEW packages will be INSTALLED:

matplotlib pkgs/main/osx-64::matplotlib-3.3.2-hecd8cb5_0

Like, did I ask for the override? No! I want to keep the requirement of matplotlib==3.2.2, I did not ask for the newer version

[D-Nations]

I'm having a hell of a time trying to make pinned work for OpenSSL.
1.1.1k breaks my access to the internet behind the company firewall, but 1.1.1i is fine.
I've tried every variation on the version number I can find, but when i run "conda upgrade --all" it still tries to update it.
I'm aware that the ordering for openssl versions is different, but I still can't figure out how to specify a version correctly.

Is this also the override or is there still something wrong with the version number?

[D-Nations]

I figured out the root of my problem, which was that I had the proxy settings configured incorrectly. Still stands that I was unable to pin the OpenSSL version.

[bkreider]

This is an issue for me as well. Pinning doesn't seem to work at all. If you don't specify the version, it still overrides the pin. If pinning is supposed to not really work, then at the very least there is a documentation bug.

[huangliang0828]

I meet the same problems in win10 conda64 python3.8 & python 3.9，the pinned file in conda-meta folder was ignoreed or overrided， I hope there will be a good solution

[hendeb]

I am having the same problem using Conda 4.13.0/Python 3.9.13/Mamba 0.24.0 running on CentOS 7.9.2009. I have two PyPI packages installed from source that Conda continuously wants to replace, seemingly no matter what I do with $CONDA_PREFIX/conda-meta/pinned and $CONDA_PREFIX/.condarc.

[hendeb]

I am having the same problem using Conda 4.13.0/Python 3.9.13/Mamba 0.24.0 running on CentOS 7.9.2009. I have two PyPI packages installed from source that Conda continuously wants to replace, seemingly no matter what I do with $CONDA_PREFIX/conda-meta/pinned and $CONDA_PREFIX/.condarc.

Actually, it seems I was wrong - Mamba told me it was going to replace the PyPI packages when I ran mamba update -d --all - in fact, it left them alone...

[austinstover]

This should really be classified as a bug, and is still an issue. conda does not respect my pin openssl ==1.1.1t when I do conda update --all OR when I run conda install spyder, which does not explicitly override the pin. Yet, conda insists on updating openssl to 3.1.0, which breaks my environment!

The documentation clearly states that pinning "prevents packages listed in the pinned file from being updated," which it clearly does not, at least for openssl, making this a bug.

[bsikander]

I have the same issue. This is a huge pain. In my pinned file, I have python set to 3.10.8 but when I run the following commands,

conda install -n base -c conda-forge r-rlang==1.1.0
conda install -n base --quiet --yes --freeze-installed --file /requirements-conda.txt

I see that it upgrades python to 3.10.11, which breaks my envrionment.

#16 6770.7   python                          3.10.8-h4a9ceb5_0_cpython --> 3.10.11-he550d4f_0_cpython 

In my conda requirement file, I am not explicitly installing python 3.10.11, this is an indirect upgrade and pin is not being respected.

My pinned file content cat /opt/conda/conda-meta/pinned are
python 3.10.8

[jaimergp]

My two cents:

• If a user went through the extra hassle of adding a file to pin stuff, that's a very explicit request. If they want to override it, then they should update or remove the pin in the pinned file.
• If the pinned (or a global config with pinned_packages?) file was added by a sysadmin or similar, it's for a good reason and a regular user should not be able to override it that easily.

So maybe it's too late for classic because this behaviour has been in the wild for too long, but I'd vote to remove this behaviour in conda-libmamba-solver. See conda/conda-libmamba-solver#289 for more details if you are curious.

[jezdez]

I'd suggest fixing this in conda-libmamba-solver, note the difference in behavior to the classic solver in the release notes of conda-libmamba-solver and move on.