Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

libmamba traceback when using a pin file on an existing env that used build_string #286

Closed
2 tasks done
wuillaum opened this issue Sep 19, 2023 · 8 comments · Fixed by #289
Closed
2 tasks done

libmamba traceback when using a pin file on an existing env that used build_string #286

wuillaum opened this issue Sep 19, 2023 · 8 comments · Fixed by #289
Labels
type::bug describes erroneous operation, use severity::* to classify the type

Comments

@wuillaum
Copy link

wuillaum commented Sep 19, 2023

Checklist

  • I added a descriptive title
  • I searched open reports and couldn't find a duplicate

What happened?

Libmamba fails to resolve an environment when the env was originally created with build strings, and has since had a pin file inserted.

Below is the stack trace encountered (apologies... it's big)

#12 [8/8] RUN conda install -c conda-forge -n myenv --solver libmamba --file requirements.txt
#12 1.191 Channels:
#12 1.191  - conda-forge
#12 1.191  - defaults
#12 1.191 Platform: linux-64
#12 1.191 Collecting package metadata (repodata.json): ...working... done
#12 32.55 Solving environment: ...working... failed
#12 32.61 
#12 32.61 # >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<
#12 32.61 
#12 32.61     Traceback (most recent call last):
#12 32.61       File "/opt/conda/lib/python3.11/site-packages/conda/exception_handler.py", line 16, in __call__
#12 32.61         return func(*args, **kwargs)
#12 32.61                ^^^^^^^^^^^^^^^^^^^^^
#12 32.61       File "/opt/conda/lib/python3.11/site-packages/conda/cli/main.py", line 84, in main_subshell
#12 32.61         exit_code = do_call(args, p)
#12 32.61                     ^^^^^^^^^^^^^^^^
#12 32.61       File "/opt/conda/lib/python3.11/site-packages/conda/cli/conda_argparse.py", line 126, in do_call
#12 32.61         return getattr(module, func_name)(args, parser)
#12 32.61                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#12 32.61       File "/opt/conda/lib/python3.11/site-packages/conda/notices/core.py", line 123, in wrapper
#12 32.61         return func(*args, **kwargs)
#12 32.61                ^^^^^^^^^^^^^^^^^^^^^
#12 32.61       File "/opt/conda/lib/python3.11/site-packages/conda/cli/main_install.py", line 22, in execute
#12 32.61         install(args, parser, "install")
#12 32.61       File "/opt/conda/lib/python3.11/site-packages/conda/cli/install.py", line 309, in install
#12 32.61         unlink_link_transaction = solver.solve_for_transaction(
#12 32.61                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#12 32.61       File "/opt/conda/lib/python3.11/site-packages/conda/core/solve.py", line 153, in solve_for_transaction
#12 32.61         unlink_precs, link_precs = self.solve_for_diff(
#12 32.61                                    ^^^^^^^^^^^^^^^^^^^^
#12 32.61       File "/opt/conda/lib/python3.11/site-packages/conda/core/solve.py", line 214, in solve_for_diff
#12 32.61         final_precs = self.solve_final_state(
#12 32.61                       ^^^^^^^^^^^^^^^^^^^^^^^
#12 32.61       File "/opt/conda/lib/python3.11/site-packages/conda_libmamba_solver/solver.py", line 200, in solve_final_state
#12 32.61         out_state = self._solving_loop(in_state, out_state, index)
#12 32.61                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#12 32.61       File "/opt/conda/lib/python3.11/site-packages/conda_libmamba_solver/solver.py", line 234, in _solving_loop
#12 32.61         solved = self._solve_attempt(in_state, out_state, index)
#12 32.61                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#12 32.61       File "/opt/conda/lib/python3.11/site-packages/conda_libmamba_solver/solver.py", line 319, in _solve_attempt
#12 32.61         out_state.prepare_specs(index)
#12 32.61       File "/opt/conda/lib/python3.11/site-packages/conda_libmamba_solver/state.py", line 597, in prepare_specs
#12 32.61         self._prepare_for_add(index)
#12 32.61       File "/opt/conda/lib/python3.11/site-packages/conda_libmamba_solver/state.py", line 688, in _prepare_for_add
#12 32.61         if sis.requested[name].match(spec):
#12 32.61            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#12 32.61       File "/opt/conda/lib/python3.11/site-packages/conda/models/match_spec.py", line 250, in match
#12 32.61         if not self._match_individual(rec, field_name, v):
#12 32.61                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#12 32.61       File "/opt/conda/lib/python3.11/site-packages/conda/models/match_spec.py", line 255, in _match_individual
#12 32.61         val = getattr(record, field_name)
#12 32.61               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
#12 32.61     AttributeError: 'MatchSpec' object has no attribute 'build'
#12 32.61 
#12 32.61 `$ /opt/conda/bin/conda install -c conda-forge -n myenv --solver libmamba --file requirements.txt`
#12 32.61 
#12 32.61   environment variables:
#12 32.61                  CIO_TEST=<not set>
#12 32.61                CONDA_ROOT=/opt/conda
#12 32.61            CURL_CA_BUNDLE=<not set>
#12 32.61                LD_PRELOAD=<not set>
#12 32.61                      PATH=/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin
#12 32.61                           :/bin
#12 32.61        REQUESTS_CA_BUNDLE=<not set>
#12 32.61             SSL_CERT_FILE=<not set>
#12 32.61 
#12 32.61      active environment : None
#12 32.61        user config file : /root/.condarc
#12 32.61  populated config files : 
#12 32.61           conda version : 23.5.2
#12 32.61     conda-build version : not installed
#12 32.61          python version : 3.11.4.final.0
#12 32.61        virtual packages : __archspec=1=x86_64
#12 32.61                           __glibc=2.31=0
#12 32.61                           __linux=6.2.0=0
#12 32.61                           __unix=0=0
#12 32.61        base environment : /opt/conda  (writable)
#12 32.61       conda av data dir : /opt/conda/etc/conda
#12 32.61   conda av metadata url : None
#12 32.61            channel URLs : https://conda.anaconda.org/conda-forge/linux-64
#12 32.61                           https://conda.anaconda.org/conda-forge/noarch
#12 32.61                           https://repo.anaconda.com/pkgs/main/linux-64
#12 32.61                           https://repo.anaconda.com/pkgs/main/noarch
#12 32.61                           https://repo.anaconda.com/pkgs/r/linux-64
#12 32.61                           https://repo.anaconda.com/pkgs/r/noarch
#12 32.61           package cache : /opt/conda/pkgs
#12 32.61                           /root/.conda/pkgs
#12 32.61        envs directories : /opt/conda/envs
#12 32.61                           /root/.conda/envs
#12 32.61                platform : linux-64
#12 32.61              user-agent : conda/23.5.2 requests/2.29.0 CPython/3.11.4 Linux/6.2.0-32-generic debian/11 glibc/2.31 solver/libmamba conda-libmamba-solver/23.5.0 libmambapy/1.4.1
#12 32.61                 UID:GID : 0:0
#12 32.61              netrc file : None
#12 32.61            offline mode : False
#12 32.61 
#12 32.61 
#12 32.61 An unexpected error has occurred. Conda has prepared the above report.

Conda Info

#8 [4/9] RUN conda info
#8 0.693 
#8 0.693      active environment : None
#8 0.693        user config file : /root/.condarc
#8 0.693  populated config files : 
#8 0.693           conda version : 23.5.2
#8 0.693     conda-build version : not installed
#8 0.693          python version : 3.11.4.final.0
#8 0.693        virtual packages : __archspec=1=x86_64
#8 0.693                           __glibc=2.31=0
#8 0.693                           __linux=6.2.0=0
#8 0.693                           __unix=0=0
#8 0.693        base environment : /opt/conda  (writable)
#8 0.693       conda av data dir : /opt/conda/etc/conda
#8 0.693   conda av metadata url : None
#8 0.693            channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
#8 0.693                           https://repo.anaconda.com/pkgs/main/noarch
#8 0.693                           https://repo.anaconda.com/pkgs/r/linux-64
#8 0.693                           https://repo.anaconda.com/pkgs/r/noarch
#8 0.693           package cache : /opt/conda/pkgs
#8 0.693                           /root/.conda/pkgs
#8 0.693        envs directories : /opt/conda/envs
#8 0.693                           /root/.conda/envs
#8 0.693                platform : linux-64
#8 0.693              user-agent : conda/23.5.2 requests/2.29.0 CPython/3.11.4 Linux/6.2.0-32-generic debian/11 glibc/2.31
#8 0.693                 UID:GID : 0:0
#8 0.693              netrc file : None
#8 0.693            offline mode : False


### Conda Config

```shell
< no output >

Conda list

#8 [4/9] RUN conda list --show-channel-urls
#8 0.625 # packages in environment at /opt/conda:
#8 0.625 #
#8 0.625 # Name                    Version                   Build  Channel
#8 0.625 _libgcc_mutex             0.1                        main    defaults
#8 0.625 _openmp_mutex             5.1                       1_gnu    defaults
#8 0.625 boltons                   23.0.0          py311h06a4308_0    defaults
#8 0.625 brotlipy                  0.7.0           py311h5eee18b_1002    defaults
#8 0.625 bzip2                     1.0.8                h7b6447c_0    defaults
#8 0.625 c-ares                    1.19.0               h5eee18b_0    defaults
#8 0.625 ca-certificates           2023.05.30           h06a4308_0    defaults
#8 0.625 certifi                   2023.5.7        py311h06a4308_0    defaults
#8 0.625 cffi                      1.15.1          py311h5eee18b_3    defaults
#8 0.625 charset-normalizer        2.0.4              pyhd3eb1b0_0    defaults
#8 0.625 conda                     23.5.2          py311h06a4308_0    defaults
#8 0.625 conda-content-trust       0.1.3           py311h06a4308_0    defaults
#8 0.625 conda-libmamba-solver     23.5.0          py311h06a4308_0    defaults
#8 0.625 conda-package-handling    2.1.0           py311h06a4308_0    defaults
#8 0.625 conda-package-streaming   0.8.0           py311h06a4308_0    defaults
#8 0.625 cryptography              39.0.1          py311h9ce1e76_2    defaults
#8 0.625 fmt                       9.1.0                hdb19cb5_0    defaults
#8 0.625 icu                       58.2                 he6710b0_3    defaults
#8 0.625 idna                      3.4             py311h06a4308_0    defaults
#8 0.625 jsonpatch                 1.32               pyhd3eb1b0_0    defaults
#8 0.625 jsonpointer               2.1                pyhd3eb1b0_0    defaults
#8 0.625 krb5                      1.20.1               h143b758_1    defaults
#8 0.625 ld_impl_linux-64          2.38                 h1181459_1    defaults
#8 0.625 libarchive                3.6.2                h6ac8c49_2    defaults
#8 0.625 libcurl                   8.1.1                h251f7ec_1    defaults
#8 0.625 libedit                   3.1.20221030         h5eee18b_0    defaults
#8 0.625 libev                     4.33                 h7f8727e_1    defaults
#8 0.625 libffi                    3.4.4                h6a678d5_0    defaults
#8 0.625 libgcc-ng                 11.2.0               h1234567_1    defaults
#8 0.625 libgomp                   11.2.0               h1234567_1    defaults
#8 0.625 libmamba                  1.4.1                h2dafd23_1    defaults
#8 0.625 libmambapy                1.4.1           py311h2dafd23_1    defaults
#8 0.625 libnghttp2                1.52.0               h2d74bed_1    defaults
#8 0.625 libsolv                   0.7.22               he621ea3_0    defaults
#8 0.625 libssh2                   1.10.0               hdbd6064_2    defaults
#8 0.625 libstdcxx-ng              11.2.0               h1234567_1    defaults
#8 0.625 libuuid                   1.41.5               h5eee18b_0    defaults
#8 0.625 libxml2                   2.10.3               hcbfbd50_0    defaults
#8 0.625 lz4-c                     1.9.4                h6a678d5_0    defaults
#8 0.625 ncurses                   6.4                  h6a678d5_0    defaults
#8 0.625 openssl                   3.0.9                h7f8727e_0    defaults
#8 0.625 packaging                 23.0            py311h06a4308_0    defaults
#8 0.625 pcre2                     10.37                he7ceb23_1    defaults
#8 0.625 pip                       23.1.2          py311h06a4308_0    defaults
#8 0.625 pluggy                    1.0.0           py311h06a4308_1    defaults
#8 0.625 pybind11-abi              4                    hd3eb1b0_1    defaults
#8 0.625 pycosat                   0.6.4           py311h5eee18b_0    defaults
#8 0.625 pycparser                 2.21               pyhd3eb1b0_0    defaults
#8 0.625 pyopenssl                 23.0.0          py311h06a4308_0    defaults
#8 0.625 pysocks                   1.7.1           py311h06a4308_0    defaults
#8 0.625 python                    3.11.4               h955ad1f_0    defaults
#8 0.625 readline                  8.2                  h5eee18b_0    defaults
#8 0.625 reproc                    14.2.4               h295c915_1    defaults
#8 0.625 reproc-cpp                14.2.4               h295c915_1    defaults
#8 0.625 requests                  2.29.0          py311h06a4308_0    defaults
#8 0.625 ruamel.yaml               0.17.21         py311h5eee18b_0    defaults
#8 0.625 setuptools                67.8.0          py311h06a4308_0    defaults
#8 0.625 six                       1.16.0             pyhd3eb1b0_1    defaults
#8 0.625 sqlite                    3.41.2               h5eee18b_0    defaults
#8 0.625 tk                        8.6.12               h1ccaba5_0    defaults
#8 0.625 toolz                     0.12.0          py311h06a4308_0    defaults
#8 0.625 tqdm                      4.65.0          py311h92b7b1e_0    defaults
#8 0.625 tzdata                    2023c                h04d1e81_0    defaults
#8 0.625 urllib3                   1.26.16         py311h06a4308_0    defaults
#8 0.625 wheel                     0.38.4          py311h06a4308_0    defaults
#8 0.625 xz                        5.4.2                h5eee18b_0    defaults
#8 0.625 yaml-cpp                  0.7.0                h295c915_1    defaults
#8 0.625 zlib                      1.2.13               h5eee18b_0    defaults
#8 0.625 zstandard                 0.19.0          py311h5eee18b_0    defaults
#8 0.625 zstd                      1.5.5                hc292b87_0    defaults

Additional Context

I've created a Dockerfile which recreates this issue easily and reliably.

There are 2 dependent files, pinned and requirements.txt

pinned:

python ==3.7.3
pandas ==1.2.5
scipy ==1.7.3

requirements.txt

scipy=1.7.3=py37hf2a6cf1_0
python=3.7.3
pandas=1.2.5=py37h295c915_0

Dockerfile

FROM continuumio/miniconda3:23.5.2-0


WORKDIR /app

COPY requirements.txt .

# No problems creating the env
RUN conda create -c conda-forge -n myenv --file requirements.txt --solver libmamba

# copy and insert pinned file to env location
COPY pinned .
RUN mv pinned /opt/conda/envs/myenv/conda-meta/pinned

# the following conda commands should be a no-op
# the classic solver handles it fine, libmamba errors

# works
RUN conda install -c conda-forge -n myenv --solver classic  --file requirements.txt

# fails
RUN conda install -c conda-forge -n myenv --solver libmamba --file requirements.txt

Simply run docker build . to experience the traceback yourself :)
Please let me know if there's anything else I can do to help! (I apologize in advance if this is the wrong repo for this issue, please let me know if you need me to recreate this somewhere else)

@wuillaum wuillaum added the type::bug describes erroneous operation, use severity::* to classify the type label Sep 19, 2023
@jaimergp
Copy link
Contributor

Thanks for such a detailed report and even a reproducer! I'll take a look today and open a PR if necessary. Looks like the MatchSpec API needs a couple fixes and in the meantime some workarounds 👀

@jaimergp
Copy link
Contributor

jaimergp commented Sep 21, 2023

After some experiments, you can fix it by making sure the pins do not use double equals, but single equals. The parsing is a bit clunky at conda.models.match_spec.MatchSpec, so we have to workaround that. This is not a bug in conda-libmamba-solver, so I'll create an issue in conda/conda.

Summary of attempts:

from conda.models.match_spec import MatchSpec as M
# left side is your CLI arg, right side is the spec in the pinned file

M("scipy=1.7.3=py37hd18d958_0").match(M("scipy =1.7.3"))  # works
M("scipy=1.7.3=py37hd18d958_0").match(M("scipy 1.7.3"))   # fails
M("scipy=1.7.3=py37hd18d958_0").match(M("scipy=1.7.3"))   # works
M("scipy=1.7.3=py37hd18d958_0").match(M("scipy==1.7.3"))  # fails
M("scipy=1.7.3=py37hd18d958_0").match(M("scipy ==1.7.3")) # fails
M("scipy=1.7.3=py37hd18d958_0").match(M("scipy 1.7.3 *")) # fails
M("scipy=1.7.3=py37hd18d958_0").match(M("scipy=1.7.3=*")) # fails
M("scipy=1.7.3=py37hd18d958_0").match(M("scipy[version=1.7.3]")) # fails
M("scipy=1.7.3=py37hd18d958_0").match(M("scipy[version=1.7.3,build=*]")) # fails

🤷

The reason why classic works is that it uses a different (more involved) code path to assess the compatibility of CLI and pinned specs, which doesn't rely on MatchSpec.match.

@jaimergp
Copy link
Contributor

I looked into it a bit a more and turns out that MatchSpec.match is only meant to target full PackageRecord objects and not other MatchSpec objects. I'll submit a PR.

@jaimergp
Copy link
Contributor

Opened #289

@wuillaum
Copy link
Author

Thank you for the quick response @jaimergp! It's pretty wild that the double equal fails because that's the only example I could find in the conda docs 😅

https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-pkgs.html#preventing-packages-from-updating-pinning

I tried using the single equals and added the following line to the Dockerfile
RUN conda install -n myenv pandas=1.3.5

but unfortunately it looks like conda completely ignores the pin file and goes ahead with the update.

#13 [9/9] RUN conda install -n myenv pandas=1.3.5
#13 0.795 Collecting package metadata (current_repodata.json): ...working... done
#13 2.954 Solving environment: ...working... unsuccessful initial attempt using frozen solve. Retrying with flexible solve.
#13 2.955 Collecting package metadata (repodata.json): ...working... done
#13 8.476 Solving environment: ...working... WARNING conda.core.solve:_add_specs(799): pinned spec pandas=1.2.5 conflicts with explicit specs.  Overriding pinned spec.
#13 18.58 done
#13 18.62 
#13 18.62 
#13 18.62 ==> WARNING: A newer version of conda exists. <==
#13 18.62   current version: 23.5.2
#13 18.62   latest version: 23.7.4
#13 18.62 
#13 18.62 Please update conda by running
#13 18.62 
#13 18.62     $ conda update -n base -c defaults conda
#13 18.62 
#13 18.62 Or to minimize the number of packages updated during conda update use
#13 18.62 
#13 18.62      conda install conda=23.7.4
#13 18.62 
#13 18.62 
#13 18.63 
#13 18.63 ## Package Plan ##
#13 18.63 
#13 18.63   environment location: /opt/conda/envs/myenv
#13 18.63 
#13 18.63   added / updated specs:
#13 18.63     - pandas=1.3.5
#13 18.63 
#13 18.63 
#13 18.63 The following packages will be downloaded:
#13 18.63 
#13 18.63     package                    |            build
#13 18.63     ---------------------------|-----------------
#13 18.63     blas-1.0                   |         openblas          46 KB
#13 18.63     bottleneck-1.3.5           |   py37h7deecbd_0         115 KB
#13 18.63     ca-certificates-2023.08.22 |       h06a4308_0         123 KB
#13 18.63     numexpr-2.8.4              |   py37hd2a5715_0         133 KB
#13 18.63     openssl-1.1.1w             |       h7f8727e_0         3.7 MB
#13 18.63     packaging-22.0             |   py37h06a4308_0          68 KB
#13 18.63     pandas-1.3.5               |   py37h8c16a72_0         9.3 MB
#13 18.63     ------------------------------------------------------------
#13 18.63                                            Total:        13.5 MB
#13 18.63 
#13 18.63 The following NEW packages will be INSTALLED:
#13 18.63 
#13 18.63   blas               pkgs/main/linux-64::blas-1.0-openblas 
#13 18.63   bottleneck         pkgs/main/linux-64::bottleneck-1.3.5-py37h7deecbd_0 
#13 18.63   numexpr            pkgs/main/linux-64::numexpr-2.8.4-py37hd2a5715_0 
#13 18.63   packaging          pkgs/main/linux-64::packaging-22.0-py37h06a4308_0 
#13 18.63 
#13 18.63 The following packages will be UPDATED:
#13 18.63 
#13 18.63   ca-certificates    conda-forge::ca-certificates-2023.7.2~ --> pkgs/main::ca-certificates-2023.08.22-h06a4308_0 
#13 18.63   pandas                               1.2.5-py37h295c915_0 --> 1.3.5-py37h8c16a72_0 

The doc example seems pretty straightforward 😕
image

@jaimergp
Copy link
Contributor

I spoke a bit too fast and while the logic of our code in conda-libmamba-solver looked solid, the underlying implementation was faulty. We were running into weird corners of MatchSpec parsing, yes, but we shouldn't be calling MatchSpec.match(MatchSpec), more like MatchSpec.match(PackageRecord). Instead we can use MatchSpec.merge([MatchSpec, ...]) and see how many we get back. If there's only one, they were compatible. Anyway, more details in the attached PR, but this is a proper bug and will be addressed in the upcoming 23.9. I am surprised we didn't hit this earlier but it's good you did, so thanks a bunch for your report!

@costrouc
Copy link
Contributor

Just wanted to message here that I'll be working on the #289 PR. @wuillaum I will work on reproducing this issue tomorrow morning. Thanks for the detailed report!

@wuillaum
Copy link
Author

Thank you all for the incredibly quick turnaround on this issue, and for addressing all of my concerns! I can't believe how fast this change got into main given the complexity, so kudos!!! 🚀

I look forward to using 23.9 soon :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type::bug describes erroneous operation, use severity::* to classify the type
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

3 participants