Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Invalid/Temporary local packages unexpectedly appearing in conda-lock.yml #556

Closed
2 tasks done
zaneselvans opened this issue Nov 20, 2023 · 2 comments · Fixed by #558
Closed
2 tasks done

Invalid/Temporary local packages unexpectedly appearing in conda-lock.yml #556

zaneselvans opened this issue Nov 20, 2023 · 2 comments · Fixed by #558

Comments

@zaneselvans
Copy link

Checklist

  • I added a descriptive title
  • I searched open reports and couldn't find a duplicate

What happened?

After updating to conda-lock v2.5.0 some odd new packages are showing up in our conda-lock.yml when it is regenerated via a GitHub action:

package:
  - name: __glibc
    version: "2.17"
    manager: conda
    platform: linux-64
    dependencies: {}
    url: file:///home/runner/work/_temp/tmp8q60deq6/linux-64/__glibc-2.17-0.tar.bz2
    hash:
      md5: ""
    category: main
    optional: false
  - name: __osx
    version: "11.0"
    manager: conda
    platform: osx-64
    dependencies: {}
    url: file:///home/runner/work/_temp/tmp8q60deq6/osx-64/__osx-11.0-0.tar.bz2
    hash:
      md5: ""
    category: main
    optional: false
  - name: __osx
    version: "11.0"
    manager: conda
    platform: osx-arm64
    dependencies: {}
    url: file:///home/runner/work/_temp/tmp8q60deq6/osx-arm64/__osx-11.0-0.tar.bz2
    hash:
      md5: ""
    category: main
    optional: false

I noticed that they exist because the lack of a hash value is causing the setup-micromamba GitHub action to fail with the following error:

  Empty environment created at prefix: /home/runner/micromamba/envs/pudl-dev
  critical libmamba either package 'package.info.hash.md5' or 'package.info.hash.sha256' must be specified, found none
  Error: The process '/home/runner/micromamba-bin/micromamba' failed with exit code 1

However, no packages named __osx or __glibc appear in the conda-lock.yml when I generate it locally using a similar command. The only obvious differences between the two environments are:

  • The GitHub runners are running Linux, and my local environment is on an M1 Macbook Pro
  • On the GitHub runner it is using conda-lock --micromamba while locally I'm using conda-lock --mamba. The command being used to generate the conda lockfiles / environment files:
conda-lock \
	--${mamba} \
	--file=pyproject.toml \
	--lockfile=environments/conda-lock.yml

However, running the above commands locally using --micromamba does not result in the __glibc or __osx packages appearing in conda-lock.yml

Conda Info

active environment : None
            shell level : 0
       user config file : /Users/zane/.condarc
 populated config files : /Users/zane/miniforge3/.condarc
                          /Users/zane/.condarc
          conda version : 23.9.0
    conda-build version : not installed
         python version : 3.10.12.final.0
       virtual packages : __archspec=1=arm64
                          __osx=14.1=0
                          __unix=0=0
       base environment : /Users/zane/miniforge3  (writable)
      conda av data dir : /Users/zane/miniforge3/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/osx-arm64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.anaconda.com/pkgs/main/osx-arm64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/osx-arm64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /Users/zane/miniforge3/pkgs
                          /Users/zane/.conda/pkgs
       envs directories : /Users/zane/miniforge3/envs
                          /Users/zane/.conda/envs
               platform : osx-arm64
             user-agent : conda/23.9.0 requests/2.31.0 CPython/3.10.12 Darwin/23.1.0 OSX/14.1
                UID:GID : 1000:20
             netrc file : None
           offline mode : False

Conda Config

==> /Users/zane/miniforge3/.condarc <==
channels:
  - conda-forge

==> /Users/zane/.condarc <==
auto_activate_base: False
default_python: 3.10
channel_priority: strict
channels:
  - conda-forge
  - defaults

Conda list

# packages in environment at /Users/zane/miniforge3:
#
# Name                    Version                   Build  Channel
annotated-types           0.6.0              pyhd8ed1ab_0    conda-forge
appdirs                   1.4.4              pyh9f0ad1d_0    conda-forge
boltons                   23.0.0             pyhd8ed1ab_0    conda-forge
brotli-python             1.1.0           py310h1253130_0    conda-forge
bzip2                     1.0.8                h3422bc3_4    conda-forge
c-ares                    1.19.1               hb547adb_0    conda-forge
ca-certificates           2023.11.17           hf0a4a13_0    conda-forge
cachecontrol              0.13.1             pyhd8ed1ab_0    conda-forge
cachecontrol-with-filecache 0.13.1             pyhd8ed1ab_0    conda-forge
cachy                     0.3.0              pyhd8ed1ab_1    conda-forge
certifi                   2023.11.17         pyhd8ed1ab_0    conda-forge
cffi                      1.15.1          py310h2399d43_3    conda-forge
charset-normalizer        3.2.0              pyhd8ed1ab_0    conda-forge
click                     8.1.7           unix_pyh707e725_0    conda-forge
click-default-group       1.2.4              pyhd8ed1ab_0    conda-forge
clikit                    0.6.2              pyhd8ed1ab_2    conda-forge
colorama                  0.4.6              pyhd8ed1ab_0    conda-forge
conda                     23.9.0          py310hbe9552e_0    conda-forge
conda-libmamba-solver     23.9.2             pyhd8ed1ab_0    conda-forge
conda-lock                2.5.0              pyhd8ed1ab_0    conda-forge
conda-package-handling    2.2.0              pyh38be061_0    conda-forge
conda-package-streaming   0.9.0              pyhd8ed1ab_0    conda-forge
crashtest                 0.4.1              pyhd8ed1ab_0    conda-forge
cryptography              41.0.3          py310hdd3b5e7_0    conda-forge
distlib                   0.3.7              pyhd8ed1ab_0    conda-forge
ensureconda               1.4.3              pyhd8ed1ab_0    conda-forge
filelock                  3.13.1             pyhd8ed1ab_0    conda-forge
fmt                       10.1.1               h1995070_0    conda-forge
gitdb                     4.0.11             pyhd8ed1ab_0    conda-forge
gitpython                 3.1.40             pyhd8ed1ab_0    conda-forge
html5lib                  1.1                pyh9f0ad1d_0    conda-forge
icu                       73.2                 hc8870d7_0    conda-forge
idna                      3.4                pyhd8ed1ab_0    conda-forge
importlib-metadata        6.8.0              pyha770c72_0    conda-forge
importlib_metadata        6.8.0                hd8ed1ab_0    conda-forge
jaraco.classes            3.3.0              pyhd8ed1ab_0    conda-forge
jinja2                    3.1.2              pyhd8ed1ab_1    conda-forge
jsonpatch                 1.32               pyhd8ed1ab_0    conda-forge
jsonpointer               2.0                        py_0    conda-forge
keyring                   24.2.0          py310hbe9552e_1    conda-forge
krb5                      1.21.2               h92f50d5_0    conda-forge
libarchive                3.7.2                h82b9b87_0    conda-forge
libcurl                   8.4.0                h2d989ff_0    conda-forge
libcxx                    16.0.6               h4653b0c_0    conda-forge
libedit                   3.1.20191231         hc8eb9b7_2    conda-forge
libev                     4.33                 h642e427_1    conda-forge
libffi                    3.4.2                h3422bc3_5    conda-forge
libiconv                  1.17                 he4db4b2_0    conda-forge
libmamba                  1.5.3                h0a6dc31_2    conda-forge
libmambapy                1.5.3           py310h3812fd7_2    conda-forge
libnghttp2                1.52.0               hae82a92_0    conda-forge
libsolv                   0.7.24               ha614eb4_3    conda-forge
libsqlite                 3.43.0               hb31c410_0    conda-forge
libssh2                   1.11.0               h7a5bd25_0    conda-forge
libxml2                   2.11.5               h25269f3_1    conda-forge
libzlib                   1.2.13               h53f4e23_5    conda-forge
lz4-c                     1.9.4                hb7217d7_0    conda-forge
lzo                       2.10              h642e427_1000    conda-forge
mamba                     1.5.3           py310ha5d4528_2    conda-forge
markupsafe                2.1.3           py310h2aa6e3c_1    conda-forge
more-itertools            10.1.0             pyhd8ed1ab_0    conda-forge
msgpack-python            1.0.6           py310h38f39d4_0    conda-forge
ncurses                   6.4                  h7ea286d_0    conda-forge
openssl                   3.1.4                h0d3ecfb_0    conda-forge
packaging                 23.1               pyhd8ed1ab_0    conda-forge
pastel                    0.2.1              pyhd8ed1ab_0    conda-forge
pip                       23.2.1             pyhd8ed1ab_0    conda-forge
pkginfo                   1.9.6              pyhd8ed1ab_0    conda-forge
platformdirs              3.11.0             pyhd8ed1ab_0    conda-forge
pluggy                    1.3.0              pyhd8ed1ab_0    conda-forge
pybind11-abi              4                    hd8ed1ab_3    conda-forge
pycosat                   0.6.4           py310h8e9501a_1    conda-forge
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pydantic                  2.4.2              pyhd8ed1ab_1    conda-forge
pydantic-core             2.10.1          py310had9acf8_0    conda-forge
pylev                     1.4.0              pyhd8ed1ab_0    conda-forge
pyopenssl                 23.2.0             pyhd8ed1ab_1    conda-forge
pysocks                   1.7.1              pyha2e5f31_6    conda-forge
python                    3.10.12         h01493a6_0_cpython    conda-forge
python_abi                3.10                    3_cp310    conda-forge
pyyaml                    6.0.1           py310h2aa6e3c_1    conda-forge
readline                  8.2                  h92ec313_1    conda-forge
reproc                    14.2.4               h1a8c8d9_0    conda-forge
reproc-cpp                14.2.4               hb7217d7_0    conda-forge
requests                  2.31.0             pyhd8ed1ab_0    conda-forge
ruamel.yaml               0.17.32         py310h2aa6e3c_0    conda-forge
ruamel.yaml.clib          0.2.7           py310h8e9501a_1    conda-forge
setuptools                68.1.2             pyhd8ed1ab_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
smmap                     5.0.0              pyhd8ed1ab_0    conda-forge
tk                        8.6.12               he1e0b03_0    conda-forge
tomli                     2.0.1              pyhd8ed1ab_0    conda-forge
tomlkit                   0.12.2             pyha770c72_0    conda-forge
toolz                     0.12.0             pyhd8ed1ab_0    conda-forge
tqdm                      4.66.1             pyhd8ed1ab_0    conda-forge
truststore                0.8.0              pyhd8ed1ab_0    conda-forge
typing-extensions         4.8.0                hd8ed1ab_0    conda-forge
typing_extensions         4.8.0              pyha770c72_0    conda-forge
tzdata                    2023c                h71feb2d_0    conda-forge
urllib3                   1.26.18            pyhd8ed1ab_0    conda-forge
virtualenv                20.24.6            pyhd8ed1ab_0    conda-forge
webencodings              0.5.1              pyhd8ed1ab_2    conda-forge
wheel                     0.41.2             pyhd8ed1ab_0    conda-forge
xz                        5.2.6                h57fd34a_0    conda-forge
yaml                      0.2.5                h3422bc3_2    conda-forge
yaml-cpp                  0.8.0                h13dd4ca_0    conda-forge
zipp                      3.17.0             pyhd8ed1ab_0    conda-forge
zstandard                 0.19.0          py310h8e9501a_0    conda-forge
zstd                      1.5.5                h4f39d0f_0    conda-forge

Additional Context

micromamba --config
channels:
  - conda-forge
  - defaults
channel_priority: strict
auto_activate_base: false
micromamba --version
1.5.3
This was referenced Nov 20, 2023
Merged
Merged
@zaneselvans zaneselvans changed the title Temporary local packages unexpectedly appearing in conda-lock.yml Invalid/Temporary local packages unexpectedly appearing in conda-lock.yml Nov 20, 2023
@maresb
Copy link
Contributor

maresb commented Nov 20, 2023

Thanks for the report! Looks like we have a regression. 😞

Please let me know if you figure out a reliable way to locally reproduce this.

For context, these are virtual packages.

@maresb
Copy link
Contributor

maresb commented Nov 20, 2023

Ah, I see the problem. Our toposort function was not just sorting, but also filtering out the virtual packages. I'll fix this...

This was referenced Nov 20, 2023
Merged
Closed
maresb added a commit to maresb/conda-lock that referenced this issue Nov 24, 2023
@maresb
Filter virtual packages separately from toposort.
051b7ae 
Otherwise it's logically confusing, and it led to conda#556
@maresb maresb mentioned this issue Nov 24, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Filter virtual packages maresb/conda-lock
2 participants
@zaneselvans @maresb