Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

KeyError when using libmamba solver with conda-build #4991

Closed
jezdez opened this issue Sep 4, 2023 · 11 comments · Fixed by #5037
Closed

KeyError when using libmamba solver with conda-build #4991

jezdez opened this issue Sep 4, 2023 · 11 comments · Fixed by #5037
Labels
source::anaconda created by members of Anaconda, Inc. type::bug describes erroneous operation, use severity::* to classify the type
Milestone
3.28.0

Comments

@jezdez
Copy link
Member

jezdez commented Sep 4, 2023 •
edited by kenodegard

What happened?

A customer has seen this traceback when using conda-build 3.26.1 with conda-libmamba-solver:

Traceback (most recent call last):
File "/path/to/workspace/conda.lzsddfplas/lib/python3.11/site-packages/conda/exception_handler.py", line 17, in __call__
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/path/to/workspace/conda.lzsddfplas/lib/python3.11/site-packages/conda/cli/main.py", line 64, in main_subshell
exit_code = do_call(args, parser)
^^^^^^^^^^^^^^^^^^^^^
File "/path/to/workspace/conda.lzsddfplas/lib/python3.11/site-packages/conda/cli/conda_argparse.py", line 143, in do_call
result = plugin_subcommand.action(getattr(args, "_args", args))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/path/to/workspace/conda.lzsddfplas/lib/python3.11/site-packages/conda_build/plugin.py", line 10, in build
execute(*args, **kwargs)
File "/path/to/workspace/conda.lzsddfplas/lib/python3.11/site-packages/conda_build/cli/main_build.py", line 568, in execute
outputs = api.build(
^^^^^^^^^^
File "/path/to/workspace/conda.lzsddfplas/lib/python3.11/site-packages/conda_build/api.py", line 253, in build
return build_tree(
^^^^^^^^^^^
File "/path/to/workspace/conda.lzsddfplas/lib/python3.11/site-packages/conda_build/build.py", line 3804, in build_tree
packages_from_this = build(
^^^^^^
File "/path/to/workspace/conda.lzsddfplas/lib/python3.11/site-packages/conda_build/build.py", line 2474, in build
output_metas = expand_outputs([(m, need_source_download, need_reparse_in_env)])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/path/to/workspace/conda.lzsddfplas/lib/python3.11/site-packages/conda_build/render.py", line 923, in expand_outputs
for output_dict, m in deepcopy(_m).get_output_metadata_set(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/path/to/workspace/conda.lzsddfplas/lib/python3.11/site-packages/conda_build/metadata.py", line 2574, in get_output_metadata_set
conda_packages = finalize_outputs_pass(
^^^^^^^^^^^^^^^^^^^^^^
File "/path/to/workspace/conda.lzsddfplas/lib/python3.11/site-packages/conda_build/metadata.py", line 934, in finalize_outputs_pass
fm = finalize_metadata(
^^^^^^^^^^^^^^^^^^
File "/path/to/workspace/conda.lzsddfplas/lib/python3.11/site-packages/conda_build/render.py", line 637, in finalize_metadata
build_unsat, host_unsat = add_upstream_pins(
^^^^^^^^^^^^^^^^^^
File "/path/to/workspace/conda.lzsddfplas/lib/python3.11/site-packages/conda_build/render.py", line 452, in add_upstream_pins
build_deps, build_unsat, extra_run_specs_from_build = _read_upstream_pin_files(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/path/to/workspace/conda.lzsddfplas/lib/python3.11/site-packages/conda_build/render.py", line 440, in _read_upstream_pin_files
extra_run_specs = get_upstream_pins(m, actions, env)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/path/to/workspace/conda.lzsddfplas/lib/python3.11/site-packages/conda_build/render.py", line 420, in get_upstream_pins
loc, dist = execute_download_actions(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/path/to/workspace/conda.lzsddfplas/lib/python3.11/site-packages/conda_build/render.py", line 387, in execute_download_actions
_loc = os.path.join(pkg_dir, index[pkg].fn)
~~~~~^^^^^
KeyError: Dist(channel='https://conda.anaconda.xxx.xxx/anaconda', dist_name='python-3.8.17-h955ad1f_0', name='python', fmt='.tar.bz2', version='3.8.17', build_string='h955ad1f_0', build_number=0, base_url=None, platform=None)

Conda Details

conda info 
active environment : None
user config file : /root/.condarc
populated config files : /root/conda-build/.condarc
conda version : 23.7.3
conda-build version : 3.26.1
python version : 3.11.4.final.0
virtual packages : __archspec=1=x86_64
__glibc=2.26=0
__linux=3.10.0=0
__unix=0=0
base environment : /root/conda-build (writable)
conda av data dir : /root/conda-build/etc/conda
conda av metadata url : None
channel URLs : https://conda.anaconda.xxxxx/arepo/linux-64
https://conda.anaconda.xxxxx/arepo/noarch
https://conda.anaconda.xxxxx/anaconda-patched/linux-64
https://conda.anaconda.xxxxx/anaconda-patched/noarch
https://conda.anaconda.xxxxx/anaconda/linux-64
https://conda.anaconda.xxxxx/anaconda/noarch
https://conda.anaconda.xxxxx/r-patched/linux-64
https://conda.anaconda.xxxxx/r-patched/noarch
https://conda.anaconda.xxxxx/r/linux-64
https://conda.anaconda.xxxxx/r/noarch
https://conda.anaconda.xxxxx/msys2-patched/linux-64
https://conda.anaconda.xxxxx/msys2-patched/noarch
https://conda.anaconda.xxxxx/msys2/linux-64
https://conda.anaconda.xxxxx/msys2/noarch
package cache : /root/conda-build/pkgs
/root/.conda/pkgs
envs directories : /root/conda-build/envs
/root/.conda/envs
platform : linux-64
user-agent : conda/23.7.3 requests/2.31.0 CPython/3.11.4 Linux/3.10.0-1160.95.1.el7.x86_64 amzn/2 glibc/2.26 solver/libmamba conda-libmamba-solver/23.7.0 libmambapy/1.4.1
UID:GID : 0:0
netrc file : None
offline mode : False
conda config 
==> /root/conda-build/.condarc <==
allow_softlinks: False
auto_update_conda: False
notify_outdated_conda: False
aggressive_update_packages:
- anaconda-patched::ca-certificates
- anaconda-patched::certifi
- arepo::condarc
- arepo::anaconda-client-config
add_anaconda_token: True
channel_alias: https://conda.anaconda.xxxxx/
channels:
- defaults
default_channels:
- arepo
- anaconda-patched
- anaconda
- r-patched
- r
- msys2-patched
- msys2
show_channel_urls: True
report_errors: False
shortcuts: False
solver: libmamba

==> cmd_line <==
debug: False
json: False
conda list 
# packages in environment at /root/conda-build:
#
# Name Version Build Channel
_libgcc_mutex 0.1 main defaults
_openmp_mutex 5.1 1_gnu defaults
anaconda-client 1.12.0 py311h06a4308_0 defaults
anaconda-client-config 2019.08.21 0 defaults
attrs 22.1.0 py311h06a4308_0 defaults
backports 1.1 pyhd3eb1b0_0 defaults
backports.functools_lru_cache 1.6.4 pyhd3eb1b0_0 defaults
backports.tempfile 1.0 pyhd3eb1b0_1 defaults
backports.weakref 1.0.post1 py_1 defaults
beautifulsoup4 4.12.2 py311h06a4308_0 defaults
boltons 23.0.0 py311h06a4308_0 defaults
brotlipy 0.7.0 py311h5eee18b_1002 defaults
bzip2 1.0.8 h7b6447c_0 defaults
c-ares 1.19.0 h5eee18b_0 defaults
ca-certificates 2023.05.30 h06a4308_0 defaults
certifi 2023.7.22 py311h06a4308_0 defaults
cffi 1.15.1 py311h5eee18b_3 defaults
chardet 4.0.0 py311h06a4308_1003 defaults
charset-normalizer 2.0.4 pyhd3eb1b0_0 defaults
click 8.0.4 py311h06a4308_0 defaults
clyent 1.2.2 py311h06a4308_1 defaults
conda 23.7.3 py311h06a4308_0 defaults
conda-build 3.26.1 py311h06a4308_0 defaults
conda-index 0.2.3 py311h06a4308_0 defaults
conda-libmamba-solver 23.7.0 py311h06a4308_0 defaults
conda-package-handling 2.2.0 py311h06a4308_0 defaults
conda-package-streaming 0.9.0 py311h06a4308_0 defaults
conda-standalone 22.11.1 h06a4308_1 defaults
conda-verify 3.4.2 py_1 defaults
conda-wrap 0.2.0 py_0 defaults
condarc 2023.03.1 0 defaults
constructor 3.4.5 py311h06a4308_0 defaults
cryptography 41.0.2 py311h22a60cf_0 defaults
defusedxml 0.7.1 pyhd3eb1b0_0 defaults
filelock 3.9.0 py311h06a4308_0 defaults
fmt 9.1.0 hdb19cb5_0 defaults
future 0.18.3 py311h06a4308_0 defaults
glob2 0.7 pyhd3eb1b0_0 defaults
icu 73.1 h6a678d5_0 defaults
idna 3.4 py311h06a4308_0 defaults
jinja2 3.1.2 py311h06a4308_0 defaults
jsonpatch 1.32 pyhd3eb1b0_0 defaults
jsonpointer 2.1 pyhd3eb1b0_0 defaults
jsonschema 4.17.3 py311h06a4308_0 defaults
jupyter_core 5.3.0 py311h06a4308_0 defaults
krb5 1.20.1 h143b758_1 defaults
ld_impl_linux-64 2.38 h1181459_1 defaults
libarchive 3.6.2 h6ac8c49_2 defaults
libcurl 8.2.1 h251f7ec_0 defaults
libedit 3.1.20221030 h5eee18b_0 defaults
libev 4.33 h7f8727e_1 defaults
libffi 3.4.4 h6a678d5_0 defaults
libgcc-ng 11.2.0 h1234567_1 defaults
libgomp 11.2.0 h1234567_1 defaults
liblief 0.12.3 h6a678d5_0 defaults
libmamba 1.4.1 h2dafd23_1 defaults
libmambapy 1.4.1 py311h2dafd23_1 defaults
libnghttp2 1.52.0 h2d74bed_1 defaults
libsolv 0.7.22 he621ea3_0 defaults
libssh2 1.10.0 hdbd6064_2 defaults
libstdcxx-ng 11.2.0 h1234567_1 defaults
libuuid 1.41.5 h5eee18b_0 defaults
libxml2 2.10.4 hf1b16e4_1 defaults
lz4-c 1.9.4 h6a678d5_0 defaults
markupsafe 2.1.1 py311h5eee18b_0 defaults
more-itertools 8.12.0 pyhd3eb1b0_0 defaults
nbformat 5.7.0 py311h06a4308_0 defaults
ncurses 6.4 h6a678d5_0 defaults
openssl 3.0.10 h7f8727e_2 defaults
packaging 23.1 py311h06a4308_0 defaults
patch 2.7.6 h7b6447c_1001 defaults
patchelf 0.17.2 h6a678d5_0 defaults
pcre2 10.37 he7ceb23_1 defaults
pip 23.2.1 py311h06a4308_0 defaults
pkginfo 1.9.6 py311h06a4308_0 defaults
platformdirs 3.10.0 py311h06a4308_0 defaults
pluggy 1.0.0 py311h06a4308_1 defaults
psutil 5.9.0 py311h5eee18b_0 defaults
py-lief 0.12.3 py311h6a678d5_0 defaults
pybind11-abi 4 hd3eb1b0_1 defaults
pycosat 0.6.4 py311h5eee18b_0 defaults
pycparser 2.21 pyhd3eb1b0_0 defaults
pyopenssl 23.2.0 py311h06a4308_0 defaults
pyrsistent 0.18.0 py311h5eee18b_0 defaults
pysocks 1.7.1 py311h06a4308_0 defaults
python 3.11.4 h955ad1f_0 defaults
python-dateutil 2.8.2 pyhd3eb1b0_0 defaults
python-fastjsonschema 2.16.2 py311h06a4308_0 defaults
python-libarchive-c 2.9 pyhd3eb1b0_1 defaults
pytz 2022.7 py311h06a4308_0 defaults
pyyaml 6.0 py311h5eee18b_1 defaults
readline 8.2 h5eee18b_0 defaults
reproc 14.2.4 h295c915_1 defaults
reproc-cpp 14.2.4 h295c915_1 defaults
requests 2.31.0 py311h06a4308_0 defaults
requests-toolbelt 1.0.0 py311h06a4308_0 defaults
ruamel.yaml 0.17.21 py311h5eee18b_0 defaults
setuptools 68.0.0 py311h06a4308_0 defaults
setuptools-scm 7.1.0 py311h06a4308_0 defaults
six 1.16.0 pyhd3eb1b0_1 defaults
soupsieve 2.4 py311h06a4308_0 defaults
sqlite 3.41.2 h5eee18b_0 defaults
tk 8.6.12 h1ccaba5_0 defaults
toolz 0.12.0 py311h06a4308_0 defaults
tqdm 4.65.0 py311h92b7b1e_0 defaults
traitlets 5.7.1 py311h06a4308_0 defaults
typing-extensions 4.7.1 py311h06a4308_0 defaults
typing_extensions 4.7.1 py311h06a4308_0 defaults
tzdata 2023c h04d1e81_0 defaults
urllib3 1.26.16 py311h06a4308_0 defaults
wheel 0.38.4 py311h06a4308_0 defaults
xz 5.4.2 h5eee18b_0 defaults
yaml 0.2.5 h7b6447c_0 defaults
yaml-cpp 0.7.0 h295c915_1 defaults
zlib 1.2.13 h5eee18b_0 defaults
zstandard 0.19.0 py311h5eee18b_0 defaults
zstd 1.5.5 hc292b87_0 defaults
@jezdez jezdez added type::bug describes erroneous operation, use severity::* to classify the type source::anaconda created by members of Anaconda, Inc. labels Sep 4, 2023
@jezdez
Copy link
Member Author

jezdez commented Sep 4, 2023

@jaimergp Have you see this before?

@jaimergp
Copy link
Contributor

jaimergp commented Sep 4, 2023

Nop. It looks like there's some kind of private mirror involved? I assume this is conda-libmamba-solver 23.7.0 too?

conda-build hacks its way to build an index dictionary that maps Dist objects to PackageRecords. I think we bypass that mechanism partially, so maybe we are not rebuilding it as we reload outputs or something? Are they able to share their recipe, or an equivalent reproducer? I can take a look once I have a way to debug it :D

@hsparra
Copy link

hsparra commented Sep 7, 2023 •
edited by kenodegard

It is conda-libmamba-solver 23.7.0.

They get the error if they use conda-build is version 3.26.1 with libmamba solver, but the build works with classic w/ 3.26.1 conda-build. They have not yet tried libmamba with older conda-build.

Not sure if this helps but:

Dockerfile 
FROM xxxxxx/amazonlinux:2 as base

COPY conda-build-0.0.0-Linux-x86_64.sh /tmp/conda-build.sh
RUN /tmp/conda-build.sh -b
RUN mkdir -p /root/conda-build/pkgs

FROM xxxxxx/amazonlinux:2
COPY --from=base /root/conda-build /root/conda-build
RUN source ~/conda-build/etc/profile.d/conda.sh && conda init
SHELL ["/bin/bash", "-lc"]
compose.yaml 
services:
image: conda/conda-build-test
build:
context: .
dockerfile: ../Dockerfile
command: /root/conda-build/bin/conda-build /test
volumes:
- ./test/test:z,ro

This is how they run test and get the error...

cd test
DOCKER_BUILDKIT=0 docker compose build test
docker compose run test

@jaimergp
Copy link
Contributor

jaimergp commented Sep 8, 2023

I just merged a PR in conda-forge/staged-recipes that switches from conda mambabuild to conda build with conda-libmamba-solver and it seems to work correctly. See latest comments conda-forge/staged-recipes#23923 for some examples. Those are using conda=23.7.3, conda-build=3.26.1 and conda-libmamba-solver=23.7.0, so there's no intrinsic incompatibility between those.

That leads me to thinking that the error is triggered by something in their system. Can I get more details on:

  • Which packages are distributed in that custom conda-build-0.0.0-Linux-x86_64.sh installer?
  • What's in the target test/ directory? Is there a chance we can see those recipes?

The traceback hints at a problem with get_upstream_pins. This call happens early in the conda build process. Is this happening right away after calling conda build or after successfully processing some packages?

Thanks!

@jezdez
Copy link
Member Author

jezdez commented Sep 8, 2023

@jaimergp I've added more info to the issue description above to help with debugging!

@h-vetinari h-vetinari mentioned this issue Sep 15, 2023
Closed
@hsparra
Copy link

hsparra commented Sep 15, 2023

This is happening right after the calling conda-build.

@hsparra
Copy link

hsparra commented Oct 6, 2023 •
edited

It seems like something with channel aliases may be involved. After lots of testing with different condarc configurations, a minimal condarc where conda-build works has been found along with several minimal configurations that cause the error.

This minimal condarc works:

channels:
- https://conda.anaconda.xxxxx/anaconda
solver: libmamba

Note: No channel Alias.

All following condarc combinations DO NOT work.

channel_alias: https://conda.anaconda.xxxxx/
channels:
- https://conda.anaconda.xxxxx/anaconda
solver: libmamba

channel_alias: https://conda.anaconda.xxxxx/
channels:
  - anaconda
solver: libmamba

In the following case, there is no channel alias but modified default channels

default_channels:
- https://conda.anaconda.xxxxx/anaconda
channels:
- defaults
solver: libmamba

channel_alias: https://conda.anaconda.xxxxx/
default_channels:
- anaconda
channels:
- defaults
solver: libmamba

@tysonclugg tysonclugg mentioned this issue Oct 12, 2023
Open
@jaimergp
Copy link
Contributor

I've been trying to reproduce this with publicly available resources that mimic the reported configuration but I've had no luck despite matching almost everything 1:1. Details below.

  • My base OS is a Docker container for linux-aarch64 (my machine is Apple Silicon). It should make no difference. It can be started like this:
$ docker run -it --rm --platform=linux/aarch64 ghcr.io/conda/conda-ci:main-linux-python3.10-conda-forge bash
  • I've configured the Docker container to report the following info, matching the versions in the first message:
$ conda info
                                                                                                                                                                                                                              
     active environment : base                                                                                                                                                                                                
    active env location : /opt/conda                                                                                                                                                                                          
            shell level : 1                                                                                                                                                                                                   
       user config file : /home/test_user/.condarc                                                                                                                                                                            
 populated config files : /opt/conda/.condarc                                                                                                                                                                                 
          conda version : 23.7.3                                                                                                                                                                                              
    conda-build version : 3.26.1.dev37                                                                                                                                                                                        
         python version : 3.10.12.final.0                                                                                                                                                                                     
       virtual packages : __archspec=1=aarch64                                                                                                                                                                                
                          __glibc=2.31=0                                                                                                                                                                                      
                          __linux=5.15.49=0
                          __unix=0=0
       base environment : /opt/conda  (read only)
      conda av data dir : /opt/conda/etc/conda
  conda av metadata url : None
           channel URLs : https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge/linux-aarch64
                          https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge/noarch
          package cache : /opt/conda/pkgs
                          /home/test_user/.conda/pkgs
       envs directories : /home/test_user/.conda/envs
                          /opt/conda/envs
               platform : linux-aarch64
             user-agent : conda/23.7.3 requests/2.31.0 CPython/3.10.12 Linux/5.15.49-linuxkit-pr debian/11 glibc/2.31 solver/libmamba conda-libmamba-solver/23.7.0 libmambapy/1.4.1
                UID:GID : 1001:1001
             netrc file : None
           offline mode : False
  • The image .condarc results in the following config. Note I am using the only public Anaconda mirror I know.
==> /opt/conda/.condarc <==
channel_alias: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
channels:
  - defaults
default_channels:
  - conda-forge
solver: libmamba

  • I will try to build the following test recipe (it also uses python, which is the package raising KeyError).

  • I then ran conda build <test-recipe>/ and it builds successfully:

$ conda-build tests/data/conda_build_recipes/jedi
/opt/conda/bin/conda-build:8: DeprecationWarning: conda_build.cli.main_build.main is deprecated and will be removed in 4.0.0. Use `conda build` instead.
  sys.exit(main())
WARNING: No numpy version specified in conda_build_config.yaml.  Falling back to default numpy value of 1.22
Adding in variants from internal_defaults
Attempting to finalize metadata for jedi
Reloading output folder (local): ...working... done
Getting pinned dependencies: ...working... done
Reloading output folder (local): ...working... done
Getting pinned dependencies: ...working... done
BUILD START: ['jedi-0.17.2-py310_1.tar.bz2']
Reloading output folder (local): ...working... done
Solving environment (_h_env): ...working... done

## Package Plan ##

  environment location: /home/test_user/conda-bld/jedi_1697102244495/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla


The following NEW packages will be INSTALLED:

    _openmp_mutex:         4.5-2_gnu                 
    bzip2:                 1.0.8-hf897c2e_4          
    ca-certificates:       2023.7.22-hcefe29a_0      
    ld_impl_linux-aarch64: 2.40-h2d8c526_0           
    libffi:                3.4.2-h3557bc0_5          
    libgcc-ng:             13.2.0-hf8544c7_2         
    libgomp:               13.2.0-hf8544c7_2         
    libnsl:                2.0.0-h31becfc_1          
    libsqlite:             3.43.2-h194ca79_0         
    libuuid:               2.38.1-hb4cce97_0         
    libzlib:               1.2.13-h31becfc_5         
    ncurses:               6.4-h2e1726e_0            
    openssl:               3.1.3-h31becfc_0          
    pip:                   23.2.1-pyhd8ed1ab_0       
    python:                3.10.12-hbbe8eec_0_cpython
    readline:              8.2-h8fc344f_1            
    setuptools:            68.2.2-pyhd8ed1ab_0       
    tk:                    8.6.13-h194ca79_0         
    tzdata:                2023c-h71feb2d_0          
    wheel:                 0.41.2-pyhd8ed1ab_0       
    xz:                    5.2.6-h9cdd2b7_0          

Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
Reloading output folder (local): ...working... done
Solving environment (_test_env): ...working... done
Source cache directory is: /home/test_user/conda-bld/src_cache
Found source in cache: jedi-0.17.2_86ed7d9b75.tar.gz
Extracting download
source tree in: /home/test_user/conda-bld/jedi_1697102244495/work
export PREFIX=/home/test_user/conda-bld/jedi_1697102244495/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla
export BUILD_PREFIX=/home/test_user/conda-bld/jedi_1697102244495/_build_env
export SRC_DIR=/home/test_user/conda-bld/jedi_1697102244495/work
Processing $SRC_DIR
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Building wheels for collected packages: jedi
  Building wheel for jedi (setup.py): started
  Building wheel for jedi (setup.py): finished with status 'done'
  Created wheel for jedi: filename=jedi-0.17.2-py2.py3-none-any.whl size=1424265 sha256=c1dabafe95aca4a06558a4534eb7843df50600e1d4e926ced739635375422e1b
  Stored in directory: /tmp/pip-ephem-wheel-cache-q52fgcmk/wheels/d4/ef/c8/938d955f0557f0fd6af3245376304ee5ce33619bf24850a065
Successfully built jedi
Installing collected packages: jedi
Successfully installed jedi-0.17.2

Resource usage statistics from building jedi:
   Process count: 4
   CPU time: Sys=0:00:00.7, User=0:00:00.9
   Memory: 82.4M
   Disk usage: 28.4K
   Time elapsed: 0:00:04.1


Packaging jedi
Packaging jedi-0.17.2-py310_1
compiling .pyc files...
number of files: 1765
Fixing permissions
INFO :: Time taken to mark (prefix)
        0 replacements in 0 files was 0.14 seconds
Package verification results:
-----------------------------
/tmp/tmpkvgw7lz9/jedi-0.17.2-py310_1.tar.bz2: C1115 Found invalid license "None" in info/index.json
DEBUG:conda_index.index:found subdirs {'noarch', 'linux-aarch64'}
DEBUG:conda_index.index.sqlitecache:CondaIndexCache channel_root=/home/test_user/conda-bld, subdir=linux-aarch64 db_filename=/home/test_user/conda-bld/linux-aarch64/.cache/cache.db cache_is_brand_new=False
DEBUG:conda_index.index.sqlitecache:linux-aarch64 listdir
DEBUG:conda_index.index.sqlitecache:linux-aarch64 save fs state
DEBUG:conda_index.index:linux-aarch64 find packages to extract
DEBUG:conda_index.index:linux-aarch64 extract 1 packages
DEBUG:conda_index.index.sqlitecache:Found jedi-0.17.2-py310_1.tar.bz2 in cache
INFO:conda_index.index:linux-aarch64 cached 962 KB from 1 packages at 5.19 GB/second
DEBUG:conda_index.index.sqlitecache:CondaIndexCache channel_root=/home/test_user/conda-bld, subdir=noarch db_filename=/home/test_user/conda-bld/noarch/.cache/cache.db cache_is_brand_new=False
DEBUG:conda_index.index.sqlitecache:noarch listdir
DEBUG:conda_index.index.sqlitecache:noarch save fs state
DEBUG:conda_index.index:noarch find packages to extract
DEBUG:conda_index.index:noarch extract 0 packages
INFO:conda_index.index:noarch cached 0 B from 0 packages at 0 B/second
INFO:conda_index.index:Subdir: linux-aarch64 Gathering repodata
DEBUG:conda_index.index.sqlitecache:CondaIndexCache channel_root=/home/test_user/conda-bld, subdir=linux-aarch64 db_filename=/home/test_user/conda-bld/linux-aarch64/.cache/cache.db cache_is_brand_new=False
INFO:conda_index.index:linux-aarch64 Writing pre-patch repodata
DEBUG:conda_index.index:_maybe_write /home/test_user/conda-bld/linux-aarch64/repodata_from_packages.json to /home/test_user/conda-bld/linux-aarch64/repodata_from_packages.json
INFO:conda_index.index:linux-aarch64 Applying patch instructions
INFO:conda_index.index:linux-aarch64 Writing patched repodata
DEBUG:conda_index.index:linux-aarch64 write patched repodata
DEBUG:conda_index.index:_maybe_write /home/test_user/conda-bld/linux-aarch64/repodata.json to /home/test_user/conda-bld/linux-aarch64/repodata.json
INFO:conda_index.index:linux-aarch64 Building current_repodata subset
DEBUG:conda_index.index:linux-aarch64 build current_repodata
INFO:conda_index.index:linux-aarch64 Writing current_repodata subset
DEBUG:conda_index.index:linux-aarch64 write current_repodata
DEBUG:conda_index.index:_maybe_write /home/test_user/conda-bld/linux-aarch64/current_repodata.json to /home/test_user/conda-bld/linux-aarch64/current_repodata.json
INFO:conda_index.index:linux-aarch64 Writing index HTML
DEBUG:conda_index.index:linux-aarch64 write index.html
DEBUG:conda_index.index:_maybe_write /home/test_user/conda-bld/linux-aarch64/index.html to /home/test_user/conda-bld/linux-aarch64/index.html
DEBUG:conda_index.index:linux-aarch64 finish
INFO:conda_index.index:Completed linux-aarch64
INFO:conda_index.index:Subdir: noarch Gathering repodata
DEBUG:conda_index.index.sqlitecache:CondaIndexCache channel_root=/home/test_user/conda-bld, subdir=noarch db_filename=/home/test_user/conda-bld/noarch/.cache/cache.db cache_is_brand_new=False
INFO:conda_index.index:noarch Writing pre-patch repodata
DEBUG:conda_index.index:_maybe_write /home/test_user/conda-bld/noarch/repodata_from_packages.json to /home/test_user/conda-bld/noarch/repodata_from_packages.json
INFO:conda_index.index:noarch Applying patch instructions
INFO:conda_index.index:noarch Writing patched repodata
DEBUG:conda_index.index:noarch write patched repodata
DEBUG:conda_index.index:_maybe_write /home/test_user/conda-bld/noarch/repodata.json to /home/test_user/conda-bld/noarch/repodata.json
INFO:conda_index.index:noarch Building current_repodata subset
DEBUG:conda_index.index:noarch build current_repodata
INFO:conda_index.index:noarch Writing current_repodata subset
DEBUG:conda_index.index:noarch write current_repodata
DEBUG:conda_index.index:_maybe_write /home/test_user/conda-bld/noarch/current_repodata.json to /home/test_user/conda-bld/noarch/current_repodata.json
INFO:conda_index.index:noarch Writing index HTML
DEBUG:conda_index.index:noarch write index.html
DEBUG:conda_index.index:_maybe_write /home/test_user/conda-bld/noarch/index.html to /home/test_user/conda-bld/noarch/index.html
DEBUG:conda_index.index:noarch finish
INFO:conda_index.index:Completed noarch
DEBUG:conda_index.index:found subdirs {'noarch', 'linux-aarch64'}
INFO:conda_index.index:Channeldata subdir: linux-aarch64
DEBUG:conda_index.index:linux-aarch64 read repodata
DEBUG:conda_index.index.sqlitecache:CondaIndexCache channel_root=/home/test_user/conda-bld, subdir=linux-aarch64 db_filename=/home/test_user/conda-bld/linux-aarch64/.cache/cache.db cache_is_brand_new=False
DEBUG:conda_index.index:linux-aarch64 channeldata finished
INFO:conda_index.index:Channeldata subdir: noarch
DEBUG:conda_index.index:noarch read repodata
DEBUG:conda_index.index.sqlitecache:CondaIndexCache channel_root=/home/test_user/conda-bld, subdir=noarch db_filename=/home/test_user/conda-bld/noarch/.cache/cache.db cache_is_brand_new=False
DEBUG:conda_index.index:noarch channeldata finished
DEBUG:conda_index.index:_maybe_write /home/test_user/conda-bld/index.html to /home/test_user/conda-bld/index.html
DEBUG:conda_index.index:write channeldata
DEBUG:conda_index.index:_maybe_write /home/test_user/conda-bld/channeldata.json to /home/test_user/conda-bld/channeldata.json
TEST START: /home/test_user/conda-bld/linux-aarch64/jedi-0.17.2-py310_1.tar.bz2
DEBUG:conda_index.index:found subdirs {'noarch', 'linux-aarch64'}
DEBUG:conda_index.index.sqlitecache:CondaIndexCache channel_root=/home/test_user/conda-bld, subdir=linux-aarch64 db_filename=/home/test_user/conda-bld/linux-aarch64/.cache/cache.db cache_is_brand_new=False
DEBUG:conda_index.index.sqlitecache:linux-aarch64 listdir
DEBUG:conda_index.index.sqlitecache:linux-aarch64 save fs state
DEBUG:conda_index.index:linux-aarch64 find packages to extract
DEBUG:conda_index.index:linux-aarch64 extract 0 packages
INFO:conda_index.index:linux-aarch64 cached 0 B from 0 packages at 0 B/second
DEBUG:conda_index.index.sqlitecache:CondaIndexCache channel_root=/home/test_user/conda-bld, subdir=noarch db_filename=/home/test_user/conda-bld/noarch/.cache/cache.db cache_is_brand_new=False
DEBUG:conda_index.index.sqlitecache:noarch listdir
DEBUG:conda_index.index.sqlitecache:noarch save fs state
DEBUG:conda_index.index:noarch find packages to extract
DEBUG:conda_index.index:noarch extract 0 packages
INFO:conda_index.index:noarch cached 0 B from 0 packages at 0 B/second
INFO:conda_index.index:Subdir: linux-aarch64 Gathering repodata
DEBUG:conda_index.index.sqlitecache:CondaIndexCache channel_root=/home/test_user/conda-bld, subdir=linux-aarch64 db_filename=/home/test_user/conda-bld/linux-aarch64/.cache/cache.db cache_is_brand_new=False
INFO:conda_index.index:linux-aarch64 Writing pre-patch repodata
DEBUG:conda_index.index:_maybe_write /home/test_user/conda-bld/linux-aarch64/repodata_from_packages.json to /home/test_user/conda-bld/linux-aarch64/repodata_from_packages.json
INFO:conda_index.index:linux-aarch64 Applying patch instructions
INFO:conda_index.index:linux-aarch64 Writing patched repodata
DEBUG:conda_index.index:linux-aarch64 write patched repodata
DEBUG:conda_index.index:_maybe_write /home/test_user/conda-bld/linux-aarch64/repodata.json to /home/test_user/conda-bld/linux-aarch64/repodata.json
INFO:conda_index.index:linux-aarch64 Building current_repodata subset
DEBUG:conda_index.index:linux-aarch64 build current_repodata
INFO:conda_index.index:linux-aarch64 Writing current_repodata subset
DEBUG:conda_index.index:linux-aarch64 write current_repodata
DEBUG:conda_index.index:_maybe_write /home/test_user/conda-bld/linux-aarch64/current_repodata.json to /home/test_user/conda-bld/linux-aarch64/current_repodata.json
INFO:conda_index.index:linux-aarch64 Writing index HTML
DEBUG:conda_index.index:linux-aarch64 write index.html
DEBUG:conda_index.index:_maybe_write /home/test_user/conda-bld/linux-aarch64/index.html to /home/test_user/conda-bld/linux-aarch64/index.html
DEBUG:conda_index.index:linux-aarch64 finish
INFO:conda_index.index:Completed linux-aarch64
INFO:conda_index.index:Subdir: noarch Gathering repodata
DEBUG:conda_index.index.sqlitecache:CondaIndexCache channel_root=/home/test_user/conda-bld, subdir=noarch db_filename=/home/test_user/conda-bld/noarch/.cache/cache.db cache_is_brand_new=False
INFO:conda_index.index:noarch Writing pre-patch repodata
DEBUG:conda_index.index:_maybe_write /home/test_user/conda-bld/noarch/repodata_from_packages.json to /home/test_user/conda-bld/noarch/repodata_from_packages.json
INFO:conda_index.index:noarch Applying patch instructions
INFO:conda_index.index:noarch Writing patched repodata
DEBUG:conda_index.index:noarch write patched repodata
DEBUG:conda_index.index:_maybe_write /home/test_user/conda-bld/noarch/repodata.json to /home/test_user/conda-bld/noarch/repodata.json
INFO:conda_index.index:noarch Building current_repodata subset
DEBUG:conda_index.index:noarch build current_repodata
INFO:conda_index.index:noarch Writing current_repodata subset
DEBUG:conda_index.index:noarch write current_repodata
DEBUG:conda_index.index:_maybe_write /home/test_user/conda-bld/noarch/current_repodata.json to /home/test_user/conda-bld/noarch/current_repodata.json
INFO:conda_index.index:noarch Writing index HTML
DEBUG:conda_index.index:noarch write index.html
DEBUG:conda_index.index:_maybe_write /home/test_user/conda-bld/noarch/index.html to /home/test_user/conda-bld/noarch/index.html
DEBUG:conda_index.index:noarch finish
INFO:conda_index.index:Completed noarch
DEBUG:conda_index.index:found subdirs {'noarch', 'linux-aarch64'}
INFO:conda_index.index:Channeldata subdir: linux-aarch64
DEBUG:conda_index.index:linux-aarch64 read repodata
DEBUG:conda_index.index.sqlitecache:CondaIndexCache channel_root=/home/test_user/conda-bld, subdir=linux-aarch64 db_filename=/home/test_user/conda-bld/linux-aarch64/.cache/cache.db cache_is_brand_new=False
DEBUG:conda_index.index:linux-aarch64 channeldata finished
INFO:conda_index.index:Channeldata subdir: noarch
DEBUG:conda_index.index:noarch read repodata
DEBUG:conda_index.index.sqlitecache:CondaIndexCache channel_root=/home/test_user/conda-bld, subdir=noarch db_filename=/home/test_user/conda-bld/noarch/.cache/cache.db cache_is_brand_new=False
DEBUG:conda_index.index:noarch channeldata finished
DEBUG:conda_index.index:_maybe_write /home/test_user/conda-bld/index.html to /home/test_user/conda-bld/index.html
DEBUG:conda_index.index:write channeldata
DEBUG:conda_index.index:_maybe_write /home/test_user/conda-bld/channeldata.json to /home/test_user/conda-bld/channeldata.json
Adding in variants from /tmp/tmp86g7_0xu/info/recipe/conda_build_config.yaml
Renaming work directory '/home/test_user/conda-bld/jedi_1697102244495/work' to '/home/test_user/conda-bld/jedi_1697102244495/work_moved_jedi-0.17.2-py310_1_linux-aarch64'
shutil.move(work)=/home/test_user/conda-bld/jedi_1697102244495/work, dest=/home/test_user/conda-bld/jedi_1697102244495/work_moved_jedi-0.17.2-py310_1_linux-aarch64)
Reloading output folder (local): ...working... done
Solving environment (_test_env): ...working... done

## Package Plan ##

  environment location: /home/test_user/conda-bld/jedi_1697102244495/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol


The following NEW packages will be INSTALLED:

    _openmp_mutex:         4.5-2_gnu                       
    bzip2:                 1.0.8-hf897c2e_4                
    ca-certificates:       2023.7.22-hcefe29a_0            
    jedi:                  0.17.2-py310_1             local
    ld_impl_linux-aarch64: 2.40-h2d8c526_0                 
    libffi:                3.4.2-h3557bc0_5                
    libgcc-ng:             13.2.0-hf8544c7_2               
    libgomp:               13.2.0-hf8544c7_2               
    libnsl:                2.0.0-h31becfc_1                
    libsqlite:             3.43.2-h194ca79_0               
    libuuid:               2.38.1-hb4cce97_0               
    libzlib:               1.2.13-h31becfc_5               
    ncurses:               6.4-h2e1726e_0                  
    openssl:               3.1.3-h31becfc_0                
    parso:                 0.7.1-pyh9f0ad1d_0              
    pip:                   23.2.1-pyhd8ed1ab_0             
    python:                3.10.12-hbbe8eec_0_cpython      
    python_abi:            3.10-4_cp310                    
    readline:              8.2-h8fc344f_1                  
    setuptools:            68.2.2-pyhd8ed1ab_0             
    tk:                    8.6.13-h194ca79_0               
    tzdata:                2023c-h71feb2d_0                
    wheel:                 0.41.2-pyhd8ed1ab_0             
    xz:                    5.2.6-h9cdd2b7_0                

Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
export PREFIX=/home/test_user/conda-bld/jedi_1697102244495/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol
export SRC_DIR=/home/test_user/conda-bld/jedi_1697102244495/test_tmp
import: 'jedi'
import: 'jedi.api'
import: 'jedi.common'
import: 'jedi.inference'
import: 'jedi.inference.compiled'
import: 'jedi.inference.compiled.subprocess'
import: 'jedi.inference.gradual'
import: 'jedi.inference.value'
import: 'jedi.plugins'
import: 'jedi'
import: 'jedi.api'
import: 'jedi.common'
import: 'jedi.inference'
import: 'jedi.inference.compiled'
import: 'jedi.inference.compiled.subprocess'
import: 'jedi.inference.gradual'
import: 'jedi.inference.value'
import: 'jedi.plugins'

Resource usage statistics from testing jedi:
   Process count: 2
   CPU time: Sys=0:00:00.0, User=0:00:00.0
   Memory: 10.5M
   Disk usage: 16B
   Time elapsed: 0:00:02.0


TEST END: /home/test_user/conda-bld/linux-aarch64/jedi-0.17.2-py310_1.tar.bz2
Renaming work directory '/home/test_user/conda-bld/jedi_1697102244495/work' to '/home/test_user/conda-bld/jedi_1697102244495/work_moved_jedi-0.17.2-py310_1_linux-aarch64_main_build_loop'
shutil.move(work)=/home/test_user/conda-bld/jedi_1697102244495/work, dest=/home/test_user/conda-bld/jedi_1697102244495/work_moved_jedi-0.17.2-py310_1_linux-aarch64_main_build_loop)
# Automatic uploading is disabled
# If you want to upload package(s) to anaconda.org later, type:


# To have conda build upload to anaconda.org automatically, use
# conda config --set anaconda_upload yes
anaconda upload \
    /home/test_user/conda-bld/linux-aarch64/jedi-0.17.2-py310_1.tar.bz2
anaconda_upload is not set.  Not uploading wheels: []

INFO :: The inputs making up the hashes for the built packages are as follows:
{
  "jedi-0.17.2-py310_1": {
    "recipe": {}
  }
}


####################################################################################
Resource usage summary:

Total time: 0:01:08.0
CPU usage: sys=0:00:00.7, user=0:00:00.9
Maximum memory usage observed: 82.4M
Total disk usage observed (not including envs): 28.4K


####################################################################################
Source and build intermediates have been left in /home/test_user/conda-bld.
There are currently 3 accumulated.
To remove them, you can run the ```conda build purge``` command

FWIW, I also tried the most recent versions of every component (conda 23.9.0, conda-build 23.7.0, conda-libmamba-solver 23.9.1, libmambapy 1.5.1) and it works too.

I'll keep trying other combinations but I would have expected to reproduce this more easily.

@jaimergp
Copy link
Contributor

One thing to notice is that the line raising KeyError is part of this code block:

conda-build/conda_build/render.py

Lines 372 to 391 in c2bfe2b

# ran through all pkgs_dirs, and did not find package or folder. Download it.
# TODO: this is a vile hack reaching into conda's internals. Replace with
# proper conda API when available.
if not pkg_loc:
try:
pkg_record = [_ for _ in index if _.dist_name == pkg_dist][0]
# the conda 4.4 API uses a single `link_prefs` kwarg
# whereas conda 4.3 used `index` and `link_dists` kwargs
pfe = ProgressiveFetchExtract(link_prefs=(index[pkg_record],))
except TypeError:
# TypeError: __init__() got an unexpected keyword argument 'link_prefs'
pfe = ProgressiveFetchExtract(link_dists=[pkg], index=index)
with utils.LoggingContext():
pfe.execute()
for pkg_dir in pkgs_dirs:
_loc = os.path.join(pkg_dir, index[pkg].fn)
if os.path.isfile(_loc):
pkg_loc = _loc
break
pkg_files[pkg] = pkg_loc, pkg_dist

"Vile hack" doesn't give me too much confidence and is also a "last resort" strategy, part of the run_exports / pins metadata calculation. Since I matched the config but had to come up with a demo recipe that might not representative of the code paths being triggered in conda-build... is there a chance we can get some insight into what kind of recipes are being built? Namely:

  • Is it single output or does it have multiple outputs? In other words, does it contain and outputs: key?
  • What kind of dependencies are present in build and host environments? Things like compilers or compiled libraries that might be exporting pins to run?
  • Is it using Jinja functions like pin_subpackage or pin_compatible?

Thanks!

@tysonclugg
Copy link

tysonclugg commented Oct 12, 2023 •
edited

I've been trying to reproduce this with publicly available resources that mimic the reported configuration but I've had no luck despite matching almost everything 1:1. Details below.

@jaimergp from my experience with this bug, you should test in an isolated network environment where you can talk to a private conda-forge mirror, but nothing else. If you have general internet access, you probably won't trigger the bug.

Edit: If network connectivity is indeed a factor, maybe we could integrate a tools like https://gitlab.freedesktop.org/hadess/resolvconf-override/ or https://github.com/pagekite/libunaccept into the test suite to test to ensure we test for regressions in this space. Monkey-patching the socket module is another option.

@jaimergp
Copy link
Contributor

After some fiddling it does reproduce with the proposed configuration above, but to hit the buggy code path you need a clean cache via conda clean -pty; otherwise the tarball is found earlier and it doesn't try to download it on the spot. I have found a simple fix that I'm going to submit to the CI to see if it breaks anything.

@jaimergp jaimergp mentioned this issue Oct 12, 2023
Merged
3 tasks
@hsparra hsparra mentioned this issue Nov 8, 2023
Closed
2 tasks
@jaimergp jaimergp mentioned this issue Nov 8, 2023
Merged
3 tasks
@jezdez jezdez added this to the 3.28.0 milestone Nov 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
source::anaconda created by members of Anaconda, Inc. type::bug describes erroneous operation, use severity::* to classify the type
Projects
🧭 Planning
Archived in project
Milestone
3.28.0
4 participants
@jezdez @tysonclugg @jaimergp @hsparra