Cannot install on osx-arm64 due to ncurses issues

[xhochy]

Solution to issue cannot be found in the documentation.

• I checked the documentation.

Issue

The bootstrapping process of mambaforge currently fails on osx-arm64 with the following message:

+ bash Mambaforge-MacOSX-arm64.sh -b -p /Users/uwe/Development/conda-forge/tensorflow-feedstock-4/tensorflow-feedstock-osx_arm64_abseil_cpp20210324.2grpc_cpp1.45python3.8.____cpython/./miniforge3
PREFIX=/Users/uwe/Development/conda-forge/tensorflow-feedstock-4/tensorflow-feedstock-osx_arm64_abseil_cpp20210324.2grpc_cpp1.45python3.8.____cpython/miniforge3
Unpacking payload ...
Extracting "libnghttp2-1.47.0-he723fca_0.tar.bz2"
Extracting "python_abi-3.9-2_cp39.tar.bz2"
Extracting "idna-3.3-pyhd8ed1ab_0.tar.bz2"
Extracting "reproc-14.2.3-h3422bc3_0.tar.bz2"
Extracting "libcxx-14.0.3-h6a5c8ee_0.tar.bz2"
Extracting "six-1.16.0-pyh6c4a22f_0.tar.bz2"
Extracting "tqdm-4.64.0-pyhd8ed1ab_0.tar.bz2"
Extracting "urllib3-1.26.9-pyhd8ed1ab_0.tar.bz2"
Extracting "cryptography-37.0.2-py39h4cb6239_0.tar.bz2"
Extracting "yaml-cpp-0.6.3-hc88da5d_4.tar.bz2"
Extracting "reproc-cpp-14.2.3-hbdafb3b_0.tar.bz2"
Extracting "libmambapy-0.22.1-py39h43fb93c_1.tar.bz2"
Extracting "xz-5.2.5-h642e427_1.tar.bz2"
Extracting "krb5-1.19.3-hf9b2bbe_0.tar.bz2"
Extracting "wheel-0.37.1-pyhd8ed1ab_0.tar.bz2"
Extracting "conda-4.12.0-py39h2804cbe_0.tar.bz2"
Extracting "lzo-2.10-h642e427_1000.tar.bz2"
Extracting "pybind11-abi-4-hd8ed1ab_3.tar.bz2"
Extracting "python-3.9.12-hfc7342c_1_cpython.tar.bz2"
Extracting "lz4-c-1.9.3-hbdafb3b_1.tar.bz2"
Extracting "pycparser-2.21-pyhd8ed1ab_0.tar.bz2"
Extracting "c-ares-1.18.1-h3422bc3_0.tar.bz2"
Extracting "icu-70.1-h6b3803e_0.tar.bz2"
Extracting "colorama-0.4.4-pyh9f0ad1d_0.tar.bz2"
Extracting "libxml2-2.9.14-h035c1df_0.tar.bz2"
Extracting "requests-2.27.1-pyhd8ed1ab_0.tar.bz2"
Extracting "libedit-3.1.20191231-hc8eb9b7_2.tar.bz2"
Extracting "pyopenssl-22.0.0-pyhd8ed1ab_0.tar.bz2"
Extracting "ruamel_yaml-0.15.80-py39h5161555_1006.tar.bz2"
Extracting "brotlipy-0.7.0-py39hb18efdd_1004.tar.bz2"
Extracting "libev-4.33-h642e427_1.tar.bz2"
Extracting "zlib-1.2.11-h90dfc92_1014.tar.bz2"
Extracting "pycosat-0.6.3-py39hb18efdd_1010.tar.bz2"
Extracting "ca-certificates-2022.5.18.1-h4653dfc_0.tar.bz2"
Extracting "setuptools-62.3.2-py39h2804cbe_0.tar.bz2"
Extracting "libsolv-0.7.22-h1280f1d_0.tar.bz2"
Extracting "pysocks-1.7.1-py39h2804cbe_5.tar.bz2"
Extracting "tzdata-2022a-h191b570_0.tar.bz2"
Extracting "certifi-2022.5.18.1-py39h2804cbe_0.tar.bz2"
Extracting "libarchive-3.5.2-h69ec738_2.tar.bz2"
Extracting "zstd-1.5.2-hd705a24_1.tar.bz2"
Extracting "readline-8.1-hedafd6a_0.tar.bz2"
Extracting "libzlib-1.2.11-h90dfc92_1014.tar.bz2"
Extracting "libiconv-1.16-h642e427_0.tar.bz2"
Extracting "bzip2-1.0.8-h3422bc3_4.tar.bz2"
Extracting "libffi-3.4.2-h3422bc3_5.tar.bz2"
Extracting "ncurses-6.3-h07bb92c_1.tar.bz2"
Extracting "cffi-1.15.0-py39h52b1de0_0.tar.bz2"
Extracting "yaml-0.2.5-h3422bc3_2.tar.bz2"
Extracting "tk-8.6.12-he1e0b03_0.tar.bz2"
Extracting "charset-normalizer-2.0.12-pyhd8ed1ab_0.tar.bz2"
Extracting "openssl-1.1.1o-ha287fd2_0.tar.bz2"
Extracting "conda-package-handling-1.8.1-py39hb18efdd_1.tar.bz2"
Extracting "libcurl-7.83.1-h2fcd78c_0.tar.bz2"
Extracting "sqlite-3.38.5-h40dfcc0_0.tar.bz2"
Extracting "pip-22.0.4-pyhd8ed1ab_0.tar.bz2"
Extracting "mamba-0.22.1-py39hde45b87_1.tar.bz2"
Extracting "libmamba-0.22.1-hdb03daf_1.tar.bz2"
Extracting "libssh2-1.10.0-hb80f160_2.tar.bz2"

                                           __
          __  ______ ___  ____ _____ ___  / /_  ____ _
         / / / / __ `__ \/ __ `/ __ `__ \/ __ \/ __ `/
        / /_/ / / / / / / /_/ / / / / / / /_/ / /_/ /
       / .___/_/ /_/ /_/\__,_/_/ /_/ /_/_.___/\__,_/
      /_/

Encountered problems while solving:
  - nothing provides ncurses >=6.2,<6.3.0a0 needed by libedit-3.1.20191231-hc8eb9b7_2
  - nothing provides ncurses >=6.2,<6.3.0a0 needed by readline-8.1-hedafd6a_0
  - nothing provides ncurses >=6.2,<6.3.0a0 needed by libedit-3.1.20191231-hc8eb9b7_2
  - nothing provides ncurses >=6.2,<6.3.0a0 needed by readline-8.1-hedafd6a_0
  - nothing provides ncurses >=6.2,<6.3.0a0 needed by readline-8.1-hedafd6a_0
  - nothing provides ncurses >=6.2,<6.3.0a0 needed by libedit-3.1.20191231-hc8eb9b7_2
  - nothing provides ncurses >=6.2,<6.3.0a0 needed by readline-8.1-hedafd6a_0
  - nothing provides ncurses >=6.2,<6.3.0a0 needed by readline-8.1-hedafd6a_0
  - nothing provides ncurses >=6.2,<6.3.0a0 needed by readline-8.1-hedafd6a_0
  - nothing provides ncurses >=6.2,<6.3.0a0 needed by readline-8.1-hedafd6a_0
  - nothing provides ncurses >=6.2,<6.3.0a0 needed by readline-8.1-hedafd6a_0
  - nothing provides ncurses >=6.2,<6.3.0a0 needed by readline-8.1-hedafd6a_0
  - nothing provides ncurses >=6.2,<6.3.0a0 needed by readline-8.1-hedafd6a_0
  - nothing provides ncurses >=6.2,<6.3.0a0 needed by readline-8.1-hedafd6a_0
  - nothing provides ncurses >=6.2,<6.3.0a0 needed by readline-8.1-hedafd6a_0
  - nothing provides ncurses >=6.2,<6.3.0a0 needed by readline-8.1-hedafd6a_0
  - nothing provides ncurses >=6.2,<6.3.0a0 needed by readline-8.1-hedafd6a_0
  - nothing provides ncurses >=6.2,<6.3.0a0 needed by readline-8.1-hedafd6a_0
  - nothing provides ncurses >=6.2,<6.3.0a0 needed by readline-8.1-hedafd6a_0
  - nothing provides ncurses >=6.2,<6.3.0a0 needed by readline-8.1-hedafd6a_0
  - nothing provides ncurses >=6.2,<6.3.0a0 needed by readline-8.1-hedafd6a_0
  - nothing provides ncurses >=6.2,<6.3.0a0 needed by readline-8.1-hedafd6a_0
  - nothing provides ncurses >=6.2,<6.3.0a0 needed by readline-8.1-hedafd6a_0
  - nothing provides ncurses >=6.2,<6.3.0a0 needed by readline-8.1-hedafd6a_0
  - nothing provides ncurses >=6.2,<6.3.0a0 needed by readline-8.1-hedafd6a_0
  - nothing provides ncurses >=6.2,<6.3.0a0 needed by readline-8.1-hedafd6a_0
  - nothing provides ncurses >=6.2,<6.3.0a0 needed by readline-8.1-hedafd6a_0
  - nothing provides ncurses >=6.2,<6.3.0a0 needed by readline-8.1-hedafd6a_0
  - nothing provides ncurses >=6.2,<6.3.0a0 needed by readline-8.1-hedafd6a_0
  - nothing provides ncurses >=6.2,<6.3.0a0 needed by readline-8.1-hedafd6a_0
  - nothing provides ncurses >=6.2,<6.3.0a0 needed by libedit-3.1.20191231-hc8eb9b7_2

The environment can't be solved, aborting the operation
error    libmamba Could not solve for environment specs
critical libmamba UnsatisfiableError

Installed packages

Nothing yet installed.

Environment info

This has been run as part of the `build-locally.py` script and I have ensured that no `.condarc` is present.

[isuruf]

Duplicate of #255

[isuruf]

Maybe not

[xhochy]

• Using rosetta leads to the same issue with the osx-64 installer
• Using 4.12.0-1 also leads to the same
• It worked yesterday evening and broke overnight while doing the Tensorflow builds. I didn't touch the machine during that time.

[isuruf]

I've marked the latest release as a pre-release for now

[xhochy]

Verbose micromamba output: https://gist.github.com/xhochy/383f56ed5451c9bd81db82c0be56803e

Interesting bit:

• readline asks for ncurses>=6.2,<6.3
• We install ncurses=6.3

[xhochy]

This also matches what is in the about.json of the readline package.

[xhochy]

I've marked the latest release as a pre-release for now

The now latest works again.

[isuruf]

micromamba is supposed to use the cache file which has the corrected dependencies, but is not.

[isuruf]

cc @wolfv, @chrisburr

[chrisburr]

Do you have tokens in ~/.continuum/anaconda-client/tokens? I can reproduce the issue with tokens but not without them.

See also mamba-org/mamba#1575

[isuruf]

Yes, I can reproduce with the tokens.

[wolfv]

I'll take a look at this today.

I think instead of looking at the URL at all (in the package cache, to determine if we have to redownload) – we will rely on the SHA256 which should be a perfect indicator of the actual contents of the package.

Although I also thought that we were already comparing "cleaned" (ie. cleaned of tokens) URLs.

[wolfv]

Hmm, I am looking at this issue and the other one (#255) and I am wondering how we want to tackle that. Shouldn't we rebuild the readline package to take into account the new ncurses pinning and use that inside the miniforge packages?

It looks to me as if we're installing packages with "conflicting" metadata.

Which cache file do you mean @isuruf? I see env.txt (list of matchspecs), urls and urls.txt.

Would be happy to fix this.

We could also force to install from the urls.txt as properly "explicit" installation (completely bypassing teh solver) but I think it will lead to problems down the road with the metadata that's installed.

[isuruf]

There is a cache/<hash>.json file with the correct dependencies.

[wolfv]

OK, got it ... that explains everything.

We do use the token value to calculate the MD5 hash of the full URL (incl. token), so the hash is different with token. The idea was that that the repodata contents could be different depending on wether a user has permissions on certain packages or not. But since this seems to break constructor it's probably better to modify this behavior.

[isuruf]

Since constructor passes --offline to the install step, maybe you can ignore the token value when --offline is used.

[isuruf]

Maybe not, since that's going to break --offline.

[wolfv]

I think we can just ignore the token value to calculate the hash and people who rely on different packages based on token value need to clean their caches using micromamba clean -i.

[wolfv]

I'll finish this tomorrow, but I think this PR should fix the issue: mamba-org/mamba#1696

[wolfv]

Actually, we could use the urls.txt to find out the channel urls and then read the proper values from the cache/<hash>.json files and put the proper values in the repodata_record.json that is placed in the packages.

That would also mean we could leave the hashing function in micromamba as it is. I think I can fix that tomorrow and make a release.

[isuruf]

@xhochy, can you check the artifacts from #313?

[isuruf]

Ping @xhochy

[heino]

As mentioned in #255 (comment), we have the same issue on Redhat RHEL 8.5 on x86-64