Flaky CI tests: KeyError: 'md5'

[maresb]

Checklist

• I added a descriptive title
• I searched open reports and couldn't find a duplicate

What happened?

In the tests for PRs many jobs are failing with KeyError: 'md5' in the _reconstruct_fetch_actions function.

Rerunning the failed jobs typically fixes the error, but it still causes a lot of confusion.

I have also seen a less common error: RuntimeError: No user found in setup of test_private_lock[micromamba-${QUETZ_API_KEY}] from our Quetz server test fixture. ✔️

We should look into this and fix it, and if you're submitting a PR, it's not your fault!

Conda Info

No response

Conda Config

No response

Conda list

No response

Additional Context

No response

[maresb]

Off-topic:

The Quetz API key seems to be particularly problematic at the moment.

Locally I try:

cd tests
docker run --rm -it -v $PWD/quetz/dev_config.toml:/etc/config.toml:ro mambaorg/quetz quetz run --copy-conf /etc/config.toml --dev --host 0.0.0.0 /run/quetz

and then I see a matching line like

Test API key created for user "carol": 2069ab85c2b3490ca4d3dc3f339144f4

so it's not immediately clear what goes wrong. Maybe we should dump the logstream before failing so that we can look for particular errors.

[maresb]

I did a quick experiment in #348 to try and get some more info about the md5 key error.

It seems to have to do with the data you get when running a command like:

docker run --rm -it mambaorg/micromamba:1.3.1
micromamba create --prefix /tmp/prefix --dry-run --json -v --channel=conda-forge zlib

This generates a dictionary where actions.FETCH and actions.LINK are lists of 6 elements, a typical element looking like

{
    "build": "h166bdaf_4",
    "build_number": 4,
    "build_string": "h166bdaf_4",
    "channel": "https://conda.anaconda.org/conda-forge/linux-64",
    "constrains": null,
    "depends": [
        "libgcc-ng >=12",
        "libzlib 1.2.13 h166bdaf_4"
    ],
    "fn": "zlib-1.2.13-h166bdaf_4.tar.bz2",
    "license": "Zlib",
    "md5": "4b11e365c0275b808be78b30f904e295",
    "name": "zlib",
    "sha256": "282ce274ebe6da1fbd52efbb61bd5a93dec0365b14d64566e6819d1691b75300",
    "size": 94099,
    "subdir": "linux-64",
    "timestamp": 1665759636,
    "track_features": "",
    "url": "https://conda.anaconda.org/conda-forge/linux-64/zlib-1.2.13-h166bdaf_4.tar.bz2",
    "version": "1.2.13"
}

If you run micromamba install -c conda-forge zlib, then the metadata gets cached, and running the previous command has actions.FETCH empty and actions.LINK as before. Thus you get everything in link_only_names.

When the KeyError occurs, it seems that this cached metadata is somehow missing the md5 and sha256 keys:

{
  "build": "pyhd8ed1ab_0",
  "build_number": 0,
  "build_string": "pyhd8ed1ab_0",
  "channel": "https://conda.anaconda.org/conda-forge/noarch",
  "constrains": null,
  "depends": [
    "setuptools",
    "wheel",
    "python >=3.7"
  ],
  "fn": "pip-23.0-pyhd8ed1ab_0.conda",
  "license": "MIT",
  "name": "pip",
  "size": 1364220,
  "subdir": "noarch",
  "timestamp": 1675141351,
  "track_features": "",
  "url": "https://conda.anaconda.org/conda-forge/noarch/pip-23.0-pyhd8ed1ab_0.conda",
  "version": "23.0"
}

I don't have any idea what's causing those keys to go missing though.

[maresb]

Closed by #376