Start using conda's libmamba solver in CI

[jakirkham]

Go ahead and opt-in to using conda's libmamba solver. This will soon become the default. So it would be good to get a head start using this.

---

Checklist

• Used a personal fork of the feedstock to propose changes
• Bumped the build number (if the version is unchanged)
• Reset the build number to 0 (if the version changed)
• Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
• Ensured the license file is being packaged.

[conda-forge-webservices]

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

[jakirkham]

@conda-forge/core please share your thoughts on this 🙂

[beckermr]

PR looks great! We do have mamba instead of conda specified in a bunch of spots in smithy and the like. So this PR alone won't fully switch us over. That's fine OFC. I just wanted to mention it for clarity and those following along!

[beckermr]

@jaimergp you had a recommendation for a minimum stable version of the libmamba solver. Are we going to be consistent with that by pulling the latest version? Should we hardcode a >= constraint somewhere?

[jakirkham]

Thanks Matt! 🙏

We do have mamba instead of conda specified in a bunch of spots in smithy and the like. So this PR alone won't fully switch us over. That's fine OFC. I just wanted to mention it for clarity and those following along!

Definitely true. Wanted to start here to make sure we have this piece handled

It's easy to explore the next step in a conda-smithy PR (and test it in a feedstock) before rolling that change out more generally

Are we going to be consistent with that by pulling the latest version? Should we hardcode a >= constraint somewhere?

Would it be ok to stick with the version set in the installer? Or do we actually need to specify this in more places (like is there risk of it being downgraded)?

[beckermr]

Another wrinkle is that we might be upgrading deps and envs on the fly when running jobs even if we have a version in the installer?

[jakirkham]

We could run a libmamba solve to produce the contents of the installer artifact. Maybe this would ensure those changes are already baked in

[jaimergp]

@jaimergp you had a recommendation for a minimum stable version of the libmamba solver. Are we going to be consistent with that by pulling the latest version? Should we hardcode a >= constraint somewhere?

We've had very few regressions (if any) so far, so sticking to latest seems to be the best thing we can do for now. I am more concerned about the libmamba C++ lib version, because since 1.4.2 we have been seeing problems here and there. We are trying to patch these inconsistencies in the next conda-libmamba-solver release, and we will require libmamba 1.4.10 by then.

That said, since those inconsistencies are "edge-casey" (heavily constrained env updates, crazy ops like switching everything to a different channel, etc) and we are mostly using it for new environments and a single channel (conda-forge), we should be fine with what we are currently shipping in Miniforge: 23.3.0.

[jaimergp]

The larger conversation is whether we start using periodically updated lockfiles to provision the installation instead of a blind "update all" that sometimes leaves us in uncertain spots, but yea, a topic for another issue.

[jakirkham]

Thanks Jaime! 🙏

So it sounds like there are no action items (for this PR) from your perspective. Or did I missing something?

[jaimergp]

Yep, no extra items I can think of. We just need to make sure we use conda instead of mamba in the right places. To that end I will submit a PR in conda-smithy similar to conda-forge/conda-smithy#1732.

[isuruf]

Can you add conda-libmamba-solver to run requirements?

[jaimergp]

I wonder whether we should also provide some configurability like in the strict channel priority setting:

conda-forge-ci-setup-feedstock/recipe/run_conda_forge_build_setup_linux

Lines 30 to 34 in 22b4448

	# Need strict priority for
	# - pypy as defaults is not fixed
	# - cos7 as defaults is not fixed
	# but ppl can turn this off
	conda config --env --set channel_priority $(cat ${FEEDSTOCK_ROOT}/conda-forge.yml | shyaml get-value channel_priority strict || echo strict)

, which might require a conda namespace in conda-forge.yml, similar to what conda-build has.

[beckermr]

How would that interact with condarc handling?

[jaimergp]

Whatever config source is last wins. The precedence is:

• config files, in this order: CONDA_ROOT/condarc < user condarc < CONDA_PREFIX/condarc
• environment variables
• command-line flags

What are the concerns for those interactions?

[beckermr]

Oh I meant our munging of the condarc. I was hoping we could do all of our changes to it in one spot in our CI system. IDK how many spots we have now.

[beckermr]

Also due to the John handles feedstocks, we cannot push to this PR directly. If we want to finish it while he is away, we'll need to move the code to another account that is not an org.

[jaimergp]

IDK how many spots we have now.

Yea, me neither 😂 This will need a deeper audit, but I can think of the following spots in our supply chain right now:

• Docker image (Linux only), which installs...
• ... Miniforge, that ships its own $CONDA_ROOT/condarc with the conda-forge channels.
• AFAIK, conda-smithy does not set any conda settings, but it does allow some configuration (channel_priority) for the default values in...
• conda-forge-ci-setup, which invokes conda config a few times.

Users could choose between conda-build and mambabuild before too, via conda-forge.yml, so it feels natural to allow that configurability here too.

If we want to finish it while he is away, we'll need to move the code to another account that is not an org.

I'll open a new PR and credit John in the commits.

[jaimergp]

Continues on #275

[jakirkham]

Thanks Jaime! 🙏