Migrate development workflow to Pixi

[VeckoTheGecko]

(posting now for visibility and to request feedback/pixi debugging help)

Overview

Fixes #10732

This PR migrates the dev workflow and CI for Xarray across to Pixi, providing the following benefits:

• Composable environments via dependency groups (in pixi called "features")
• Support for multiple environments
• Task running
• lock file support

See the original issue for more info.

Changes so far in this PR:

• Add pixi badge to readme
• Migrated most environment files to Pixi config in pixi.toml split apart into features that I thought were sensible . I left out environment-benchmarks.yml, environment-min-all-deps.yml, binder/environment as that has interactions with asv, the min-dep checker, and Binder - this PR is already big enough, and I think those should be explored another time.
  • I made the environments in Pixi have similar names as the original conda environments to ease migration
• Introduced a cache-pixi-lock.yml workflow (see below section "Considerations")
• Updated all CI
  • Fixed now! 98% there - for some reason the CI of Pixi is finding which pytest to be .pixi/envs/default/Scripts/pytest while local pixi run -e test-all-deps-py313 which pytest is finding .pixi/envs/test-all-deps-py313/bin/pytest (see test-pixi-dust branch, example action run) . Any ideas why @lucascolley ?
• Update RTD build
• Update contributing guidelines (see "Feedback wanted" section below)

I've tried to make the commits tidy to help with reviewing commit by commit, which might be easier. I also was quite diligent when migrating from the old env files to make sure versions were the same.

Testing instructions

Resources: Pixi Scipy 2025 talk | Docs: Manifest Reference

• pixi info -> show info about the pixi environments
• Build documentation: pixi run doc
• Run tests: pixi run test then choose the environment you want to run the tests in (or pixi run -e environment_name test)
  • Most often you'll want the test-all-deps-py313 environment (corresponding to the old environment ci/requirements/environment.yml)
• Run pre-commit: pixi run pre-commit
• Run mypy typing: pixi run typing

Enter an environment (equivalent to conda activate): pixi shell -e env_name
Exit an environment (equivalent to conda activate): exit or Ctrl+D

See all tasks: pixi run

Considerations

Lock files o' lock files

There was some interesting conversation in #10732 (comment) about lock files. To summarise:

We have two choices to handle the lock files, either (a) generate them in CI, or (b) commit them to the repo and periodically update them.

(a) generating in CI (done in this PR):

• add pixi.lock to .gitignore
• have a workflow which generates the lock file. Cache this under a key that is date + hash(pixi.toml)
• have all workflows restore this pixi.lock file for environment creation

Pros:

• lock file is only generated once a day and shared across workflows - saving 40s per run
• close to what was previously done (with daily caches)
• minimal changes to workflows (only need to add a few lines - cache-pixi-lock.yml is re-usable across different projects).

Cons:

• Mismatch of devs pixi.lock and what's in CI. Local developers need to periodically delete pixi.lock and regenerate it.
• Missed benefit of perfectly reproducible dev environments cross developers and with CI

(b) commit the lock files

(I think this is the gist of it)

• commit the lock file (now local devs and CI can use this lockfile) - around 40k lines
• add GitHub PR automation to automatically update the lockfile every 3 weeks
• Most of CI works from committed lockfile, but there can be a job bleeding-edge which runs every few days by taking the current lockfile, running an update, and then running tests. Any failures can be automatically reported in an issue
  • Then, to "resolve" that issue you can add a pin in the pixi.toml manifest and talk with upstream to see whats up

Pros:

• no need to generate lock files in CI
• perfectly reproducible dev environments cross developers and CI

Cons:

• there is a bunch of added complexity/maintainer burden to setting this up (automated workflows etc)

@lucascolley knows the full extent as he's been exploring this setup at Scipy

Conclusion

Approach (a) has minimal setup/maintenance with little downside. I think that it's a good solution for smaller projects in particular (we've adopted it at Parcels - cc @maxrjones might be interesting based on your comment )

Approach (b) is more robust if having the same environment between all devs is highly valued (@shoyer mentioned during a dev meeting that this would be good for xarray), but requires more setup.

I recommend we go for (a) as is done in this PR, and consider (b) separately .

@lucascolley would it be beneficial to do a write-up of all this on prefix.dev sometime to help guide others dealing with this? I'm happy to write or collab on a blog post.

Feedback wanted: To what extent do we promote Conda dev workflows

Yeah - I don't know. In the projects I'm working on I've gone full Pixi, but those are smaller projects.

I've deleted the old environment files to avoid duplication, but can re-add them to the extent which you want to support conda dev workflows.

I've held off on updating the contributing instructions for this reason.

Feedback wanted: Reporting of Python version in codecov

• The python-version strategy var and PYTHON_VERSION env var no longer exists as its managed by Pixi. This means that its not picked up by Codecov. Not sure how to remedy this...

Nightlies

I have not considered nightlies in this PR and also haven't yet thought about how to go about that.

---

I think that's about it! I don't think I've forgotten anything, but it is late on a Friday so maybe - will update if that's the case :)

Let me know if you want me to drop by the dev meeting on 5 Nov - but I'm happy to keep this async otherwise.

---

(🎉 for my first significant contribution to Xarray!!!)

[VeckoTheGecko]

I just realised that environment-min-all-deps.yml is also used in testing, and not just the version policy. I need to add a corresponding environment here.

[VeckoTheGecko]

It feels like a bit of an explosion of environments in this file, but its also what was there in the previous codebase so maybe that's just needed. Not sure how that can be cleaned up further - or maybe just leave that for a future PR

[lucascolley]

• for some reason the CI of Pixi is finding which pytest to be .pixi/envs/default/Scripts/pytest while local pixi run -e test-all-deps-py313 which pytest is finding .pixi/envs/test-all-deps-py313/bin/pytest

Even when only one environment is installed via the argument to setup-pixi, I think you still need to specify the environment when you want to use a non-default environment for arbitrary commands. So pixi run -e test-all-deps-py313 which pytest should work in CI too.

[lucascolley]

@lucascolley would it be beneficial to do a write-up of all this on prefix.dev sometime to help guide others dealing with this? I'm happy to write or collab on a blog post.

Yes! The timeline as I have been imagining it for my work is:

• merge DEV: move Pixi workspace to repo root scipy/scipy#23844 (any time now)
• set up automated lock file updates on cron (perhaps monthly PRs from the GHA bot) which pixi update
• set up the 'bleeding-edge' job. For SciPy it was suggested that we may want a bot account to run these jobs on its fork, to avoid cluttering the main repo, and report status in an issue.
• implement something like rfc: add no-upgrade field to opt-out of pixi upgrade in manifest prefix-dev/pixi#4644
• switch from pixi update to pixi upgrade which will allow constraints in the manifest for readability and solver performance

Maybe towards the end of that process would be a good time to write something about the various options and why we have chosen to do things however we end up doing them.

[max-sixty]

Conclusion

Approach (a) has minimal setup/maintenance with little downside. I think that it's a good solution for smaller projects in particular (we've adopted it at Parcels - cc @maxrjones might be interesting based on your comment )

Approach (b) is more robust if having the same environment between all devs is highly valued (@shoyer mentioned during a dev meeting that this would be good for xarray), but requires more setup.

speaking just for myself — I would vote for (b). this is one of the big advantages of something like uv

fine to push off to another change if it's additional work ofc (though is it more work? I see a decent amount of the added code is responsible for generating & caching lockfiles...)

[VeckoTheGecko]

though is it more work?

Yes, a significant amount of work to do it properly. Hence why I think we should wait until Lucas' work at Scipy has stabilized and it becomes clearer how to best handle this. I recommend reading the discussion at #10732 (comment) and the linked issues which go more in depth

[keewis]

As far as I remember the one big issue with persisting the lock file is that this doesn't play well with dynamic versioning using git tags: this seems to require bumping the lock file every now and then, even when we don't really want to (or at least that was the case when we had that discussion a couple of weeks ago, I didn't check if anything changed since then).

[VeckoTheGecko]

re. dynamic versioning support in pixi prefix-dev/pixi#2923 for those interested (probably other issues there as well)