Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implementing conda lock #2193

Merged
merged 31 commits into from Feb 11, 2022
Merged

Implementing conda lock #2193

merged 31 commits into from Feb 11, 2022

Conversation

valeriupredoi
Copy link
Contributor

@valeriupredoi valeriupredoi commented Jun 10, 2021

Sister PR to ESMValGroup/ESMValCore#1164

Description

A fast conda environment creation is possible using the provided conda lock files.
To build a conda lock file install the conda-lock package first, then
run conda-lock lock --platform linux-64 -f environment.yml --mamba
(mamba activated for speed) to create a conda lock file for Linux platforms,
or run conda-lock lock --platform osx-64 -f environment.yml --mamba to create a lock file
for OSX platforms. A conda lock file is an explicit environment file that contains pointers to
dependency packages as they are hosted on the Anaconda cloud; these have frozen version numbers
and build hashes, established at the time of the conda lock file creation, so may be obsolete after a while,
but they allow for a robust environment creation while they're still up-to-date. We regenerate these lock files every 10 days, so to minimize the risk of dependencies becoming obsolete. After the lock file has been created,
conda environment creation then happens as per normal use with e.g.
conda create --name esmvaltool --file conda-linux-64.lock (NOTE that pip and conda are NOT installed
in this environment so manual installation is needed, conda install -c conda-forge pip at the very
minimum so we can install esmvaltool).

Functionality

Currently the lock file works fine on Linux, we have a dedicated build and verify Github Actions workflow that runs every 10 days. The lock files are created and verified every 10 days at 4am, the choice of hour is so that it doesn't conflict with the GA test that runs nightly and performs the installation and test from the conda lock file (that runs at midnight). Upon (re-)creation of the lock files an automated PR is generated e.g. this one if the lock file has changed in any way. Note that the auto PR will have me assigned and reviewer, and will only include one changed file, the lock file.

Benefits

Once we have a conda lock file, the conda env creation is blindingly fast (order 2min), and this can be used as a lifeboat if we have issues with the current solving, the env from the lock file being at most 10 days old, so really not too bad. Also the lockfile it serves as a reference for fallbacks, if an environment breaks down due to a dependency being updated and entering conflict with one or more other dependencies.

What's working

Linux OS with latest Python (3.10) conda lock file generation, env creation from it, package installation and testing, auto PR generation when the lock file has changed.

What's not working

OSX: we still have to understand what's going on in OSX land - all sorts of issues related to both lock file generation and env creation from it.

@bvreede
Copy link
Contributor

bvreede commented Jun 14, 2021

I think this could benefit from finally fixing PR #2110 😬 which I completely forgot about... ahem.

On it.

@bvreede
Copy link
Contributor

bvreede commented Jun 24, 2021

How does this PR do now tests are fixed for OSX?? 🥁

@valeriupredoi
Copy link
Contributor Author

sussspense 😁 -> should be fine I guess, am afraid to touch it at the moment 😆

@zklaus zklaus added this to the v2.3.0 milestone Jun 28, 2021
@zklaus zklaus modified the milestones: [locked] v2.3.0, v2.4.0 Jul 14, 2021
@valeriupredoi valeriupredoi modified the milestones: v2.4.0, v2.5.0 Oct 20, 2021
@valeriupredoi
Copy link
Contributor Author

@zklaus am reassigning this to v2.5.0 milestone since it needs a bit more work to automate the PR process (quite a bit, in fact, since I am not yet familiar with the automated Github Action Pull Request creation and approval, but I'll learn, no time for it now though) 🍺

@valeriupredoi valeriupredoi marked this pull request as draft November 12, 2021 12:26
@valeriupredoi valeriupredoi marked this pull request as ready for review January 17, 2022 15:15
@valeriupredoi
Copy link
Contributor Author

OK I did the auto PR generation too #2483 - this is ready for review, guys! At your leisure 🖖

Copy link
Contributor

@zklaus zklaus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice work! Great to see more advanced usage of github actions.

I left a few comments. Additionally, do we want the initial lock file to be here?
Also, we will want some documentation about how to use this for users and what we want to do with it. I think this can go into a different PR so that we can get some experience first, but we should put up an issue for discussion and as a reminder, possibly flagged for 2.5.0 or 2.6.0.

I think one big benefit for us can be the notification of changing dependencies early on, but for that, we need to have a review of the dependencies as part of the process of review for this particular PR.

@valeriupredoi
Copy link
Contributor Author

Very nice work! Great to see more advanced usage of github actions.

I left a few comments. Additionally, do we want the initial lock file to be here? Also, we will want some documentation about how to use this for users and what we want to do with it. I think this can go into a different PR so that we can get some experience first, but we should put up an issue for discussion and as a reminder, possibly flagged for 2.5.0 or 2.6.0.

I think one big benefit for us can be the notification of changing dependencies early on, but for that, we need to have a review of the dependencies as part of the process of review for this particular PR.

cheers muchly @zklaus ! All your points - very good points! Yes, I think it may be a good idea to remove the initial lock file - let me do it now. And yes about documentation too, a separate PR is defo what we'd need since we still have to come to grips with this whole new thingie, as you say 🍺

@valeriupredoi
Copy link
Contributor Author

actually - no, maybe we should keep the initial locklfile after all, otherwise the install GA test would fail before the first auto PR gets merged...

@valeriupredoi
Copy link
Contributor Author

@zklaus what say you, maybe we can get this one over the 2.5 hurdle? We have it in Core now (thanks to your review there too 👏 )

@valeriupredoi
Copy link
Contributor Author

eugh seems I completely forgot to add documentation and a link to the current lockfile, I'll do that now 📖

@valeriupredoi
Copy link
Contributor Author

@zklaus a wee bit of -re-review here would be highly appreciated and beer-rewarded when you gots time 🍺

Comment on lines 4 to 7
push:
branches:
- main
- condalock
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I now think we should remove the triggering on pushes to main. What do you think?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, totally agreed, it's more annoying than a fly around a 💩

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just deactivated the test run on push, I'll do the same for Core too in a wee bit

.github/workflows/install-from-condalock-file.yml Outdated Show resolved Hide resolved
@valeriupredoi
Copy link
Contributor Author

total legend, Klaus, cheers, mate! 🍺

@valeriupredoi valeriupredoi merged commit c5bdc9a into main Feb 11, 2022
@valeriupredoi valeriupredoi deleted the condalock branch February 11, 2022 14:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants