Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use mamba to install Continuous Integration dependencies #841

Merged
merged 13 commits into from
Sep 18, 2021
Merged

Use mamba to install Continuous Integration dependencies #841

merged 13 commits into from
Sep 18, 2021

Conversation

weiji14
Copy link
Member

@weiji14 weiji14 commented Feb 6, 2021 •
edited
Loading

Description of proposed changes

To cut down on our Continuous Integration's installation time (~9 min on Windows), let's replace conda with a more efficient installer called mamba! Specifically using mambaforge which is a faster miniconda replacement that defaults to using the conda-forge channel and mamba instead of conda by default.

Continuation of #813 which sped up our Continuous Documentation builds.

Update September 2021: Below shows comparison on Windows - Python 3.9 with a 5min speedup (18m44s -> 13m42s), this is without caching.

Old using conda (18m 44s) New using mamba (13m 42s)
image image

More benchmarks:

Build Old time New time Diff
ubuntu-latest - Python 3.7 / NumPy 1.18 10m27s 9m8s -1m19s
macOS-latest - Python 3.7 / NumPy 1.18 14m42s 10m11s -4m31s
windows-latest - Python 3.7 / NumPy 1.18 17m12s 13m0s -4m12s
ubuntu-latest - Python 3.9 / NumPy 1.21 13m22s 7m32s -5m50s
macOS-latest - Python 3.9 / NumPy 1.21 17m4s 13m15s -3m49s
windows-latest - Python 3.9 / NumPy 1.21 19m12s 14m41s -4m41s

References:

Addresses #584

Reminders

  • Run make format and make check to make sure the code follows the style guide.
  • Add tests for new features or tests that would have caught the bug that you're fixing.
  • Add new public functions/methods/classes to doc/api/index.rst.
  • Write detailed docstrings for all functions/methods.
  • If adding new functionality, add an example to docstrings or tutorials.

Slash Commands

You can write slash commands (/command) in the first line of a comment to perform
specific operations. Supported slash commands are:

  • /format: automatically format and lint the code
  • /test-gmt-dev: run full tests on the latest GMT development version

@weiji14
Use mamba to install Continuous Integration dependencies
79cd804 
Specifically using `mambaforge` which is a `miniconda`
replacement that defaults to using the `conda-forge`
channel and `mamba` instead of `conda` by default.
@weiji14 weiji14 added the maintenance Boring but important stuff for the core devs label Feb 6, 2021
@weiji14 weiji14 changed the title Use mamba to install Continuous Integration dependencies WIP: Use mamba to install Continuous Integration dependencies Feb 6, 2021
@weiji14
Move dev dependency specification to environment.yml
5612a68 
Delete the requirements-dev.txt file, and update the ci_test.yml to install GMT dependencies while setting up mamba.
@seisman
Copy link
Member

seisman commented Feb 19, 2021

Should we try the cache action?

@weiji14
Copy link
Member Author

weiji14 commented Feb 19, 2021

Should we try the cache action?

Sure, feel free to try it out in this branch. I'll be heading out for most of the day.

@seisman
Copy link
Member

seisman commented Feb 25, 2021

As you can see, caching (bf1e97d) doesn't help.

See PR #932 for another test. It's clear that fewer packages mean faster speed.

@weiji14 weiji14 changed the title WIP: Use mamba to install Continuous Integration dependencies Use mamba to install Continuous Integration dependencies Sep 17, 2021
@weiji14 weiji14 marked this pull request as ready for review September 17, 2021 11:44
weiji14
weiji14 commented Sep 17, 2021
View reviewed changes
Copy link
Member Author

@weiji14 weiji14 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As you can see, caching (bf1e97d) doesn't help.

See PR #932 for another test. It's clear that fewer packages mean faster speed.

I've reverted the caching step, and configured setup-miniconda to use strict channel priority (conda-forge only, with no packages from default anaconda channel) which speeds up the conda/mamba resolver step (see https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-channels.html#strict-channel-priority). See if this looks ok @seisman.

.github/workflows/ci_tests.yaml Outdated Show resolved Hide resolved
@seisman
Copy link
Member

seisman commented Sep 17, 2021

It looks great. Maybe also update ci_docs.yml, cache_data.yml and even vercel's package.json file.

@weiji14
Copy link
Member Author

weiji14 commented Sep 17, 2021

It looks great. Maybe also update ci_docs.yml, cache_data.yml and even vercel's package.json file.

Ok, updated ci_docs.yml and cache_data.yml to use mamba! I won't update the vercel package.json file because using micromamba didn't seem to give a noticeable speedup last time we tried (ref #813 (review)).

@weiji14 weiji14 self-assigned this Sep 17, 2021
@weiji14 weiji14 added this to the 0.5.0 milestone Sep 17, 2021
@weiji14 weiji14 mentioned this pull request Sep 17, 2021
Closed
4 tasks
@weiji14 weiji14 added the final review call This PR requires final review and approval from a second reviewer label Sep 18, 2021
@weiji14 weiji14 added this to In progress in Release v0.5.x Sep 18, 2021
weiji14
weiji14 commented Sep 18, 2021
View reviewed changes
Copy link
Member Author

@weiji14 weiji14 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @seisman, I'll merge this PR on second approval or in ~12 hours, and that should give us some good CI time savings!

.github/workflows/ci_tests_dev.yaml Outdated Show resolved Hide resolved
@seisman
Copy link
Member

seisman commented Sep 18, 2021

Don't forget to remove the python-version: 3.7 line before merging.

@weiji14 weiji14 merged commit 8409764 into main Sep 18, 2021
@weiji14 weiji14 deleted the mambaforge branch September 18, 2021 10:41
@seisman seisman removed the final review call This PR requires final review and approval from a second reviewer label Sep 18, 2021
@weiji14 weiji14 moved this from In progress to Done in Release v0.5.x Sep 20, 2021
@weiji14 weiji14 mentioned this pull request Dec 5, 2021
Merged
6 tasks
sixy6e pushed a commit to sixy6e/pygmt that referenced this pull request Dec 21, 2022
@weiji14 @seisman
Use mamba to install Continuous Integration dependencies (GenericMapp…
1901ab5 
…ingTools#841)

Specifically using `mambaforge` which is a `miniconda`
replacement that defaults to using the `conda-forge`
channel and `mamba` instead of `conda` by default.

* Set strict channel priority to conda-forge only with no
defaults to see if it speeds up resolver. See also
https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-channels.html#strict-channel-priority

* Use mamba on GMT Dev Tests CI build
* Use mamba in CI docs and Cache Data workflows
* Pin gmt dev version to 6.3 on Windows CI

Co-authored-by: Dongdong Tian <seisman.info@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@maxrjones maxrjones maxrjones left review comments

@seisman seisman seisman approved these changes

Assignees

@weiji14 weiji14

Labels
maintenance Boring but important stuff for the core devs
Projects
No open projects
Release v0.5.x
  
Done
Milestone
0.5.0
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@weiji14 @seisman @maxrjones