Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speed up conda installation #1677

Merged
merged 2 commits into from
Jun 17, 2020
Merged

Speed up conda installation #1677

merged 2 commits into from
Jun 17, 2020

Conversation

bouweandela
Copy link
Member

@bouweandela bouweandela commented Jun 12, 2020

Pin some lower versions of packages to make it easier for conda to solve the environment. See ESMValGroup/ESMValCore#660 and #1676.


Tasks

  • Create an issue to discuss what you are going to do, if you haven't done so already (and add the link at the bottom)
  • Give this pull request a descriptive title that can be used as a one line summary in a changelog
  • Circle/CI tests pass. Status can be seen below your pull request. If the tests are failing, click the link to find out why.
  • Preferably Codacy code quality checks pass, however a few remaining hard to solve Codacy issues are still acceptable. Status can be seen below your pull request. If there is an error, click the link to find out why. If you suspect Codacy may be wrong, please ask by commenting.
  • Please use yamllint to check that your YAML files do not contain mistakes

Copy link
Contributor

@valeriupredoi valeriupredoi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have you used these forcings to build the new conda package @bouweandela ?

@valeriupredoi
Copy link
Contributor

about the new conda package - am a happy camper, well done Bouwe! 🍺 A few remarks:

  • it is much much faster!
  • it still consumes considerable amounts of RAM: up to 2GB even more; on a machine with 4GB max and running other stuff (eg the absolutely inefficient Firefox) it'll struggle, we should probably mention this
  • I am still getting the dlopen error but that's just my ancient Ubuntu
  • I think it's still useful to keep a footnote a la: if your conda is struggling, install esmvalcore first (the previous PR)
  • in case of a rollback-ed installation (eg one forgets to install and link julia) there are a lot of ClobberError's since conda doesn't remove the __pycache__ dirs: lib/python3.7/site-packages/esmvaltool/diag_scripts/zmnam/__pycache__/zmnam_preproc.cpython-37.pyc will persist, we should give a pro tip for the real pros to remove all these useless dirs
  • very nice and convenient once the R packages are installed with one installation even if it rolls back and doesn't install the esmvaltool package, those compiled and made R packages are used for a second time try of installation and no need to watch cc1plus take forever and half again
  • I still need to install it on Jasmin too (can't log in at the mo) but it worked really well on my ancient box so 👍 🍺

@bouweandela
Copy link
Member Author

Have you used these forcings to build the new conda package @bouweandela ?

Yes

I think it's still useful to keep a footnote a la: if your conda is struggling, install esmvalcore first (the previous PR)

Let's discuss that in the other PR. It might work some of the time, but it might not work other times, it's conda.

in case of a rollback-ed installation (eg one forgets to install and link julia) there are a lot of ClobberError's since conda doesn't remove the pycache dirs: lib/python3.7/site-packages/esmvaltool/diag_scripts/zmnam/pycache/zmnam_preproc.cpython-37.pyc will persist, we should give a pro tip for the real pros to remove all these useless dirs

This shouldn't be a problem, since conda will hopefully overwrite those files with identical copies. Apparently it is not able to completely roll back the installation.

@bouweandela bouweandela merged commit f11ae60 into master Jun 17, 2020
@bouweandela bouweandela deleted the versions branch June 17, 2020 07:39
@valeriupredoi
Copy link
Contributor

yes and yes, and good stuff again! 🍺

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants