Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use mamba for environment update and creation in the Docker container build on DockerHub #2297

Merged
merged 3 commits into from Sep 11, 2021

Conversation

valeriupredoi
Copy link
Contributor

@valeriupredoi valeriupredoi commented Sep 9, 2021

Description

Local build works only when using mamba instead of conda for env creation and update. We've really reached the end of the line for conda! The DockerHub build fails with a multitude of deps conflicts, starting apparently from esmvalcore, see output - with mamba I am able to sort out the env in less than 12min and it creats it no problemo!

...
Removing intermediate container c09ae7e0e016
 ---> 28e5e2f200ef
Successfully built 28e5e2f200ef
Successfully tagged esmvaltool:development

This in about half an hour, compared to the failed build of 220min on DockerHub with conda πŸ€¦β€β™‚οΈ


Before you get started

Checklist

It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the πŸ›  Technical or πŸ§ͺ Scientific review.

@valeriupredoi
Copy link
Contributor Author

cheers @zklaus 🍺 Pls don't merge just yet coz I'd like to test a couple examples recipes, @bouweandela suggested we test the 4 horsemen (python, R, NCL and Julia) examples recipes and he's got a good point πŸ‘

@valeriupredoi
Copy link
Contributor Author

valeriupredoi commented Sep 10, 2021

@bouweandela here's a question for you - why isn't the container esmvaltool able to replace tags like {latestversion} during a run? I am getting this:

2021-09-10 11:35:42,099 UTC [507] DEBUG   Skipping non-existent
/home/valeriu/climate_data/cmip5/output1/CCCma/CanESM2/historical/mon/atmos/Amon/r1i1p1/{latestversion}/tas

but my data exists in

/home/valeriu/climate_data/cmip5/output1/CCCma/CanESM2/historical/mon/atmos/Amon/r1i1p1/latest

EDIT: nevermind, I is an idiot!

What's the DRS for esgf-downloaded data? It's not BADC, it's smething else, gotta find the node where the data was downloaded first?

@bouweandela
Copy link
Member

bouweandela commented Sep 10, 2021

@valeriupredoi
Copy link
Contributor Author

no, it dont work, I still get

2021-09-10 12:05:08,771 UTC [507] DEBUG   Skipping non-existent /home/valeriu/climate_data/cmip5/output1/CCCma/CanESM2/historical/mon/atmos/Amon/r1i1p1/{latestversion}/tas

even if now my data is in:

valeriu@valeriu-PORTEGE-Z30-C:~$ ls -la /home/valeriu/climate_data/cmip5/output1/CCCma/CanESM2/historical/mon/atmos/Amon/r1i1p1/latest/tas
total 59972
drwxrwxr-x 2 valeriu valeriu     4096 Sep 10 13:04 .
drwxrwxr-x 3 valeriu valeriu     4096 Sep 10 13:03 ..
-rw-rw-r-- 1 valeriu valeriu 61401248 Sep 10 13:04 tas_Amon_CanESM2_historical_r1i1p1_185001-200512.nc

When was the last time you ran a recipe from a container you built?

@bouweandela
Copy link
Member

Did you mount the directory containing the data in the container as described in the docs: https://docs.esmvaltool.org/en/latest/quickstart/installation.html#install-with-docker? Can you share the output of ls when you run that command inside the container?

@valeriupredoi
Copy link
Contributor Author

OK the stuff that needs to run runs fine, that is a serious problem though - the incapacity of a container to solve for those {latestversion} tags - if I put all my data into one dir and use drs: default all works fine

@valeriupredoi
Copy link
Contributor Author

I'm running this - btw you don't need to mount the data dir inside the container, if you specify a correct path in the config user all works fine:

sudo docker run -e HOME -v "$HOME":"$HOME" -v /data:$HOME/climate_data esmvaltool:development run $HOME/ESMValTool/esmvaltool/recipes/examples/recipe_python.yml

@valeriupredoi
Copy link
Contributor Author

Now that I can use my data (in a common dir with drs: default), the preproccing works fine only to have a fail in the diagnostic right at the top:

Traceback (most recent call last):
  File "/home/valeriu/ESMValTool/esmvaltool/diag_scripts/examples/diagnostic.py", line 8, in <module>
    from esmvaltool.diag_scripts.shared import (
ModuleNotFoundError: No module named 'esmvaltool'

Either I got no idea what I'm doing or there is a serious inherent issue with the container πŸ€·β€β™‚οΈ

@valeriupredoi
Copy link
Contributor Author

valeriupredoi commented Sep 10, 2021

yeah it looks like esmvaltool's not in the container?

valeriu@valeriu-PORTEGE-Z30-C:~$ sudo docker run -e HOME -v "$HOME":"$HOME" -v /data:$HOME/climate_data esmvaltool:development version
ESMValCore: 2.3.1

(note the empty space where ESMValTool version should be)

But the container built OK and it looks of the correct size:

valeriu@valeriu-PORTEGE-Z30-C:~$ sudo docker images
REPOSITORY               TAG                IMAGE ID       CREATED        SIZE
esmvaltool               development        28e5e2f200ef   20 hours ago   6.3GB
continuumio/miniconda3   latest             67414e5844b6   7 weeks ago    391MB
node                     12.6-buster-slim   e6e2b19326d7   2 years ago    161MB

@bouweandela
Copy link
Member

bouweandela commented Sep 10, 2021

Indeed, the development container contains all the development dependencies but not ESMValTool itself, we clone and pip install that for running the tests on CircleCI. Try the esmvaltool:latest container instead.

@valeriupredoi
Copy link
Contributor Author

now you tell me πŸ€¦β€β™‚οΈ 😁 Which one's the file for latest Dockerfile or Dockerfile.exp coz I need to build it - this is the locally built container I'm running not the one checked out from DockerHub - I am testing the new mamba use

@valeriupredoi
Copy link
Contributor Author

valeriupredoi commented Sep 10, 2021

also what's the deal, man, I spent 30min (with mamba, otherwise 200+ minutes with conda) building a full container with ESMValTool (deps and all) and now I can use only the Core bit of it - not optimal, we should just build a Core container for dev then

@valeriupredoi
Copy link
Contributor Author

nay bother, am rebuilding the dev container without the pip uninstall esmvaltool bit so I can have ESMValTool too - beautifully, since am building on top of the previous one, all the env/deps/R guff is already done 😁

@bouweandela
Copy link
Member

bouweandela commented Sep 10, 2021

Which one's the file for latest Dockerfile or Dockerfile.exp

Dockerfile, the second line of each file contains the docker tag for that file. You can also see this in the DockerHub autobuild configuration.

also what's the deal

We use this container to run the tests on CircleCI, so we do not need to use conda to solve the environment and compile R dependencies etc for every test run, this makes the test run much faster:

- image: esmvalgroup/esmvaltool:development

Also, if you want to set up a development environment without bothering with conda, you could use it.

@valeriupredoi
Copy link
Contributor Author

ah it does make sense then 😁 We just need better documentation (or some documentation) on how to build yer own fully functional ESMValTool container, I'd also use more descriptive names like Dockerfile.dev -> DockerEnvironmentWithESMValCoreOnlyAndNOTESMValToolFile.dev 😁

@bouweandela
Copy link
Member

I'll try to write something on what is what next week #2275 (comment)

@valeriupredoi
Copy link
Contributor Author

good man Bouwe 🍺

@valeriupredoi
Copy link
Contributor Author

OK here's me ready for the weekend: I'vebuilt the full Core+Tool container (cheers, Bouwe, Dockerfile file was indeed the one!):

valeriu@valeriu-PORTEGE-Z30-C:~$ sudo docker run -e HOME -v "$HOME":"$HOME" -v /data:$HOME/climate_data esmvaltool:fullInstall version
ESMValCore: 2.3.1
ESMValTool: 2.3.0

and used it and ran the example recipes: Python OK, R OK, NCL OK, Julia not so much but that failed for the actual normal install too see #2303 - however, these all ran only when I used the default input data and drs, otherwise always stumbling on the issue in #2297 (comment) - either Im doing something wrong or there is an inherent issue in the container, either way, nothing to do with mamba and not using it, I say we let this PR go through πŸ‘ Happy weekend, folks!

@bouweandela bouweandela merged commit 9272d14 into main Sep 11, 2021
@bouweandela bouweandela deleted the use_mamba_docker branch September 11, 2021 08:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Automatic builds on DockerHub are failing
3 participants