Use mamba for environment update and creation in the Docker container build on DockerHub

[valeriupredoi]

Description

Local build works only when using mamba instead of conda for env creation and update. We've really reached the end of the line for conda! The DockerHub build fails with a multitude of deps conflicts, starting apparently from esmvalcore, see output - with mamba I am able to sort out the env in less than 12min and it creats it no problemo!

...
Removing intermediate container c09ae7e0e016
 ---> 28e5e2f200ef
Successfully built 28e5e2f200ef
Successfully tagged esmvaltool:development

This in about half an hour, compared to the failed build of 220min on DockerHub with conda 🤦‍♂️

• Closes Automatic builds on DockerHub are failing #2276

---

Before you get started

• ☝ Create an issue to discuss what you are going to do

Checklist

It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.

• 🛠 This pull request has a descriptive title
• 🛠 Tests run successfully
• 🛠 All checks below this pull request were successful

[valeriupredoi]

cheers @zklaus 🍺 Pls don't merge just yet coz I'd like to test a couple examples recipes, @bouweandela suggested we test the 4 horsemen (python, R, NCL and Julia) examples recipes and he's got a good point 👍

[valeriupredoi]

@bouweandela here's a question for you - why isn't the container esmvaltool able to replace tags like {latestversion} during a run? I am getting this:

2021-09-10 11:35:42,099 UTC [507] DEBUG   Skipping non-existent
/home/valeriu/climate_data/cmip5/output1/CCCma/CanESM2/historical/mon/atmos/Amon/r1i1p1/{latestversion}/tas

but my data exists in

/home/valeriu/climate_data/cmip5/output1/CCCma/CanESM2/historical/mon/atmos/Amon/r1i1p1/latest

EDIT: nevermind, I is an idiot!

What's the DRS for esgf-downloaded data? It's not BADC, it's smething else, gotta find the node where the data was downloaded first?

[bouweandela]

Use the config-developer.yml from ESMValGroup/ESMValCore#1217: https://github.com/ESMValGroup/ESMValCore/blob/esgf-python/esmvalcore/config-developer.yml

[valeriupredoi]

no, it dont work, I still get

2021-09-10 12:05:08,771 UTC [507] DEBUG   Skipping non-existent /home/valeriu/climate_data/cmip5/output1/CCCma/CanESM2/historical/mon/atmos/Amon/r1i1p1/{latestversion}/tas

even if now my data is in:

valeriu@valeriu-PORTEGE-Z30-C:~$ ls -la /home/valeriu/climate_data/cmip5/output1/CCCma/CanESM2/historical/mon/atmos/Amon/r1i1p1/latest/tas
total 59972
drwxrwxr-x 2 valeriu valeriu     4096 Sep 10 13:04 .
drwxrwxr-x 3 valeriu valeriu     4096 Sep 10 13:03 ..
-rw-rw-r-- 1 valeriu valeriu 61401248 Sep 10 13:04 tas_Amon_CanESM2_historical_r1i1p1_185001-200512.nc

When was the last time you ran a recipe from a container you built?

[bouweandela]

Did you mount the directory containing the data in the container as described in the docs: https://docs.esmvaltool.org/en/latest/quickstart/installation.html#install-with-docker? Can you share the output of ls when you run that command inside the container?

[valeriupredoi]

OK the stuff that needs to run runs fine, that is a serious problem though - the incapacity of a container to solve for those {latestversion} tags - if I put all my data into one dir and use drs: default all works fine

[valeriupredoi]

I'm running this - btw you don't need to mount the data dir inside the container, if you specify a correct path in the config user all works fine:

sudo docker run -e HOME -v "$HOME":"$HOME" -v /data:$HOME/climate_data esmvaltool:development run $HOME/ESMValTool/esmvaltool/recipes/examples/recipe_python.yml

[valeriupredoi]

Now that I can use my data (in a common dir with drs: default), the preproccing works fine only to have a fail in the diagnostic right at the top:

Traceback (most recent call last):
  File "/home/valeriu/ESMValTool/esmvaltool/diag_scripts/examples/diagnostic.py", line 8, in <module>
    from esmvaltool.diag_scripts.shared import (
ModuleNotFoundError: No module named 'esmvaltool'

Either I got no idea what I'm doing or there is a serious inherent issue with the container 🤷‍♂️

[valeriupredoi]

yeah it looks like esmvaltool's not in the container?

valeriu@valeriu-PORTEGE-Z30-C:~$ sudo docker run -e HOME -v "$HOME":"$HOME" -v /data:$HOME/climate_data esmvaltool:development version
ESMValCore: 2.3.1

(note the empty space where ESMValTool version should be)

But the container built OK and it looks of the correct size:

valeriu@valeriu-PORTEGE-Z30-C:~$ sudo docker images
REPOSITORY               TAG                IMAGE ID       CREATED        SIZE
esmvaltool               development        28e5e2f200ef   20 hours ago   6.3GB
continuumio/miniconda3   latest             67414e5844b6   7 weeks ago    391MB
node                     12.6-buster-slim   e6e2b19326d7   2 years ago    161MB

[bouweandela]

Indeed, the development container contains all the development dependencies but not ESMValTool itself, we clone and pip install that for running the tests on CircleCI. Try the esmvaltool:latest container instead.

[valeriupredoi]

now you tell me 🤦‍♂️ 😁 Which one's the file for latest Dockerfile or Dockerfile.exp coz I need to build it - this is the locally built container I'm running not the one checked out from DockerHub - I am testing the new mamba use

[valeriupredoi]

also what's the deal, man, I spent 30min (with mamba, otherwise 200+ minutes with conda) building a full container with ESMValTool (deps and all) and now I can use only the Core bit of it - not optimal, we should just build a Core container for dev then

[valeriupredoi]

nay bother, am rebuilding the dev container without the pip uninstall esmvaltool bit so I can have ESMValTool too - beautifully, since am building on top of the previous one, all the env/deps/R guff is already done 😁

[bouweandela]

Which one's the file for latest Dockerfile or Dockerfile.exp

Dockerfile, the second line of each file contains the docker tag for that file. You can also see this in the DockerHub autobuild configuration.

also what's the deal

We use this container to run the tests on CircleCI, so we do not need to use conda to solve the environment and compile R dependencies etc for every test run, this makes the test run much faster:

ESMValTool/.circleci/config.yml

Line 26 in f9350ab

- image: esmvalgroup/esmvaltool:development

Also, if you want to set up a development environment without bothering with conda, you could use it.

[valeriupredoi]

ah it does make sense then 😁 We just need better documentation (or some documentation) on how to build yer own fully functional ESMValTool container, I'd also use more descriptive names like Dockerfile.dev -> DockerEnvironmentWithESMValCoreOnlyAndNOTESMValToolFile.dev 😁

[bouweandela]

I'll try to write something on what is what next week #2275 (comment)

[valeriupredoi]

good man Bouwe 🍺

[valeriupredoi]

OK here's me ready for the weekend: I'vebuilt the full Core+Tool container (cheers, Bouwe, Dockerfile file was indeed the one!):

valeriu@valeriu-PORTEGE-Z30-C:~$ sudo docker run -e HOME -v "$HOME":"$HOME" -v /data:$HOME/climate_data esmvaltool:fullInstall version
ESMValCore: 2.3.1
ESMValTool: 2.3.0

and used it and ran the example recipes: Python OK, R OK, NCL OK, Julia not so much but that failed for the actual normal install too see #2303 - however, these all ran only when I used the default input data and drs, otherwise always stumbling on the issue in #2297 (comment) - either Im doing something wrong or there is an inherent issue in the container, either way, nothing to do with mamba and not using it, I say we let this PR go through 👍 Happy weekend, folks!