Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update docker recipe #603

Merged
merged 26 commits into from
Jun 19, 2020
Merged

Update docker recipe #603

merged 26 commits into from
Jun 19, 2020

Conversation

jvegreg
Copy link
Contributor

@jvegreg jvegreg commented Apr 1, 2020

Updated docker recipe to install from source. Allows automation on the master branch

Tasks

  • Create an issue to discuss what you are going to do, if you haven't done so already (and add the link at the bottom)
  • This pull request has a descriptive title that can be used in a changelog
  • Add unit tests
  • Public functions should have a numpy-style docstring so they appear properly in the API documentation. For all other functions a one line docstring is sufficient.
  • If writing a new/modified preprocessor function, please update the documentation
  • Circle/CI tests pass. Status can be seen below your pull request. If the tests are failing, click the link to find out why.
  • Codacy code quality checks pass. Status can be seen below your pull request. If there is an error, click the link to find out why. If you suspect Codacy may be wrong, please ask by commenting.
  • Please use yamllint to check that your YAML files do not contain mistakes
  • If you make backward incompatible changes to the recipe format, make a new pull request in the ESMValTool repository and add the link below

If you need help with any of the tasks above, please do not hesitate to ask by commenting in the issue or pull request.


Closes #566

@bouweandela
Copy link
Member

Hi Javier,

Thanks for picking this up! What is your plan with the docker images? How many with what content?

I thought we might build images for releases and maybe some other images called something like esmvalcore-dev and esmvaltool-dev with only the dependencies we can use instead of the plain conda docker image to speed up the tests on CircleCI?

The links to ci/dockercloud to not seem to work, is this expected?

@jvegreg
Copy link
Contributor Author

jvegreg commented May 5, 2020

That is a good idea. So we keep three images for each repository:

  • Only dependencies, to speed up tests
  • Master branch, with the latest developments
  • Stable, with the latest conda release

And the links failing is not expected, no

@jvegreg
Copy link
Contributor Author

jvegreg commented May 7, 2020

Unfortunately, pip has no way to install only the dependencies required for a package. What I am doing for the dev container is to install ESMValCore in develop mode and remove it afterwards. I have seen other solutions (e.g. creating the egg and looking for its requirements.txt) but I thought they were more confusing.

Copy link
Member

@bouweandela bouweandela left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, I think this is a very good start. Could you please also

docker/Dockerfile.dev Outdated Show resolved Hide resolved
.circleci/config.yml Outdated Show resolved Hide resolved
@jvegreg
Copy link
Contributor Author

jvegreg commented May 20, 2020

It does make sense to change the starting image for other jobs? I doubt it make sense for install, develop and conda_build, as we are testing the installation procedure itself, but I think it can make sense for the documentation job

@bouweandela
Copy link
Member

It does make sense to change the starting image for other jobs? I doubt it make sense for install, develop and conda_build, as we are testing the installation procedure itself,

I agree

it can make sense for the documentation job

It would be best to leave that unchanged too, because now it mimics the steps followed by readthedocs.

Javier Vegas-Regidor and others added 2 commits May 20, 2020 15:57
@bouweandela
Copy link
Member

I added the compilers package to environment.yml so vmprof compiles and changed the develop test so it tests that doing a development installation with just environment.yml actually works.

doc/quickstart/install.rst Outdated Show resolved Hide resolved
doc/quickstart/install.rst Outdated Show resolved Hide resolved
doc/quickstart/install.rst Outdated Show resolved Hide resolved
doc/quickstart/install.rst Outdated Show resolved Hide resolved
doc/quickstart/install.rst Outdated Show resolved Hide resolved
doc/quickstart/install.rst Outdated Show resolved Hide resolved
doc/quickstart/install.rst Outdated Show resolved Hide resolved
doc/quickstart/install.rst Outdated Show resolved Hide resolved
Co-authored-by: Bouwe Andela <b.andela@esciencecenter.nl>
@jvegreg jvegreg added this to the v2.0.0 milestone Jun 8, 2020
Copy link
Member

@bouweandela bouweandela left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mattiarighi @valeriupredoi Could you please test?

@bouweandela bouweandela changed the title Update docker recipe to install from source Update docker recipe Jun 8, 2020
@valeriupredoi
Copy link
Contributor

Cheers muchly @jvegasbsc 🍺 - my sincere apols for being so late testing this! I have actually postponed it back when you asked me then I completely forgot!

My bit of testing: installed docker (Ubuntu/trusty) and checked:

(base) valeriu@valeriu:~$ sudo docker run hello-world

Hello from Docker!
This message shows that your installation appears to be working correctly.

but next thing hit the wall:

(base) valeriu@valeriu:~$ sudo docker pull esmvalgroup/esmvaltool:2.0
Error response from daemon: manifest for esmvalgroup/esmvaltool:2.0 not found

also worth mentioning that sudo should be used when locally, in some OS cases.

So the image I am pulling is esmvalgroup/esmvaltool:2.0.0b3 - why in the world is it 2.7GB?? It's 'uge! (sha: 4e86810a0a90). My entire miniconda3 (containing multiple envs, tarballs of all the packages etc is 7GB. root on a Debian OS may get completely filled up (as in my case). Are you sure it has to be this big?

OK - it installed:

Digest: sha256:6959d905c82b30d73805acfabc05709376d7b9bfd82f397b75f5d247332a3e45
Status: Downloaded newer image for esmvalgroup/esmvaltool:2.0.0b3

Help works but with a couple warnings:

(base) valeriu@valeriu:~$ sudo docker run esmvalgroup/esmvaltool:2.0.0b3 --help
[sudo] password for valeriu: 
Warning 1: +init=epsg:XXXX syntax is deprecated. It might return a CRS with a non-EPSG compliant axis order.
ERROR 1: PROJ: proj_create: Open of /opt/conda/share/proj failed
ERROR 1: PROJ: proj_create: cannot expand +init=epsg:4326 +type=crs
[esmvaltool help follows]

Then it comes to this:

(base) valeriu@valeriu:~/ESMValCore$ sudo docker run -v /home/valeriu/:/data/ esmvalgroup/esmvaltool:2.0.0b3 ~/ESMValTool/esmvaltool/recipes/examples/recipe_preprocessor_test.yml
...
FileNotFoundError: [Errno 2] Specified recipe file does not exist: '/home/valeriu/ESMValTool/esmvaltool/recipes/examples/recipe_preprocessor_test.yml'

but that unfortunately exists. Also why are using nml/....xml v1-style instructions?

How do you tell it where to get the recipes from?

Copy link
Contributor

@valeriupredoi valeriupredoi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pls see me comments 🍺

@valeriupredoi
Copy link
Contributor

correction: as per the new instructions:

sudo docker run -e HOME -v "$HOME":"$HOME" -v /data:/data esmvalgroup/esmvaltool:2.0.0b3 ~/ESMValTool/esmvaltool/recipes/examples/recipe_preprocessor_test.yml

runs fine (I got no data, but it's complaining exactly bc of that, it finds the recipe). Can you pls remove the obsolete docker/docker_guide.md since I was erroneously following that one 🍺

@valeriupredoi
Copy link
Contributor

I am still very concerned about the size of the container, can we do something about that?

@bouweandela
Copy link
Member

bouweandela commented Jun 19, 2020

Hi V,

Please note that this pull request is for the docker container for ESMValCore (not ESMValTool). The tags stable and latest that are mentioned in the documentation are not available yet on dockerhub (they would require merging this pull request), but you can test this branch with the tag update_docker. E.g. the command would be

docker run esmvalgroup/esmvalcore:update_docker

@bouweandela
Copy link
Member

How do you tell it where to get the recipes from?

If you build the documentation locally, you should see a nice webpage explaining this, as part of the ESMValCore installation instructions.

@valeriupredoi
Copy link
Contributor

docker run esmvalgroup/esmvalcore:update_docker

running it now

@valeriupredoi
Copy link
Contributor

valeriupredoi commented Jun 19, 2020

How do you tell it where to get the recipes from?

If you build the documentation locally, you should see a nice webpage explaining this, as part of the ESMValCore installation instructions.

it's fine, Javi's new instructions are spot-on! Just got confused by the old instructions file

@valeriupredoi
Copy link
Contributor

yarp! works fine! 🍺 Am gonna approve this since I was testing the wrong container but we need to be careful about esmvalgroup/esmvaltool:stable when it will be built - size matters 😁

@bouweandela bouweandela merged commit 4a8776f into master Jun 19, 2020
@bouweandela bouweandela deleted the update_docker branch June 19, 2020 12:57
@bouweandela
Copy link
Member

The development tag still needs to be updated on dockerhub: https://hub.docker.com/repository/docker/esmvalgroup/esmvalcore/builds/edit (it's still pointing to this branch), but it says the repository has been removed.

@jvegreg
Copy link
Contributor Author

jvegreg commented Jun 19, 2020

We can remove it, I think. 'stable' with the latest releases and 'latest' for master is all we need for now

Very stupi error. Already changed to master

@bouweandela
Copy link
Member

We use it for running the unit tests:

test:
# Run tests
working_directory: /test
docker:
- image: esmvalgroup/esmvalcore:development
steps:
- checkout
- run:
command: |
python setup.py test --addopts '-m "not installation"'

They now take less than 3 minutes to complete: https://app.circleci.com/pipelines/github/ESMValGroup/ESMValCore?branch=master

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Container (docker/singularity) outdated
3 participants