New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Singularity containers for models #1391

Open
ashiklom opened this Issue May 9, 2017 · 7 comments

Comments

Projects
None yet
4 participants
@ashiklom
Contributor

ashiklom commented May 9, 2017

Description

Singularity is a container-based infrastructure tool specifically geared towards HPC clusters. From my understanding, it roughly combines the portability of virtual machines with the performance and lightness of Docker.

I am proposing that we move towards creating Singularity containers for ecosystem models that have nontrivial system dependencies (e.g. ED, CLM, FATES).

Context

Installing models on different machines is hard, especially when the infrastructures vary and you don't have root access. Using Singularity would mean that we could compile models exactly once (on any machine with Singularity installed) and then simply distrubute the compiled containers to users that want to use different models.

An important advantage over Docker is that, once the Singularity software is installed (which does require root access, but only for this initial installation step), running Singularity containers does not require a background daemon or anything like that, which makes them well suited for execution via qsub.

Possible Implementation

Here is an example for building an ED2 container: https://github.com/ashiklom/ed-singularity

@robkooper

This comment has been minimized.

Member

robkooper commented May 10, 2017

This is what I hope we can do with the docker GSOC as well. Create docker images for PEcAn/BETY/etc and images for each model. Next we need a way to use a messaging system to run each of the models on a specific input. One solution is to use something like RabbitMQ, each image then becomes a worker for that specific model, and we make sure all data is mounted in each container. This will allow the model to access the data, run and save the results. If we want multiple runs of ED in parallel we can add multiple instances of this image.

@robkooper

This comment has been minimized.

Member

robkooper commented May 10, 2017

So you start with a docker image, could this docker image be the ED2 docker image? That way all you need is a small wrapper around the docker image to make it into a singularity image? This would allow us to use the models either with docker (easy on the mac/windows/linux) or with singularity (easy on linux, and liked in HPC centers).

@ashiklom

This comment has been minimized.

Contributor

ashiklom commented May 10, 2017

@ashiklom

This comment has been minimized.

Contributor

ashiklom commented Aug 16, 2017

Update: The BU cluster now has Singularity installed, so this is something we should definitely consider seriously. I tried building and running an ED image, and hit some MPI snags, but got pretty far with very little effort.

@mdietze

This comment has been minimized.

Member

mdietze commented Aug 16, 2017

I was talking to Ethan White at ESA and he thought the support and maturity on Singularity was far behind Docker (which makes sense since the latter is commercial). What I don't know, and would be interested in learning, is

  1. how different is the build process? Is building both 2x as much work or 5% more work over just building Docker? I think if it's 2x then supporting both won't be sustainable
  2. how different would the RabbitMQ be for each? If we can pass messages to either within one message queue, that sounds like a great path to go down. If we need two entirely separate message queues then it gets really cumbersome to have to support even more ways to run a model (local no queue, local in queue, remote no queue, remote in queue, Docker, Singularity, etc)
@ashiklom

This comment has been minimized.

Contributor

ashiklom commented Aug 16, 2017

My understanding is that Docker isn't typically available on University HPCs because it's really intrusive and requires sudo privileges, whereas Singularity is designed with HPCs in mind.

I think Singularity interfaces pretty cleanly with Docker and can actually install almost directly off of Dockerfiles, so it shouldn't be that much work to add it anyway. More importantly, the model of Singularity is that you build an image once on your host machine and then literally copy the entire file over to other machines, as long as both machines have Singularity installed. Almost like a VM, but leveraging the host hardware more directly (through dark magic, as far as I can tell). This means "supporting" Singularity would only require distributing the compiled images, assuming Singularity is installed (which is pretty simple in my experience).

No idea how Rabbit MQ works, or really even what it is, so I can't speak to that.

@dlebauer

This comment has been minimized.

Member

dlebauer commented Aug 17, 2017

There was a lot of discussion about Singularity on HPC systems at a Container Analysis Workshop hosted by NDS at NCSA this week. Some notes from the discussion are in this google doc (search for 'singularity': https://docs.google.com/document/d/1lF92L-4gGr0OZiArFEsM_pn1VUK61m7w7V9hRWRLWEk/edit#

Here is the docker2singularity tool from TACC https://github.com/TACC/docker2singularity

@serbinsh serbinsh referenced this issue Sep 29, 2018

Open

Implement PEcAn.CTSM #2122

0 of 3 tasks complete
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment