Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

switch to docker in Travis-CI to avoid messy and time consuming dependency builds #1648

Closed
grondo opened this Issue Sep 10, 2018 · 8 comments

Comments

Projects
None yet
3 participants
@grondo
Copy link
Contributor

commented Sep 10, 2018

In flux-framework/flux-sched#386, @trws notes:

After seeing this, I'm thinking two things might be good: 1) get docker going, if this is desirable I'd be happy to set it up, I have all the infrastructure done from doing it for RAJA 2) set up the python/lua stuff in a virtual environment so we can test more than one and be sure everything is installed in the env we test. Since it isn't based on home, or the paths of the installed python, that should keep this kind of thing from cropping up again

@trws, thanks for the offer to set docker up! I think this is the way to go, and if you have time to get us an image with Flux dependencies, I'd be happy to work on redoing the travis-ci hooks such that things nominally work as they do now.

I haven't made use of Docker Hub before, does it allow multiple users to have "commit" access to add and update docker images?

Since use of Docker divorces dependencies from the test environment, how do you handle PRs that add or modify dependencies, especially from contributors that don't have access to modify the "blessed" docker image? (I can think of a couple strategies, but I'm wondering how other docker-based CI builds do it.)

@SteVwonder

This comment has been minimized.

Copy link
Member

commented Sep 10, 2018

One option is we can create another repo under the flux-framework organization that contains a Dockerfile and builds/deploys the Docker image. Then, if your PR to flux-core requires new dependencies, you will need to submit a parallel PR to the new Docker repo.

Alternatively, we could store the Dockerfile in flux-core, but then dependencies will be installed and/or built with every PR.

@grondo

This comment has been minimized.

Copy link
Contributor Author

commented Sep 10, 2018

Starting with a Dockerfile is not a bad idea, then we don't have to deal with Docker Hub at first. It also might allow us to switch to Ubuntu 18.04, for which many of the main dependencies don't need to be built from scratch.

@SteVwonder

This comment has been minimized.

Copy link
Member

commented Sep 10, 2018

Starting with a Dockerfile is not a bad idea, then we don't have to deal with Docker Hub at first.

Works for me! I suspect that will be easier to setup and debug initially. It would be cool if eventually we push a flux-core/master image up to Docker Hub, that way flux-sched can build it's image off the flux-core image.

dependencies will be installed and/or built with every PR.

I was actually wrong. docker build has a --cache-from argument that lets you specify an image as a cache source. AFAIK, if we were to specify flux-core/master then only layers that differ between the image for the current PR and the flux-core/master image would be built. Repo example and the blog post describing the broader workflow.

@SteVwonder

This comment has been minimized.

Copy link
Member

commented Sep 10, 2018

How would be handle the build matrix under Docker? Would we just install all of the different compilers in the same image, and then use the Travis env variables to switch between them? @trws probably has experience with this from RAJA.

@grondo

This comment has been minimized.

Copy link
Contributor Author

commented Sep 10, 2018

Looks like RAJA uses a different docker image for each compiler. At first I was going to keep things as simple as possible and just pass environment variables in from .travis.yml, but try to just have one Dockerfile.

@grondo

This comment has been minimized.

Copy link
Contributor Author

commented Sep 10, 2018

I found that Docker Hub does indeed have Organizations and Teams. For now I created a "fluxframework" organization (it didn't seem to like dashes 🤷‍♂️), where we could potentially post our current "master" images. I'm not sure exactly how you put other users in your organization, but I think perhaps I could just invite by email if this is useful.

We could start with a Dockerfile for flux-security which pulls from ubuntu:18.04, then flux-core pulls from fluxframework/flux-security:master, and flux-sched pulls from fluxframework/flux-core:master. We'll have to look into how to auto-update these images from Travis if we want to keep them up to date, but that doesn't seem too difficult from the examples online.

@grondo grondo referenced this issue Sep 11, 2018

Closed

Docker travis #1652

@trws

This comment has been minimized.

Copy link
Member

commented Sep 11, 2018

We have been doing one compiler per because some of the compilers are really large, mainly nvcc but some of the clangs as well, so putting them all in one made the image unreasonably large (order 15GB IIRC). For flux I currently just have 2-3 images, but we can change that as desired.

For adding dependencies, raja hasn't had added system-level dependencies. We'd probably need to do an update of the base images as part of such an update.

@trws

This comment has been minimized.

Copy link
Member

commented Sep 11, 2018

Looking back a bit farther, we could set something up that would build an updated version of the docker image as part of CI, but usually that pattern is used for deploying docker images with the built version of the software being tested. For now it's set up to assume that the images are managed externally, with the docker files checked in, but if it turns into a maintenance issue we can certainly automate that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.