Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow Galaxy to use cached mulled containers for Singularity. #4179

Merged
merged 3 commits into from Jun 12, 2017

Conversation

Projects
None yet
2 participants
@jmchilton
Copy link
Member

commented Jun 12, 2017

This builds on #4173 and so that open PR is included here. (That has been merged.)

  • Change undocumented (and probably unused) config option container_image_cache_path default to database/contianer_cache.
  • Update container resolver framework to let plugins assert they only produce containers of a single type.
  • Update existing mulled container resolvers to indicate they produce Docker containers only - update names also.
  • Refactor mulled caching stuff to allow reuse with Singularity (where the names come from is different - but a lot of the logic is the same).
  • Add a cached singularity container resolver - that searches <config.container_image_cache_path>/singularity/mulled for an image with the correct name. This mirrors the cached Docker which looks for similar image names for Docker but just in the output for docker images.

These cached images can be built with the mulled tooling included with galaxy-lib - it should be as easy as mulled-build-tool --singularity <tool_path> build to produce such images.

Future work includes a subsequent PR will allow Galaxy to build these images and populate that cache automatically and a longer term open infrastructure project is to build these and publish them to depot.galaxyproject.org (or Singularity hub).

@bgruening

This comment has been minimized.

Copy link
Member

commented Jun 12, 2017

@jmchilton we need a rebase here. Do you want me to do it?

jmchilton and others added some commits Jun 10, 2017

Allow Galaxy to use cached mulled containers for Singularity.
- Change undocumented (and probably unused) config option container_image_cache_path default to "database/contianer_cache".
- Update container resolver framework to let plugins assert they only produce containers of a single type.
- Update mulled container resolvers to indicate they produce Docker containers only - update names also.
- Refactor mulled caching stuff to allow reuse with Singularity (where the names come from is different - but a lot of the logic is the same).
- Add a cached singularity container resolver - that searches "config.container_image_cache_path/singularity/mulled" for an image with the correct name.

Next up - similarly refactor the BuildMulledDockerContainerResolver to allow reuse and implement the BuildMulledSingularityContainerResolver.

@jmchilton jmchilton force-pushed the jmchilton:singularity_mulled_cached branch from 9d44287 to 4904633 Jun 12, 2017

@jmchilton

This comment has been minimized.

Copy link
Member Author

commented Jun 12, 2017

@bgruening We did need a rebase - I've done this - thanks!

@bgruening

This comment has been minimized.

Copy link
Member

commented Jun 12, 2017

The failing API test is unrelated. I integrated this as part of the Galaxy Docker project and put it under CI testing.

bgruening/docker-galaxy-stable#349
https://travis-ci.org/bgruening/docker-galaxy-stable/jobs/242178166

I think we are good here!
Thanks John!

@bgruening bgruening merged commit 3e36af0 into galaxyproject:dev Jun 12, 2017

4 of 5 checks passed

api test Build finished. 279 tests run, 0 skipped, 1 failed.
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
framework test Build finished. 150 tests run, 0 skipped, 0 failed.
Details
integration test Build finished. 34 tests run, 0 skipped, 0 failed.
Details
toolshed test Build finished. 579 tests run, 0 skipped, 0 failed.
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.