Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Container interface abstraction #3863

Merged
merged 1 commit into from Apr 13, 2017

Conversation

Projects
None yet
3 participants
@natefoo
Copy link
Member

commented Apr 3, 2017

Support Docker and Docker Swarm mode (>1.12 SwarmKit). Container interface configuration is performed via the containers_conf.yml config file, for which a documented sample is provided.

The GIE launcher has been modified to use the container interface if the enable_beta_containers_interface option is set in galaxy.ini. This removes some of the previous GIE-specific swarm mode work that was done.

The Galaxy docker swarm manager also uses the container interface now. It also includes updates for ensuring a minimum number of slots are available for running containers.

The container interface is Python 3 compatible and app-independent.

TODO: The API docker interfaces should be done fairly soon before we have many consumers of the docker model objects, because they'll probably be changed to more closely reflect their API representations. The only reason the CLI versions remain is that they were already written.

Fixes #3804

try:
from shlex import quote
except ImportError:
from pipes import quote

This comment has been minimized.

Copy link
@jmchilton

jmchilton Apr 3, 2017

Member

Can you use this from six.moves import shlex_quote instead and in the other place these lines appear?

@@ -0,0 +1,560 @@
"""

This comment has been minimized.

Copy link
@jmchilton

jmchilton Apr 3, 2017

Member
#!/usr/bin/env python
configure_logging,
find_path,
find_root,
)

This comment has been minimized.

Copy link
@jmchilton

jmchilton Apr 3, 2017

Member

It seems like there are good abstractions here I'd like to leverage in Pulsar and Planemo. I'd like to just move this all into galaxy-lib but this script can't be because of these galaxy.config things. I could put just a subset of the module in of course - but I usually take this as a sign that a potential library/utility package is attempting to access too much of Galaxy. Maybe it makes some sense to move this one script into galaxy.webapps.galaxy or keep this script here and move everything else into galaxy.util.containers.

This comment has been minimized.

Copy link
@natefoo

natefoo Apr 3, 2017

Author Member

I think I'd prefer moving docker_swarm_manager.py rather than moving the entire container package. And also, we should probably standardize whether packages in galaxy should tend to include a plural form or not.

This comment has been minimized.

Copy link
@natefoo

natefoo Apr 3, 2017

Author Member

galaxy.webapps.galaxy seems like the wrong place for it too, though.

This comment has been minimized.

Copy link
@natefoo

natefoo Apr 4, 2017

Author Member

Having given this a bit more thought... the only thing the Galaxy stuff is used for is locating the Galaxy config file so that containers_conf.yml can be located - except that the automatic loading in galaxy.container.docker_swarm doesn't use that option, it specifies the path to containers_conf.yml directly via the --containers-conf-file option. Outside of that, the swarm manager is useful on its own and apart from Galaxy.

tl;dr I can just remove the Galaxy stuff. Thoughts?

This comment has been minimized.

Copy link
@jmchilton

jmchilton Apr 4, 2017

Member

+1 then

@natefoo natefoo force-pushed the natefoo:containers-abstraction branch from 46b5a78 to 4e00adf Apr 3, 2017

Container interface abstraction. Support Docker and Docker Swarm mode
(>1.12 SwarmKit). Container interface configuration is performed via the
containers_conf.yml config file, for which a documented sample is
provided.

The GIE launcher has been modified to use the container interface if the
`enable_beta_containers_interface` option is set in galaxy.ini. This
removes some of the previous GIE-specific swarm mode work that was done.

The Galaxy docker swarm manager also uses the container interface now.

TODO: The API docker interfaces should be done fairly soon before we
have many consumers of the docker model objects, because they'll
probably be changed to more closely reflect their API representations.

@natefoo natefoo force-pushed the natefoo:containers-abstraction branch from 4e00adf to 29331b3 Apr 4, 2017

@natefoo

This comment has been minimized.

Copy link
Member Author

commented Apr 4, 2017

@jmchilton Okay, no more Galaxy dependencies in the swarm manager. I also renamed to galaxy.containers, squashed and rebased.

@bgruening

This comment has been minimized.

Copy link
Member

commented Apr 4, 2017

This IS awesome! Thanks Nate!!!
@natefoo do you know how to start a IE via API or selenium, so I can test this automatically in Galaxy-Docker?

@natefoo

This comment has been minimized.

Copy link
Member Author

commented Apr 5, 2017

@bgruening you should only need to hit the path to the IE plugin for an existing dataset, e.g. /plugins/interactive_environments/jupyter/show?dataset_id=4b187121143038ff

@jmchilton jmchilton merged commit 695293f into galaxyproject:dev Apr 13, 2017

5 checks passed

api test Build finished. 271 tests run, 0 skipped, 0 failed.
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
framework test Build finished. 140 tests run, 0 skipped, 0 failed.
Details
integration test Build finished. 28 tests run, 0 skipped, 0 failed.
Details
toolshed test Build finished. 579 tests run, 0 skipped, 0 failed.
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.