Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Interactive Environment container waiting and swarm management #3740

Merged
merged 1 commit into from Mar 15, 2017

Conversation

Projects
None yet
4 participants
@natefoo
Copy link
Member

commented Mar 10, 2017

  1. Create a new API endpoint and client spin to wait for GIE containers to become ready. IEs are not required to use it, but usage does require a slight change to their load_notebook() functions.

  2. Add a Docker swarm manager daemon that runs automatically to add nodes when services are waiting, clean up exited services, and remove idle nodes. Shuts down automatically when there's nothing to do.

The format of the GIE proxy session database has changed, if GIEs are running when you restart Galaxy, they will be interrupted. If you run the GIE proxy separate from Galaxy, you should restart it after upgrading.

The cron job to clean services has been removed since it is obviated by the swarm manager.

I am not excited about the placement of the swarm manager code, but couldn't come up with a better location for it. Ideas welcome.

@@ -256,6 +256,7 @@ def populate_api_routes( webapp, app ):
webapp.mapper.resource( 'remote_file', 'remote_files', path_prefix='/api' )
webapp.mapper.resource( 'group', 'groups', path_prefix='/api' )
webapp.mapper.resource_with_deleted( 'quota', 'quotas', path_prefix='/api' )
webapp.mapper.connect( '/api/interactive_environment_ready', controller='interactive_environment_ready' )

This comment has been minimized.

Copy link
@martenson

martenson Mar 10, 2017

Member

fwiw this is not very RESTful

This comment has been minimized.

Copy link
@natefoo

natefoo Mar 10, 2017

Author Member

Yeah... suggestions welcome.

This comment has been minimized.

Copy link
@martenson

martenson Mar 10, 2017

Member

Should this be part of the /api/user?

This comment has been minimized.

Copy link
@dannon

dannon Mar 10, 2017

Member

I know they're launched from dataset(hda)->visualizations, but are they actually tied to them? Since they ultimately end up being a history item, it feels like history/contents/<whatever_it_is> should show status.

This comment has been minimized.

Copy link
@martenson

martenson Mar 10, 2017

Member

@dannon iiuc this API is tied to user's session - not to a particular GIE

This comment has been minimized.

Copy link
@natefoo

natefoo Mar 13, 2017

Author Member

Eventually we did want to have the ability to run multiple concurrent GIEs. Only one per session right now, though.

This comment has been minimized.

Copy link
@dannon

dannon Mar 13, 2017

Member

Planning for the future (since this is the API and all), maybe go with something GET /api/interactive_environment/<key>/ready?

This comment has been minimized.

Copy link
@jmchilton

jmchilton Mar 13, 2017

Member

I'd consider a web controller for this instead of an API controller - it is what I think we use for something like getting the current history ID that isn't stateless or RESTful.

@natefoo natefoo force-pushed the natefoo:docker-swarm-mode-wait branch 2 times, most recently from 174294c to 8007a07 Mar 11, 2017

@martenson
Copy link
Member

left a comment

LGTM. I would still like to see the API under /api/user or better.

GIE container waiting and swarm management.
1. Create a new API endpoint and client spin to wait for GIE containers to
   become ready. IEs are not required to use it, but usage does require
   a slight change to their `load_notebook()` functions.

2. Add a Docker swarm manager daemon that runs automatically
   to add nodes when services are waiting, clean up exited services, and
   remove idle nodes. Shuts down automatically when there's nothing to
   do.

The format of the GIE proxy session database has changed, if GIEs are
running when you restart Galaxy, they will be interrupted. If you run
the GIE proxy separate from Galaxy, you should restart it after
upgrading.

@natefoo natefoo force-pushed the natefoo:docker-swarm-mode-wait branch from 8007a07 to 08f66a9 Mar 14, 2017

@natefoo

This comment has been minimized.

Copy link
Member Author

commented Mar 14, 2017

Added some changes with a rebase. Moved /api/interactive_environment_ready to a regular controller at /interactive_environments/ready and moved the container package to the top level of galaxy, plus a couple small bug fixes.

@jmchilton

This comment has been minimized.

Copy link
Member

commented Mar 15, 2017

Cool stuff - thanks!

@jmchilton jmchilton merged commit 149bbf8 into galaxyproject:dev Mar 15, 2017

5 checks passed

api test Build finished. 263 tests run, 0 skipped, 0 failed.
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
framework test Build finished. 140 tests run, 0 skipped, 0 failed.
Details
integration test Build finished. 25 tests run, 0 skipped, 0 failed.
Details
toolshed test Build finished. 580 tests run, 0 skipped, 0 failed.
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.