Skip to content

Commit

Permalink
Fusion of what was the galaxy-vm-launcher into CloudBioLinux. The gal…
Browse files Browse the repository at this point in the history
…axy-vm-launcher was an alternative method for building and launching Galaxy cloud images to the mi-deployment/CloudMan workflow. The galaxy-vm-launcher is used at the University of Minnesota as the launching point for a CLIA-certified Galaxy-based variant detection workflow.

As I helped migrate mi-deployment into CloudBioLinux, I made numerous modifications to allow reuse between the CloudMan setup code and the fucntionality in this changeset. This changeset isn't just moving galaxy-vm-launcher into the same repository as CloudBioLinux, this code base has been heavily modified to make use existing code in CloudBioLinux and vise versa.

While this mechanism for building Galaxy images does not result in all of the bells and whistles available via CloudMan (auto-scaling, admin console, FTP server, share-strings), it does allow a user to launch and clean Ubuntu image, configure it with CloudBioLinux+Galaxy, and transfer data to the VM and into a Galaxy data library all from a single command-line execution and configuration file. This gives rise to easy, automated multi-cloud reproducibility and could provide a path to real hybrid cloud computing.

As part of the migration several enhancements to this code were made, these include:
 - Added some initial CloudBioLinux and CloudMan support. Actions for install CloudBioLinux and launching packaged CloudMan instances have been added. Added ability to tweak CloudBioLinux fabricrc commands, specify CloudBioLinux flavor, and set CloudMan user data from unified settings.yaml file.
 - Refactored the code to provide uniform access to options (be they coming from command-line arguments or a YAML settings file.).
 - Moved all command-line argument and yaml parsing logic into its own small file - main.py. This files simply builds up a python dictionary and passes that to the deploy function in cloudbio.deploy. The upshot of this is that the python dictionary could be built up in some other fashion (perhaps by a webui such BioCloudCentral of Refinery for instance)  and all of the functionality can utilized as a python library.
 - Input settings file can now be specified at runtime (deploy.sh --settings=/path/to/settings.yaml).
 - As part of synchronizing properties between CloudBioLinux and galaxy-vm-launcher, many new paths and settings were parmaterized and can now be overridden if needed.
 - Implemented automated packaging of of instances for OpenStack.
 - Support for versions of Ubuntu prior to Ubuntu 12.04 LTS was dropped.
  • Loading branch information
jmchilton committed Jan 28, 2013
1 parent 7ba2a09 commit 7026231
Show file tree
Hide file tree
Showing 21 changed files with 1,595 additions and 2 deletions.
5 changes: 5 additions & 0 deletions .gitignore
Expand Up @@ -7,3 +7,8 @@ dist/
dpkg.lst
*.log
venv/
deploy/build
deploy/keys
deploy/settings.yaml
deploy/.venv-deploy
deploy/.vagrant

4 comments on commit 7026231

@jmchilton
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Getting self conscious about the pull requests, so despite the complexity of this commit I will just comment on this and say feel free to backout and request changes.

Thanks,
-John

@chapmanb
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

John;
Wow. You rock. This'll take some time to digest but great to be getting this merged into a single project. This is incredible refactoring.
One quick thing I noticed while reading through was all the virtualenv shell scripting. virtualenv-burrito might help take care of some of this:

https://github.com/brainsik/virtualenv-burrito

It would be sweet to get this virtualenv isolation into CloudMan as well. It would help avoid the annoying library dependency conflicts as AMIs and Galaxy libraries change.

Anyhow, thanks again for all the work to get this together. I'm looking forward to learning everything that's in there.

@jmchilton
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! I had never heard of virtual-burrito, this was a great idea. I have implement this in 518c314 for the deployer. Isolating CloudMan sounds like a great idea as well but I leave that up to Enis :)

@afgane
Copy link
Collaborator

@afgane afgane commented on 7026231 Jan 30, 2013

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

... Isolating CloudMan sounds like a great idea as well but I leave that up to Enis :)

Agreed - added to the todo list :)

Please sign in to comment.