Galaxy CloudMan
Geoff Nelson edited this page Jul 3, 2019
·
79 revisions
- Login to be able to use the new Cloud Launch UI: https://launch.usegalaxy.org/login (You'll have to register for an account if you haven't already)
- Navigate to: https://launch.usegalaxy.org/catalog/appliance/galaxy-cloud
-
Which version of this appliance would you like to launch?
:17.05
-
On which cloud would you like to launch your appliance?
:Amazon US East 1 - N. Virginia
-
What type of credentials do you want to use?
:Temporary Credentials
- Input your Refinery-Platform AWS Account Access Key & Secret Access Key
- Click
Next >
- Underneath
Provide a name for your deployment
specify a new name (This will be used to name EC2 instances used by the cluster) - Provide a password to be able to access the CloudMan web interface and the cluster via ssh underneath:
What Password would you like to use for this CloudMan instance?
-
How Large should the Storage Volume be?
:600
- Click the toggle to show
Advanced CloudMan options
- Change the default:
SLURM cluster with Galaxy
toDo not set cluster type now
-
Shared Cluster String
:cm-28d680029604c47e24c3a123ca3164aa/shared/2019-01-15--17-54
-
What type of virtual hardware would you like to use?
:c4.xlarge
- Click the toggle to show
Advanced cloud launch options
-
Which keypair would you like to use for this Virtual Machine?
: Choose your desired keypair - In which placement zone would you like to launch this appliance? Select the availability zone of the Refinery EC2 instance
- Click the checkbox for
EBS Optimized
- Click
Launch
- On the next screen wait until the
Status
column shows:RUNNING
and click the URL underneath theAccess address
column to access your cluster - In the new window or tab, log in with the username:
ubuntu
and the password you have chosen previously (step 8) - Add your cluster name (from step 7) to the newly created Galaxy EBS volume via Volumes on the AWS EC2 Dashboard (this makes it a lot easier to identify them later)
- It will take a few minutes for all the applications to get setup and once that is done, a message popup will inform you that the cluster is ready for use
- Configure Auto-Scaling:
- In the CloudMan console you should see
Autoscaling is off. Turn on?
Click on the link from the "on" in "Turn on" -
Minimum number of worker nodes to maintain
:0
-
Maximum number of worker nodes to maintain
2
-
Type of Nodes(s)
Custom Instance Type
-
Enter a desired instance type
:m4.2xlarge
- Click
Turn autoscaling on
- In the CloudMan console you should see
- Navigate to the CloudMan admin console for your cluster (
<cluster IP>/cloud/admin
) and underneathSystem controls
clickSwitch master to not run jobs
. This will stop the master Node from being an execution host, and will allow for a worker node to be spun up automatically when a job is received. - Update Route 53 CNAME record of the Galaxy instance (
{dev,test,prod}-galaxy.aws.stemcellcommons.org
) with the hostname of the CloudMan master node (via AWS Route 53 Dashboard) - Update refinery instance to communicate to this cluster.
- Follow steps 1-6 from Creating a new cluster using
Cloud Launch v0.2
- Click on:
Fetch saved clusters
and select your saved cluster to use - Continue with the remaining steps while omitting Steps 9, 11, and 12 from Creating a new cluster using
Cloud Launch v0.2
- Do not launch a second instance of a cluster if it is already running (might lead to unpredictable results)
- If you are cloning an existing shared cluster, the storage blocks on the Galaxy FS volume are restored from a snapshot and must be initialized (pulled down from Amazon S3 and written to the volume). This can cause a significant IO performance drop when you start using the cluster for the first time. You can use the
dd
utility to read from all of the blocks on a volume:sudo dd if=/dev/xvdf of=/dev/null bs=1M
where/dev/xvdf
should be the Galaxy FS volume. This process could take a few hours to complete. - Do not restart clusters using "Reboot the cluster" option in the Admin console. This results in strange behavior. Shut down the cluster and then bring it up using Cloud Launch instead.
In the CloudLaunch V2 interface:
- Click on the trash can icon next to your instance
- Click on
Delete Permanently and Archive
The EC2 instances will be terminated but data will remain to allow relaunching (unless you select the option to delete the cluster).
- When introducing new Galaxy Tools/Workflows, editing existing ones a cluster upgrade is warranted and a new shared cluster string will have to be created.
- Please refer to the CloudMan Upgrade notes here
- It may be necessary to use an
instance type
for the master node with at least 4 CPUs for toolshed installations to work properly.
Administration
- Operations
- Setting Up Galaxy
- Galaxy CloudMan
- Annotating & Importing Refinery Tools
- Batch Import ISA-Tabs
- Backup & Restore
- Google reCAPTCHA v2
Development