"Installing" SMRT Portal the easy way Launching A SMRT Portal AMI

Eric van der Helm edited this page Jun 3, 2015 · 70 revisions
Clone this wiki locally

Having the hardware and system administration resources to install SMRT Portal and keep it up to date can be a challenge. There is a very practical alternative for those who only need occasional access to SMRT Portal: You can run SMRT Portal on the Amazon Cloud using a public machine image that Pacific Biosciences maintains and upgrades - you pay only for the machine time that you use. This option is useful for running small to medium-sized secondary analysis jobs (bacterial genome assembly, bacterial base modification analysis, long amplicon analysis, Iso-Seq™ analysis), and for using SMRT Portal command-line tools. The steps are:

  1. Launch the Amazon Machine Image.
  2. Configure SMRT Portal on the Amazon Machine Image.
  3. Use SSH to access your instance. (putty.exe or Terminal in OS X.)
  4. Upload your Data to SMRT Portal using one of these methods:

    A) FileZilla (Windows)

    B) scp (Terminal in OS X or Linux)

    C) Mount an existing S3 bucket to the EC2 instance.

  5. Stop or terminate the Amazon Machine Image Instance when your work is complete.

Step 1: Setting up the Amazon Machine Image

  1. Go to http://aws.amazon.com and create an Amazon Web Services™ account.

  2. Click My Account/Console, then choose AWS Management Console. (Sign in first if asked.)

  3. Set the location to US East N. Virginia on the upper-right of the page.

  4. Click EC2.

    Figure 1: Click on EC2

  5. Click Instances, then click Launch Instance.

    Figure 2: Set the region, select Instances, and Launch.

  6. Click the Community AMIs tab, under Quick Start.

    Figure 3: Click on COmmunity AMIs

  7. Search for smrt, choose the latest instance, then click Select. (This may take several minutes to load.)

  8. Choose an appropriate Instance Type for your analysis, then click Next: Configure Instance Details. A micro instance is not sufficient to analyze lambda.

    • For bacterial analysis, choose m3.2xlarge from the General Purpose category.

    • For mammalian analysis, select r3.8xlarge from the Memory Optimized category.

    • Be sure to adjust, after completing the setup, the NPROC setting in /analysis/etc/smrtpipe.rc to the correct amount of cores selected in this step.

    Figure 4: Choose an instance type

  9. Leave Step 3: Configure Instance Detail at the default, and click Next: Add Storage.

  10. Leave Step 4: Add Storage at default, then click Next: Tag Instance.

  11. Enter an arbitrary string for the Name value (smrtportal is a good default), then click Next: Configure Security Group.

  12. Under Assign a security group, select the Create a new security group option. If you have launched a SMRT Portal AMI before, you can Select an existing security group and use one that you previously configured.

  13. To access SMRT Portal both via the web and via ssh, you need to open the required ports on the server you are launching. Under Protocol, SSH with Port Range 22 should already be listed.

  14. Click Add Rule.

  15. From the Protocol pull-down menu, select Custom TCP Rule. Enter 8080 in the Port Range field. (This opens the port necessary for SMRT Portal web access.)

  16. Click Review and Launch.

    Figure 5: Configure the Security Group

  17. Alternately, use a previously configured security group.

    Figure 7b: Alternately, use a previously configured security group.

  18. Disregard the security group and free usage tier warning and click Launch.

  19. In the pop-up window, select Create a new key pair and name the key pair. Click Download Key Pair and save the file to a safe location on your local machine. (The key pair file has a .pem extension.)

  20. Click Launch Instances.

    Figure 8: Create a key pair.

  21. To view your launched instance, click View Instances in the bottom right corner of the page. This will take you back to the Instances page of the EC2 Management Console (where we started). Your instance will take a few minutes to boot up and run through a status check.

    Figure 9: View Instances

  22. Select the instance you just created and note the Public DNS address in the Description tab. Note: You will need this address and .pem file later on.

    Figure 10: Note the public DNS of your instance.

Step 2: Setting up SMRT Portal on the Amazon Machine Image - Create a SMRT Portal administrative user.

(You only need to do this once.)

  1. Use your web browser to access the cloud SMRT Portal instance by entering http://<public_dns>:8080/smrtportal.

  2. Click Register at the top right.

    Figure 11: Browse to SMRTPortal

  3. Create a user named administrator (all lowercase). This user is special, as it is the only user that does not require activation on creation.

  4. Enter the user name administrator.

  5. Enter an email address. All administrative emails, such as new user registrations, will be sent to this address. Note: You won’t actually be able to receive email unless you specify an email server that your instance of AMI can see.

  6. Enter the password and confirm the password.

  7. Click Register.

    Figure 12: Register the admin account

Step 3: Using SSH to Access your Instance

  1. For Mac OS or Linux, ensure that the permissions for the key pair file (with a .pem extension) you saved in Step 20 are correct. From a terminal window, enter the following:
    `$ chmod 400 KEYPAIR.pem`
    `$ ssh -i KEYPAIR.pem ubuntu@{amazon-dns}`
  1. For Windows, use PuTTY and PuTTYgen to convert your .pem to a .ppk file (once per key) then access the AMI. Open PuTTYgen and click Load. You need to view all files to see and select the .pem file.

    Figure 15: Open PuTTYgen and Load your .pem file.

  2. Click Save the private key.

    Figure 16: Save the private key.

  3. Close PuTTYgen and open PuTTY. Enter the public DNS address of your AMI in the hostname field.

    Figure 17: Enter the DNA address.

  4. In the left menu, expand SSH and click Auth. Browse to select the private key .ppk file you just made, then Open the connection to your AMI. When prompted, log in as ubuntu.

    Figure 18: Upload your private key into PuTTY.

Note: For the next step, you can upload data to SMRT Portal using one of three methods: FileZilla, scp, or mounting an existing S3 bucket.

Step 4a: Uploading Your Data to SMRT Portal using FileZilla (Windows)

  1. Download and install FileZilla, a free FTP client.

  2. In FileZilla, choose Edit > Settings, then click Connection > SFTP.

    Figure 13: Upload the new keyfile to FileZilla

  3. Click Add keyfile... and select either the .ppk file you generated with PuTTYgen (if you used that for SSH access) or the .pem file you downloaded in Step 14.

  4. If you selected the .pem file, you see a dialog box asking for permission to convert the file. Click Yes and save the file with a different name, such as smrtportal_filezilla.ppk.

  5. Click OK.

    Figure 14: Connect to your AMI

  6. Enter the Public DNS address (from Step 23) in the Host field.

  7. Enter ubuntu in the username field.

  8. Enter 22 in the Port field.

  9. Click Quickconnect to log on to the remote SMRT Portal instance.

  10. In the Remote Site box, enter /opt/smrtanalysis/userdata/inputs_dropbox. You see the path name change as FileZilla connects to the folder, as the name is a symlink. Proceed fearlessly!

  11. In the Local Site box, locate the files to transfer and drag them over to the Remote Site box. You can now use SMRT Portal to scan inputs and run analysis jobs.

Step 4b: Uploading Your Data to SMRT Portal using scp (Mac OS)

Open a terminal session and use scp to push, from your local machine, input data into the SMRT Portal dropbox:

    scp -r -i customer.pem <path to local data directory> ubuntu@{amazon-dns}:/opt/smrtanalysis/userdata/inputs_dropbox

Step 4c: Copying data from an existing S3 bucket to the EC2 instance

The simplest way to share data from an amazon S3 instance is to have the S3 owner make the data public through their S3 Console. Use this command in your AMI to copy the data into your current directory:

    $ wget http://s3.amazonaws.com/public_S3/folder/file.ext

If the data is private, you can use Amazon Web Services Command Line Interface (AWS CLI) to copy data from a S3 volume whose owner has shared the security key with you.

To import SMRT Cells you have copied into your server, do the following within SMRT Portal:

  1. Click Import and Manage.

  2. Click Import SMRT Cells.

  3. Click Add and enter the path to the data: /home/username/my_data.

  4. Click OK, then click Scan.

Step 5: Stopping or Terminating the Amazon Machine Instance

  • When the instance is stopped, you are not charged hourly, but you are charged for storage.
  • When the instance is terminated , you are not charged hourly, nor are you charged for storage.

  1. Click Instances, then right-click the running instance and choose Stop.
  2. Click Instances, then right-click a non-running instance and choose Terminate.

Pricing Information

http://aws.amazon.com/ec2/pricing/


For Research Use Only. Not for use in diagnostic procedures. © Copyright 2010 - 2014, Pacific Biosciences of California, Inc. All rights reserved. Information in this document is subject to change without notice. Pacific Biosciences assumes no responsibility for any errors or omissions in this document. Certain notices, terms, conditions and/or use restrictions may pertain to your use of Pacific Biosciences products and/or third party products. Please refer to the applicable Pacific Biosciences Terms and Conditions of Sale and to the applicable license terms at http://www.pacificbiosciences.com/licenses.html.