Here are my notes for building and EC2 image on Amazon Web Services (AWS). These notes are created specifically for creating a enviroment that uses a Jupyter Notebook Server to work with deep learning algorithms in Python with Tensorflow. There is also information for installing bioinformatic tools to handle sequencing data.
After setting up the EC2 image on AWS, you need to attach volume so you have disk space. The volume size is super flexable, whatever you put on the volume is there and can be attached and detached from any image. Think of as a flash drive. You pay for the size of the volume as long as it is active. If you only want to pay for the storage you are using, you can take a snapshot, then shut down your volume. You pay for snapshot size. The snapshot can then be restarted on a new volume.
Steps for attaching volume
- Start old snapshot as a new volume
- Attach volume to instance in gui
- ONLY FIRST TIME USING IT: Create file system on your volume:
sudo mkfs -t ext4 device_name. I ran into a bug in which this doesn't work, but this does:mkfs -t ext4 -L rootfs device_name sudo mount device_name mount_point(examplesudo mount /dev/xvdf data- change permission for directory:
sudo chmod 777 -R mount_point
Useful commands
lsblk- look at your volumessudo file -s /dev/xvdfask if your volume is formatted. If output is data then it is not formatted.df- ask how much spacedf -Th
* There are ways to assign a volume to an instance and automatically attach at every bootup. But currently not using these, but def useful in future.
Steps for detaching volume
- take snapshot of your volume
- unmount:
umount -d /dev/sdh - detach volume
- delete volume
If you change the size of the volume you attach, you have to tell your image to resize the image or it only sees the previous size.
resize2fs /dev/xvda1
ssh
Example:
ssh -i key.pem ubuntu@ec2-54-67-80-121.us-west-1.compute.amazonaws.com
- don't forget the
ubuntu@
Just go to server address. Make sure you change password right away.
<server IP>/julia
<server IP>/shiny/rstudio
Make sure your final directory has the right permissions.
Example: sudo scp -r -i ~/Downloads/aws2.pem final_assemblies_10_15_17/ ubuntu@54.183.227.160:/home/data
apt-get install bedtools
but then I just did this: sudo apt install ncbi-blast+
Followed these directions:
wget https://github.com/arq5x/bedtools2/releases/download/v2.25.0/bedtools-2.25.0.tar.gz
pip install seqmagick
Install:
apt-get install bison
git clone https://github.com/lh3/bioawk
cd bioawk
sudo make
sudo cp bioawk /usr/local/bin/
Install:
sudo git clone https://github.com/lh3/seqtk.git
cd seqtk
sudo make
sudo cp seqtk /usr/local/bin/
sudo apt install gnome-system-monitor
sudo apt insall fastx-toolkit
git clone https://github.com/scapella/trimal.git
cd trimal /usr/local/bin
samtools faidx test.fa two
sudo wget https://sourceforge.net/projects/bbmap/files/latest/download
tar -xvzf download
sudo apt install linuxbrew-wrapper
sudo wget https://sco.h-its.org/exelixis/resource/download/software/Parsimonator-1.0.1.tar.bz2
sudo tar -vxjf Parsimonator-1.0.1.tar.bz2
cd Parsimonator-1.0.1
make -f Makefile.gcc
(pretty useless)
sudo git clone https://github.com/jimhester/fasta_utilities.git
sudo perl Makefile.PL PREFIX=$HOME (there was some prereqs missing
sudo make
sudo make install
sudo npm install -g dat
What is the difference between stopping and terminating the instance?
When you stop an instance, we shut it down. We don't charge usage for a stopped instance, or data transfer fees, but we do charge for the storage for any Amazon EBS volumes. ref
Called Bearbuy: Phone: (510) 664-9000 option 1, then option 2
Contact?: aws-uc-procurement@amazon.com
Storage: block Storage for EC2
https://aws.amazon.com/premiumsupport/compare-plans/
keycode: https://aws.amazon.com/premiumsupport/compare-plans/
Basically, I followed this great medium article and This dude created and maintains a sweet Image with everything you need.