Here are my notes for building and EC2 image on Amazon Web Services (AWS). These notes are created specifically for creating a enviroment that uses a Jupyter Notebook Server to work with deep learning algorithms in Python with Tensorflow. There is also information for installing bioinformatic tools to handle sequencing data.
After setting up the EC2 image on AWS, you need to attach volume so you have disk space. The volume size is super flexable, whatever you put on the volume is there and can be attached and detached from any image. Think of as a flash drive. You pay for the size of the volume as long as it is active. If you only want to pay for the storage you are using, you can take a snapshot, then shut down your volume. You pay for snapshot size. The snapshot can then be restarted on a new volume.
Steps for attaching volume
- Start old snapshot as a new volume
- Attach volume to instance in gui
- ONLY FIRST TIME USING IT: Create file system on your volume:
sudo mkfs -t ext4 device_name. I ran into a bug in which this doesn't work, but this does:
mkfs -t ext4 -L rootfs device_name
sudo mount device_name mount_point(example
sudo mount /dev/xvdf data
- change permission for directory:
sudo chmod 777 -R mount_point
lsblk- look at your volumes
sudo file -s /dev/xvdfask if your volume is formatted. If output is data then it is not formatted.
df- ask how much space
* There are ways to assign a volume to an instance and automatically attach at every bootup. But currently not using these, but def useful in future.
Steps for detaching volume
- take snapshot of your volume
umount -d /dev/sdh
- detach volume
- delete volume
If you change the size of the volume you attach, you have to tell your image to resize the image or it only sees the previous size.
More info on Volume
Connecting to your image
ssh -i key.pem email@example.com
- don't forget the
Connecting to Rstudio
Just go to server address. Make sure you change password right away.
Connecting to Ijupyter Notebook (Python and Julia)
Make sure your final directory has the right permissions.
sudo scp -r -i ~/Downloads/aws2.pem final_assemblies_10_15_17/ firstname.lastname@example.org:/home/data
Image Add ons
apt-get install bedtools
but then I just did this:
sudo apt install ncbi-blast+
Followed these directions:
seqmagick for file
pip install seqmagick
apt-get install bison
git clone https://github.com/lh3/bioawk
sudo cp bioawk /usr/local/bin/
sudo git clone https://github.com/lh3/seqtk.git
sudo cp seqtk /usr/local/bin/
sudo apt install gnome-system-monitor
sudo apt insall fastx-toolkit
git clone https://github.com/scapella/trimal.git
cd trimal /usr/local/bin
samtools faidx test.fa two
sudo wget https://sourceforge.net/projects/bbmap/files/latest/download
tar -xvzf download
sudo apt install linuxbrew-wrapper
sudo wget https://sco.h-its.org/exelixis/resource/download/software/Parsimonator-1.0.1.tar.bz2
sudo tar -vxjf Parsimonator-1.0.1.tar.bz2
make -f Makefile.gcc
sudo git clone https://github.com/jimhester/fasta_utilities.git
sudo perl Makefile.PL PREFIX=$HOME (there was some prereqs missing
sudo make install
sudo npm install -g dat
What is the difference between stopping and terminating the instance?
When you stop an instance, we shut it down. We don't charge usage for a stopped instance, or data transfer fees, but we do charge for the storage for any Amazon EBS volumes. ref
Contacting and Research at UC Berkeley
Called Bearbuy: Phone: (510) 664-9000 option 1, then option 2
Storage: block Storage for EC2