Repository navigation
Project 1
Analysis of data from Cartography of opportunistic pathogens and antibiotic resistance genes in a tertiary hospital environment
You will need to log into your newly created VM by clicking the 'SSH' link.
cd
mkdir project1
cd project1
We've done this task for you because it is complex, time consuming, and illustrates a problem with the way data is included from many papers today. Little thought is given to how the data can be accessed even when it is "publicly available". To get the dataset to do further analysis, here's what we did Project 1: Get IDs. You will just copy the processed list of accessions into a work directory
cd ~/project1
mkdir step1
cd step1
gsutil cp gs://pathogen-workshop/project1/step1/singapore_samples_srafull.txt .
We've used SKESA to assemble the isolates because it is a fast and high-accuracy assembler. You could use another, more thorough assembler such as SPAdes to gain some additional sequence, but at the cost of some expense and accuracy.
We're going to set you up to do an assembly in parallel using dsub. In the interests of saving some compute we'll set up for computation of just two assemblies, but the same setup was used to do all 106 assemblies from the paper.
cd ~/project1
mkdir step2
cd step2
# Link in the sample accession file (from the previous step)
ln -s ../step1/singapore_samples_srafull.txt
# Small script to create the submit_list.tsv file for dsub
cat <<END > create_submit_list.sh
cut -f 3 singapore_samples_srafull.txt \
| perl -ne 'BEGIN { print "--env ERR\t--output ASM\n" }
chomp;
print join("\t",
"\$_",
"gs://asm-ngs-$USER/project1/step2/asm/\$_.asm.fa"), "\n"
'
END
bash create_submit_list.sh > submit_list.tsv
# For the workshop we want to just run two assemblies
head -3 submit_list.tsv > submit_list_two.tsv
# The assembly command. I like to create a script for easier editing and record-keeping
cat > assemble <<'END'
dsub \\
--provider google-cls-v2 \\
--project ncbi-pd-amr \\
--regions us-central1 \\
--logging gs://asm-ngs-$USER/project1/logs \\
--image staphb/skesa \\
--tasks submit_list_two.tsv \\
--command 'skesa --sra_run ${ERR} --contigs_out ${ASM} --cores 6 ' \\
--min-ram 24 \\
--min-cores 6 \\
--wait
END
# Run the assemblies in parallel on new VMs using dsub
# You should see new VMs created and running in the VM instances list
time bash -x assemble 2>&1 > assemble.out
You should have to wait 5-10 minutes for this to finish.
# copy the assemblies from the bucket onto your local disk.
mkdir asm
gsutil -m cp gs://asm-ngs-$USER/acinetobacter/asm/* asm/
ls -l asm/*
mkdir ~/project1/step3
cd ~/project1/step3
...