Walkthrough of DNA Subway Purple Line (alpha testing documentation)
Known alpha-version bugs: Purple line is in alpha release. There are several known issues. One common issue is that after completing a step, a subsequent step may be blocked. To solve, refresh DNA Subway in your browser to unblock this step. Please send feedback to DNALC Admin
The Purple Line provides the capability for analysis of microbiome and eDNA (environmental DNA) by implementing a simplified version of the QIIME 2 (pronounced "chime two") workflow. Using the Purple Line, you can analyze uploaded high throughput sequencing reads to identify species in microbial or environmental DNA samples.
Metabarcoding uses high-throughput sequencing to analyze hundreds of thousands of DNA barcodes from complex mixtures of DNA. In a typical experiment, DNA is isolated from sterile swabs or material taken from different environmental locations or conditions. PCR is used to amplify a variable region, such as COI, or 12S or 16S ribosomal RNA genes, and sequence reads identify the variety and abundance of species from different samples. The analysis requires specialized software, such as QIIME 2.
The Purple Line integrates sequence data and metadata imported from CyVerse's Data Store, demultiplexing of samples, quality control, and taxonomic identification and quantitation. Once sequences are analyzed, the results can be visualized to allow comparisons between samples and different conditions summarized in the metadata.
Some things to remember about the platform
- You must be a registered CyVerse user to use Purple Line (register for a CyVerse account at user.cyverse.org )
- The Purple line was designed to make microbiome/eDNA data analysis "simple". However, we ask that users very carefully and thoughtfully decide what "jobs" they want to submit.
- A single Purple Line project may take hours to process since HPC computing is subject to queues which may support hundreds of other jobs. These systems also undergo regular maintenance and are subject to periodic disruption.
- DNA Subway implements the QIIME 2 software. This software is in continual development. Our version may not be the most current, and our documentation and explanation is not meant to replace the full QIIME 2 documentation.
- We have made design decisions to create a straightforward classroom-friendly workflow. While this Subway Line does not have all possible features of QIIME 2, we purpose to cover important concepts behind microbiome and eDNA analysis.
Sample dataHow to use provided sample data
In this guide, we will use a microbiome dataset ("ubiome-test-data") collected from various water sources in Montana (down-sampled and de-identified). Where appropriate, a note (in this orange colored background) in the instructions will indicate which options to select to make use of this provided dataset.
DNA Subway Purple Line - Metadata file and Sequencing Prerequisites
For QIIME 2 to run, a valid metadata file is required. This file must conform to strict guidelines, or analyses will fail. QIIME 2 metadata is stored in a TSV (tab-separated values) file. These files typically have a .tsv or .txt file extension, though it doesn't matter to QIIME 2 what file extension is used. TSV files are simple text files used to store tabular data, and the format is supported by many types of software, such as editing, importing, and exporting from spreadsheet programs and databases. Thus, it's usually straightforward to manipulate QIIME 2 metadata using the software of your choosing. If in doubt, we recommend using a spreadsheet program such as Microsoft Excel or Google Sheets to edit and export your metadata files.
Here are a few reminders for formatting your metadata.
Leading and trailing whitespace characters
If any cell in the metadata contains leading or trailing whitespace characters (e.g. spaces, tabs), those characters will be ignored when the file is loaded. Thus, leading and trailing whitespace characters are not significant, so cells containing the values 'gut' and ' gut ' are equivalent. This rule is applied before any other rules described below
The first column MUST be the ID column name (i.e. ID header) and the first line of this column should be #SampleID or one of a few alternative.
- Case-insensitive: id; sampleid; sample id; sample-id; featureid; feature id; feature-id.
- Case-sensitive: #SampleID; #Sample ID; #OTUID; #OTU ID; sample_name
For the sample IDs, there are some simple rules to comply with QIIME 2 requirements:
- IDs may consist of any Unicode characters, with the exception that IDs must not start with the pound sign (#), as those rows would be interpreted as comments and ignored. IDs cannot be empty (i.e. they must consist of at least one character).
- IDs must be unique (exact string matching is performed to detect duplicates).
- At least one ID must be present in the file.
- IDs cannot use any of the reserved ID column names (the sample ID names, above).
- The ID column can optionally be followed by additional columns defining metadata associated with each sample or feature ID. Metadata files are not required to have additional metadata columns, so a file containing only an ID column is a valid QIIME 2 metadata file.
- May consist of any Unicode characters.
- Cannot be empty (i.e. column names must consist of at least one character).
- Must be unique (exact string matching is performed to detect duplicates) .
- Column names cannot use any of the reserved ID column names.
- May consist of any Unicode characters.
- Empty cells represent missing data. Note that cells consisting solely of whitespace characters are also interpreted as missing data.
QIIME 2 currently supports categorical and numeric metadata columns. By default, QIIME 2 will attempt to infer the type of each metadata column: if the column consists only of numbers or missing data, the column is inferred to be numeric. Otherwise, if the column contains any non-numeric values, the column is inferred to be categorical. Missing data (i.e. empty cells) are supported in categorical columns as well as numeric columns. For more details, and for how to define the nature of the data when needed, see the QIIME 2 metadata documentation.
A. Create Metadata file
Using a spreadsheet editor, create a metadata sheet that provides descriptions of the sequencing files used in your experiment. Export this file as a tab-delimited .txt or .tsv file. following the QIIME 2 metadata documentation recommendations.
See an example metadata file used for our sample data here: metadata file. Click the :guilabel:`&Download` button on the linked page to download and examine the file. (Note: This is an Excel version of the metadata file, you must save Excel files as .TSV (tab-separated) to be compatible with the QIIME 2 workflow.)
DNA Subway Purple Line - Create a Microbiome Analysis Project
A. Create a project in Subway
- Log-in to DNA Subway (unregistered users may NOT use Purple Line, register for a CyVerse account at user.cyverse.org)
- Click the purple square ("Microbiome Analysis") to begin a project.
Purple line can be used to analyze eDNA (Environmental DNA). The workflow is largely the same, with eDNA typically using a 12S RNA database of sequences for identification.
For 'Select Project Type' select either Single End Reads or Paired End Reads
Select Single End Reads
For 'Select File Format' select the format the corresponds to your sequence metadata.
Select Illumina Casava 1.8
- Enter a project title, and description; click :guilabel:`&Continue`.
B. Upload read data to CyVerse Data Store
The sequence read files used in these experiments are too large to upload using the Subway interface. You must upload your files (either .fastq or .fastq.gz) directly to the CyVerse Data Store:
- Upload your
- FASTQ sequence reads
- Sample metadata file (.tsv or .txt formatted according to QIIME 2 metadata documentation)to the CyVerse Data Store using Cyberduck. See instructions: CyVerse Data Store Guide.
DNA Subway Purple Line - Metadata and QC
A. Select files using Manage Data
- Click on the “Manage Data” stop: this opens a window prompting you to "Select your FASTQ files from the Data Store" (if you are not logged in to CyVerse, it will ask you to do so); click the add data link.
- Select your metadata file; click on the folder that matches your CyVerse username and Navigate to the folder where your sequencing files are located. Click :guilabel:`&Add selected files` to add your metadata file.
Navigate to: Shared Data > SEPA_microbiome_2016 > ubiome-test-data
Select the mappingfile_MT_corrected.tsv and then click :guilabel:`&Add selected files`.
- To validate the metadata file, click "validate sample mapping file", header columns will be displayed. Next, click :guilabel:`&Validate`.
- To add sequence data, click the "add data" link. Click on the folder that matches your CyVerse username and navigate to the folder where your sequencing files are located.
Navigate to: Shared Data > SEPA_microbiome_2016 > ubiome-test-data
Select all 11 fastq files (they are compressed and will have the fastq.gz file extension). Then click :guilabel:`&Add selected files` or :guilabel:`&Add all files in this directory` (only files with a .fastq.gz extension will be added).
- Click the "add data" link to add the sequencing data to your project. Close the "Manage data" window, or repeat this step as appropriate until all your sequence data files have been added.
Known alpha-version bug After adding data, the next stop (Demultiplex reads) will still be blocked. Refresh DNA Subway in your browser to unblock this step.
B. Demultiplex reads
At this step, reads will be grouped according to the sample metadata. This includes separating reads according to their index sequences if this was not done prior to running the Purple Line. For demultiplexing based on index sequences, the index sequences must be defined in the metadata file.
Even if your files were previously demultimplexed (as will generally be the case with Illumina data) you must still complete this step to have your sequence read files appropriately associated with metadata.
- Click the 'Demultiplex reads' stop, then click :guilabel:`&demux reads` to demultiplex your sample reads.
- In 'Random sequences to sample for QC', enter a value (1000 is recommended), then click demux reads to demultiplex your sample reads.
- When demultiplexing is complete, you will generate a file (.qzv) click this link to view a visualization and statistics on the sequence and metadata for this project.
C. Check sequencing quality and Trim Reads
It is important to only work with high quality data. This step will generate a sequence quality histogram which can be used to determine parameter for trimming.
Click the 'Demultiplex reads' stop, then click the results label ending in .qzv will appear. Click this link to view your results.
One of the features of QIIME 2 are the variety of visualizations provided at several analysis steps. Although this guide will not cover every feature of every visualization, here are some important points to note.
- QIIME2 View: DNA Subway uses the QIIME 2 View plugin to display visualizations. Like the standalone QIIME 2 software, you can navigate menus, and interact with several visualizations. Importantly, many files and visualizations can be directly download for your use outside of DNA Subway, including in report generation, or in your custom QIIME 2 analyses. You can view downloaded .qza or .qzv files at view.qiime2.org.
Quality Graphs Explained
After demultiplexing, you will be presented with a visualization that displays the following tables and graphs:
- Demultiplexed sequence counts summary: For each of the fastq files (each of which may generally correspond to a single sample), you are presented with comparative statistics on the number of sequences present. This is followed by a histogram that plots number of sequences by the number of samples.
- Per-sample sequence counts: These are the actual counts of sequences per sample as indicated by the sample names you provided in your metadata sheet.
Interactive Quality Plot
This is an interactive plot that gives you an average quality Phred score (y-axis) by the position along the read (x-axis). This box plot is derived from a random sampling of a subset of sequences. The number of sequences sampled will be indicated in the plot caption.
Click the "Interactive Quality Plot" tab to view a histogram of sequence quality. Use this plot at the tip below to determine a location to trim.
Tips on trimming for sequence quality
On the Interactive Quality Plot you are shown an histogram, plotting the average quality (X axis) Phred score vs. the position on the read (y axis) in base pairs for a subsample of reads.
Zooming to determine 3' trim location
Click and drag your mouse around a collection of base pair positions you wish to examine. Clicking on a given histogram bar will also generate a text report and metrics in the table below the chart. Using these metrics, you can choose a position to trim on the right side (e.g. 3' end of the sequence read). The 5' (left trim) is specific to your choice of primers and sequencing adaptors (e.g. the sum of the adaptor sequence you expect to be attached to the 5' end of the read). Poor quality metrics will generate a table colored in red, and those base positions will also be colored red in the histogram. Double-clicking will return the histogram to its original level of zoom.
It is important to maximize the length of the reads while minimizing the use of low quality base calls. To this end, a good guideline is to trim the right end of reads to a length where the 25th percentile is at a quality score of 25 or more. However, the length of trimming will depend on the quality of the sequence, so you may have to use a lower quality threshold to retain enough sequence for informative sequence searches and alignments. This may require multiple runs of the analysis to find the optimal trim length for your data.
Quality drops significantly at base 35
Improved quality sequence
- Click on the 'Trim reads' stop. Click :guilabel:`&run` and then select values for "trimLeft" (the position starting from the left you wish to trim) and "TruncLen" (this is the position where reads should be trimmed, truncating the 3' end of the read. Reads shorter than this length will be discarded). Finally, click the "trim reads" link.
Based on the histogram for our sample, we recommend the following parameters:
- trimLeft: 17 (this is specific to primers and adaptors in this experiment)
- TruncLen: 200 (this is where low quality sequence begins, in this case because our sequence length is lower than the expected read length)
D. Check Results of Trimming Once trimming is complete, the following outputs are expected:
Click on the generated result links to view summary statistics on your sequences.
QIIME 2 output names
Naming of QIIME outputs in Purple Line will often contain a 4-digit number corresponding to a job number on the computing system the analysis was completed on. In this documentation four octothorps (####) will be used in place of the numbers.
- ####.table-trim####.qzv: This file summarizes the dataset post-trimming including the number of samples and the number of features per sample. The "Interactive Sample Detail" tab contains a sampling depth tool that will be used in computation of the core matrix.
- ####.re-seqs.gzv: This table contains a listing of features observed in the sequence data, as well as the DNA sequence that defines a feature. Clicking on the DNA sequence will submit that sequence for BLAST at NCBI in a separate browser tab.
The feature table contains two columns output by DADA2. DADA2 (Divisive Amplicon Denoising Algorithm 2) determines what sequences are in the samples. DADA2 filters the sequences and identifies probable amplification or sequencing errors, filters out chimeric reads, and can pair forward and reverse reads to create the best representation of the sequences actually found in the samples and eliminating erroneous sequences.
- Feature ID: A unique identifier for sequences.
- Sequence: A DNA Sequence associated with each identifier.
Clicking on any given sequence will initiate at BLAST search on the NCBI website. Click "View report" on the BLAST search that opens in a new web browser tab to obtain your results. Keep in mind that if your sequences are short (due to read length or trimming) many BLAST searches may not return significant results.
Although the term "feature" can (unfortunately) have many meanings as used by the QIIME2 documentation, unless otherwise noted in this documentation it can be thought of as an OTU (operational taxonomic unit); another substitution for the word species. OTU is a convenient and common terminology for referring to an unclassified or undetermined species. Ultimately, we are attempting to identify an organism from a sample of DNA which may not be informative enough to reach a definitive conclusion.
DNA Subway Purple Line - Cluster Sequences
At this step, you can visualize summaries of the data. A feature table will generate summary statistics, including how many sequences are associated with each sample.
- Click 'Feature table' and then the "Build feature table" link. When processed, you will get a link to a visualization file (.qzv). Open this file to examine your results. The QIIME 2 view window will also have a link to download a FASTA file of your sequences.
- Click on 'Alpha rarefaction. Select "run" and designate the minimum and maximum rarefaction depth. A minimum value should be set at 1. The maximum value is specific to your data set. To determine what the maximum value should be set to, open the "Interactive Sample Detail" tab of the "Trim Reads" step. Identify the maximum Sequence Count value and enter that number as the maximum value. Click "submit job".
DNA Subway Purple Line - Calculate Alpha and Beta Diversity
At this stop, you will examine Alpha Diversity (the diversity of species/taxa present within a single sample) and Beta Diversity (a comparison of species/taxa diversity between two or more samples). Alpha diversity answers the question - "how many species are in a sample?"; beta diversity answer the question - "what are the differences in species between samples?".
Known alpha-version bug After computing Core matrix, other diversity steps may be blocked. Refresh DNA Subway in your browser to unblock these steps.
A. Calculate core matrix
Click on 'Core matrix' and then click the "run" link. Choose a sampling depth based upon the "Sampling depth" tool (described in Section D Step 1, in the table-trim####.qzv output; Interactive Sample Detail tab). Choose an appropriate classifier (see comments in the tip below) and click :guilabel:`&Submit job`.
Choosing Core matrix parameters
In downstream steps, you will need to choose a sampling depth for your sample comparisons. You can choose by examining the table generated at the Trim reads step. In the table-trim####.qzv output, Interactive Sample Detail tab, use the "Sampling depth" tool to explore how many sequences can be sampled during the Core matrix computation. As you slide the bar to the right, more sequences are sampled, but samples that do not have this many sequences will be removed during analysis. The sampling depth affects the number of sequences that will be analyzed for taxonomy in later steps: as the sampling depth increases, a greater representation of the sequences will be analyzed. However, high sampling depth could exclude important samples, so a balance between depth and retaining samples in the analysis must be found.
Classifier Choose a classifier pertaining to your experiment type. For Microbiome choose Grenegenes (16s rRNA) classifier. For an eDNA experiment chose Custom 12s rRNA, take 3 or if you are specifically looking for marine fishes you may elect to choose the Mitofish JO classifier.
We recommend the following parameters:
- Sampling Depth: 3000
- Classifier: Grenegenes (16s rRNA)
When complete, you should generate several visualization results including:
####.bray-curtis-emperor.qzv: Three-dimensional PCoA (principle coordinates analysis) plots
####.eveness-correlation.qzva.qzv: Measure of community evenness using correlation tests
####.eveness-group-significance.qzv: Analysis of differences between features across group
####.faith-pd-correlation.qzv: Faith Phylogenetic Diversity (a measure of community richness) with correlation tests
####.faith-pd-group-significance.qzv: Faith Phylogenetic Diversity ( a measure of community richness)
####.taxa-bar-plots.qzv: An interactive stacked bar plot of species diversity
####.taxononmy.gzv: A table indicating the identified "features", their taxa, and an indication of confidence.
####.unweighted unifrac-emporor.qzv: unweighted interactive PCoA plot
You can download and interact with any of the available plots.
B. Calculate Alpha diversity
- Click on the 'Alpha diversity' stop. Then click the "Build alpha diversity" link. No visualization will be created.
C. Calculate Beta diversity
- Click on the 'Beta diversity' stop. Then click the "Build beta diversity" link. No visualization will be created.
D. Calculate Taxonomic diversity
- Click on the 'Taxonomic diversity' stop and click the "Process diversity" link. Results generated will include several visualizations:
- .taxa-bar-plots.qzv: An interactive stacked bar plot of species diversity.
- .taxononmy.gzv: A table indicating the identified "features", their taxa, and an indication of confidence.
- Other expected results: [MORE INFO]
E. Calculate differential abundance
- Click on the 'Differential abundance' stop. Then click on the "Submit new "Differential abundance" job" link. Choose a metadata category to group by, and a level of taxonomy to summarize by. Then click :guilabel:`&submit job`.
We recommend the following parameters:
- Group data by: CollectionMethod
- Level of taxonomy to summarize: 5
DNA Subway Purple Line - Visualize data with PiCrust and PhyloSeq
More help and additional information
- Search for an answer:
- CyVerse Learning Center or CyVerse Wiki
- Post your question to the user forum:
- Ask CyVerse
Fix or improve this documentation: