4. Reference Creation

Overview

EPITOME creates references from a set of input sequences that have passed the initial QC filters. Below is a descript of each step of the reference creation process.

Note

We are aware that this approach is not particularly robust to large structural variations. Please feel free to share any ideas about how we could improve this!

Sequence Clustering

The first step of reference creation is to split quality filtered sequences into closely related clusters (controlled using --dist_threshold). This is accomplished using a multi-step process, which was designed to handle very large data sets. A general overview of this process is described below:

1. Hierarchal clustering

• A subset of sequences are randomly selected (controlled using --max_cluster).
• The subset is split into clusters using the "complete linkage" of pairwise average nucleotide identity (ANI) between sequences (controlled using --dist_threshold).

2. Assign clusters

• Representatives of each cluster are selected.
• Remaining sequences are assigned to one of the clusters by comparing to the representatives using the pairwise ANI and the distance threshold.
• Unassigned sequences are passed to step 1.

This process is repeated until all sequences have been assigned into a cluster.

Note

Average nucleotide identity is estimated using sourmash.

Below is a visual example of the main clustering step (step 1 above). The x-axis is the genetic distance (1-[%ANI/100]) and the dashed line represents the distance threshold used to create clusters (branch colors are not relevant).

Consensus Generation

Reference sequences are created for each cluster by taking the most common nucleotide identity at each position. This is accomplished using the general process below:

1. Sequences are aligned within each cluster using mafft.
2. The frequency of each nucleotide identity is calculated for each position in the alignment and the most common nucleotide is returned. Nucleotide codes are selected at random for ambiguous bases (i.e., RYSWKMBDHVN; e.g., N = A | T | C | G). Nucleotides with the same frequency are selected at random.

Condensing References

References are condensed to avoid redundancy. This is necessary because the consensus generation step tends to normalize clusters into references that are more similar than the desired distance threshold. This process shares some similarities with the initial clustering process. A summary is provided below:

1. References are clustered based on their pairwise ANI using complete linkage.
2. The best reference is selected based on the number of QC'd input sequences that were used to create it. Ties are then selected based on the longest sequence length. Any remaining ties are then selected at random.

This process is repeated until all references are within the desired threshold.

Note

The sample data associated with a "condensed" reference is merged into the "selected" reference in the final summary.