3. Method validation
Method validation allows to assess the accuracy of identification depending on the percentage of genes supporting the identification.
For method validation, you need to provide a validation dataset with samples of known identity, the Validation dataset. This needs to be provided as a directory containing multiple unaligned genes, each comprising all retrieved sequences for the calibration samples.
Note: The validation dataset should be independent of the calibration dataset to avoid over-fitting. The guidelines and formatting requirements for the calibration and formatting datasets are identical.
- Validation samples should be expert-identified, e.g. by taxonomists.
- Voucher specimens of the validation samples should ideally be deposited in a museum or herbarium to allow validation of the identification.
- Reflect the taxonomic diversity of species across the lineage of interest. This is to ensure that the method is validated for the entire lineage, including taxonomically challenging species.
- Reflect different data qualities. This helps to identify minimum thresholds for data quality.
- Sufficient number of validation samples. The more samples, the more reliable method validation will be.
- Gene sequences need to be fasta files ending with a
.FNAsuffix. - Gene names need to match the gene names in the reference dataset.
- Each test sample name needs to start with genus and species name
<Genus>_<species>, followed by a unique identifier for the sample. - Use underscores
_as separators in the sample names. - Sample names in each file need to be identical.
Gene1.FNA:
>Entandrophragma_angolense_Validation_CQL_38
CTCCCATATAGGGGTGCTTGGTTGTGGGTGGGTTCTGAGATGATTCATTTAATGGAACTT
CCAAATCCAGATCCCTTAACTGGACGACCGGCACATGGTGGTCGAGATCGTCATACTTGT
ATTGCGATTCGAGATGTGTCTAAACTCAAAATAATCCTTGATAAAGCAGGTATTCCTTAC
>Entandrophragma_bussei_Validation_CQL_EB63
CTCCCATATAGGGGTGCTTGGTTGTGGGTGGGTTCTGAGATGATTCATTTAATGGAACTT
CCAAATCCAGACCCCTTAACTGGACGACCGGCACATGGTGGTCGAGATCGTCATACTTGT
ATTGCGATTCGAGATGTGTCTAAACTCAAAATAATCCTTGATAAAGCAGGTATTCCTTAC
>Entandrophragma_candollei_Validation_CQL_72
CTTCCGTATAGGGGTGCTTGGTTGTGGGTGGGTTCTGAGATGATTCATTTAATGGAACTT
CCAAATCCAGACCCCTTAACGGGACGACCGGCACATGGTGGTCGAGATCGTCATACTTGT
ATTGCGATTCGAGATGTGTCTAAACTGAAAATAATCCTTGACAAAGCAGGTGTTCCTTAC
Gene2.FNA:
>Entandrophragma_angolense_Validation_CQL_38
GTTGGGGATCTACACGGAGACCTTGATCAAGCGAGATGTGCACTTGAGATTGCTGGTGTC
>Entandrophragma_bussei_Validation_CQL_EB63
GTTGGGGATCTACACGGAGACCTTGATCAAGCGAGATGTGCACTTGAGATTGCTGGTGTC
>Entandrophragma_candollei_Validation_CQL_72
GCTGGGGATCTACATGGAGACCTTGATCAAGCAAGATGTGCACTTGAGATTGCTGGTGTC
Gene3.FNA:
>Entandrophragma_angolense_Validation_CQL_38
GAATCTCAAGGATTAACATCTCGACGATTGTGTCTTACATGTATATGTTCAACTCTAGCT
CTGATTAACAGTTCCGGCACGTTGGTTTCTGTACAAAAGGCAATTGCTTTGGAAGGAAAA
>Entandrophragma_bussei_Validation_CQL_EB63
GATATGTGTGGTGGTACAGGAAAATGGAAAGCTCTCAACAGAAAACGTGCTAAAGATGTT
TACGAGTTTACAGAATGTCCAAATTGTTATGGTCGTGGGAAACTTGTGTGTCCGGTTTGC
>Entandrophragma_candollei_Validation_CQL_72
GAATCTCAGATATCAACATCTCGCCGTTGGTGCCTTACGTGTATACTTACATGTATATGT
TCAACTCTAGCTCTGATTAACAGTTCCGGCACATTGGTTTCTGTACAAAAGGCAATTGCT
To prepare the validation dataset for method validation, run:
gpid validate prepare -r <reference dataset directory> -i <validation dataset directory>
Required options:
-r Reference directory containing one FASTA file per gene and the corresponding BLAST databases. This is usually prepared with gpid reference, see 1. Reference construction.
-i Validation dataset directory containing one FASTA file per gene
This matches each sample in the validation dataset against all samples in the reference dataset using BLAST, gene by gene. The results from this are used as input for the subsequent method validation analyses.
Two output files are created:
validation/preparations/validation_blast.tsvvalidation/preparations/validation_prepared.rds
To assess the accuracy of identification depending on support for the identification, run:
gpid validate confidence [-i <prepared validation RDS>] -g <gene performance CSV> -t <filtering thresholds CSV>
Required:
-g Gene performance file produced by gpid calibrate genes. By default, this file is saved to calibration/calibration_gene_performance.csv.
-t Filtering thresholds file produced by gpid calibrate combine. By default, this file is saved to calibration/calibration_filtering_thresholds.csv.
Optional:
-i Intermediate RDS produced by gpid validate prepare. This file is saved by default to validation/preparations/validation_prepared.rds.
This calculates the percentage of test samples with correct, close or wrong identification depending on the percentage of genes supporting the identification. This is conducted in the following steps:
- The top identification for each sample and the percentage of genes supporting this identification are retrieved.
- The percentage of support is divided into equal-sized "bins", and each identification is assigned to one of these bins depending on the percentage of genes supporting the top identification. For example, if the number of bins is set to 5, five categories of
0-20%,>20-40%,>40-60%,>60-80%and>80-100%are created. A top identification supported by 37% of genes would be assigned the category>20-40%. - For each bin, the percentage of samples with correct, close and wrong identifications is estimated. If a bin contains no samples, the percentages are given as
NA.
Confidence estimates are produced for between 1 and 10 bins. For each number of bins, a plot showing support for each bin is produced. These plots are summarised in a single file:
validation/tests/validate_confidence.pdf
Based on inspecting the results in validation/tests/validate_confidence.pdf, the optimal number of bins for the dataset needs to be specified. To do this, run:
gpid validate bins -b <number of bins> [-i <validate top IDs RDS>]
Required:
-b The specified optimal number of bins
Optional:
-i The file produced by gpid validate confidence. By default, this file is placed in validation/tests/validate_top_ids.rds
This command produces a table validation/validation_confidence_support.csv. It details the probability of the top identification being correct, close or wrong depending on the percentage of genes supporting the identification. This file can be used as input for sample identification.
Example file:
| range_support | probability_correct | probability_close | probability_wrong |
|---|---|---|---|
| [0,20] | 0 | 0 | 100 |
| (20,40] | 78.57 | 21.43 | 0 |
| (40,60] | 82.14 | 17.86 | 0 |
| (60,80] | 100 | 0 | 0 |
| (80,100] | 100 | 0 | 0 |
Guidelines:
- There should be no bins that contain no samples, as the accuracy of identification cannot be assessed in this case.
- Each bin should contain multiple samples to allow an estimate of the accuracy of identification for the bin
- There is a trade-off between too few bins and too many bins
- Too few bins: loss of resolution
- Too many bins: some bins may be unrepresentative due to the low number of samples, or even contain no samples at all
- Increasing the number of test samples may allow to also increase the number of bins and therefore the granularity of confidence assessments
Example:
The below figures shows confidence estimates of the same test data depending on whether support is divided into 3, 5 or 7 bins. Note that there are very few samples falling into the first (leftmost) bin.
The figure with 5 bins includes samples in all bins and contains more detailed information than the figure with 3 bins. While the figure with 7 bins contains even more detailed information, there are no samples in the first bin, and confidence can therefore not be estimated for this bin.
In this case, estimating confidence using 5 bins might be the preferred choice for a maximum granularity in the confidence assessments whilst being able to assess confidence for all bins.
Upon completion of Method calibration, continue to 4. Sample identification.