3. Method validation

Overview

Method validation allows to assess the accuracy of identification depending on the percentage of genes supporting the identification.

The base command is:
gpid validate

Method validation is structured into three subcommands:
gpid validate prepare: Prepare input files for calibration by matching each validation sample against the reference dataset
gpid validate confidence: Estimate validation confidence for different numbers of support bins
gpid validate bins: Save validation confidence support probabilities for selected number of bins

Optionally, accuracy of identification to user-defined groups of species can also be calculated.

Validation dataset

For method validation, you need to provide a validation dataset with samples of known identity, the Validation dataset. This needs to be provided as a directory containing multiple unaligned genes, each comprising all retrieved sequences for the calibration samples.

Note: The validation dataset should be independent of the calibration dataset to avoid over-fitting. The guidelines and formatting requirements for the calibration and formatting datasets are identical.

Guidelines for assembling a good validation dataset

• Validation samples should be expert-identified, e.g. by taxonomists.
• Voucher specimens of the validation samples should ideally be deposited in a museum or herbarium to allow validation of the identification.
• Reflect the taxonomic diversity of species across the lineage of interest. This is to ensure that the method is validated for the entire lineage, including taxonomically challenging species.
• Reflect different data qualities. This helps to identify minimum thresholds for data quality.
• Sufficient number of validation samples. The more samples, the more reliable method validation will be.

Formatting requirements

• Gene sequences need to be fasta files ending with a .FNA suffix.
• Gene names need to match the gene names in the reference dataset.
• Each validation sample name needs to start with genus and species name <Genus>_<species>, followed by a unique identifier for the sample.
• Use underscores _ as separators in the sample names.
• Sample names in each file need to be identical.

Example contents of validation directory

Gene1.FNA:

>Entandrophragma_angolense_Validation_CQL_38
CTCCCATATAGGGGTGCTTGGTTGTGGGTGGGTTCTGAGATGATTCATTTAATGGAACTT
CCAAATCCAGATCCCTTAACTGGACGACCGGCACATGGTGGTCGAGATCGTCATACTTGT
ATTGCGATTCGAGATGTGTCTAAACTCAAAATAATCCTTGATAAAGCAGGTATTCCTTAC
>Entandrophragma_bussei_Validation_CQL_EB63
CTCCCATATAGGGGTGCTTGGTTGTGGGTGGGTTCTGAGATGATTCATTTAATGGAACTT
CCAAATCCAGACCCCTTAACTGGACGACCGGCACATGGTGGTCGAGATCGTCATACTTGT
ATTGCGATTCGAGATGTGTCTAAACTCAAAATAATCCTTGATAAAGCAGGTATTCCTTAC
>Entandrophragma_candollei_Validation_CQL_72
CTTCCGTATAGGGGTGCTTGGTTGTGGGTGGGTTCTGAGATGATTCATTTAATGGAACTT
CCAAATCCAGACCCCTTAACGGGACGACCGGCACATGGTGGTCGAGATCGTCATACTTGT
ATTGCGATTCGAGATGTGTCTAAACTGAAAATAATCCTTGACAAAGCAGGTGTTCCTTAC

Gene2.FNA:

>Entandrophragma_angolense_Validation_CQL_38
GTTGGGGATCTACACGGAGACCTTGATCAAGCGAGATGTGCACTTGAGATTGCTGGTGTC
>Entandrophragma_bussei_Validation_CQL_EB63
GTTGGGGATCTACACGGAGACCTTGATCAAGCGAGATGTGCACTTGAGATTGCTGGTGTC
>Entandrophragma_candollei_Validation_CQL_72
GCTGGGGATCTACATGGAGACCTTGATCAAGCAAGATGTGCACTTGAGATTGCTGGTGTC

Gene3.FNA:

>Entandrophragma_angolense_Validation_CQL_38
GAATCTCAAGGATTAACATCTCGACGATTGTGTCTTACATGTATATGTTCAACTCTAGCT
CTGATTAACAGTTCCGGCACGTTGGTTTCTGTACAAAAGGCAATTGCTTTGGAAGGAAAA
>Entandrophragma_bussei_Validation_CQL_EB63
GATATGTGTGGTGGTACAGGAAAATGGAAAGCTCTCAACAGAAAACGTGCTAAAGATGTT
TACGAGTTTACAGAATGTCCAAATTGTTATGGTCGTGGGAAACTTGTGTGTCCGGTTTGC
>Entandrophragma_candollei_Validation_CQL_72
GAATCTCAGATATCAACATCTCGCCGTTGGTGCCTTACGTGTATACTTACATGTATATGT
TCAACTCTAGCTCTGATTAACAGTTCCGGCACATTGGTTTCTGTACAAAAGGCAATTGCT

Prepare validation dataset for method validation

To prepare the validation dataset for method validation, run:
gpid validate prepare -r <reference dataset directory> -i <validation dataset directory>

Required options:
-r Reference directory containing one FASTA file per gene and the corresponding BLAST databases. This is usually prepared with gpid reference, see 1. Reference construction.
-i Validation dataset directory containing one FASTA file per gene

This matches each sample in the validation dataset against all samples in the reference dataset using BLAST, gene by gene. The results from this are used as input for the subsequent method validation analyses.

Two output files are created:

• validation/preparations/validation_blast.tsv
• validation/preparations/validation_prepared.rds

Species groups file

File specifying for each species to which user-defined group of closely-related species it belongs, such as species complex or genus.

Note: Providing a species groups file is optional.

Requirements:

• Comma-separated .csv file
• Each row corresponds to a reference species and the corresponding species group
• First column contains genus and species name
  • Use underscores _ as separator
  • Spelling needs to be exactly as for the reference species
• Second column contains name of species group

Example species group file
File species_groups.csv:

genus_species	species_group
Entandrophragma_angolense	Group_A
Entandrophragma_congoense	Group_A
Entandrophragma_bussei	Group_B
Entandrophragma_candollei	Group_B
Entandrophragma_delevoyi	Group_B

Method validation

Assess accuracy of identification depending on support

To assess the accuracy of identification depending on support for the identification, run:
gpid validate confidence [-i <prepared validation RDS>] -g <gene performance CSV> -t <filtering thresholds CSV>

Required:
-g Gene performance file produced by gpid calibrate genes. By default, this file is saved to calibration/calibration_gene_performance.csv.
-t Filtering thresholds file produced by gpid calibrate combine. By default, this file is saved to calibration/calibration_filtering_thresholds.csv.

Optional:
-i Intermediate RDS produced by gpid validate prepare. This file is saved by default to validation/preparations/validation_prepared.rds.

This calculates the percentage of validation samples with correct, close or wrong identification depending on the percentage of genes supporting the identification. This is conducted in the following steps:

1. The top identification for each sample and the percentage of genes supporting this identification are retrieved.
2. The percentage of support is divided into equal-sized "bins", and each identification is assigned to one of these bins depending on the percentage of genes supporting the top identification. For example, if the number of bins is set to 5, five categories of 0-20%, >20-40%, >40-60%, >60-80% and >80-100% are created. A top identification supported by 37% of genes would be assigned the category >20-40%.
3. For each bin, the percentage of samples with correct, close and wrong identifications is estimated. If a bin contains no samples, the percentages are given as NA.

Confidence estimates are produced for between 1 and 10 bins. For each number of bins, a plot showing support for each bin is produced. These plots are summarised in a single file:
validation/tests/validate_confidence.pdf

Specify the optimal number of confidence bins

Based on inspecting the results in validation/tests/validate_confidence.pdf, the optimal number of bins for the dataset needs to be specified. To do this, run:
gpid validate bins -b <number of bins> [-i <validate top IDs RDS>]

Required: -b The specified optimal number of bins

Optional: -i The file produced by gpid validate confidence. By default, this file is placed in validation/tests/validate_top_ids.rds

This command produces a table validation/validation_confidence_support.csv. It details the probability of the top identification being correct, close or wrong depending on the percentage of genes supporting the identification. This file can be used as input for sample identification.
Example file:

range_support	probability_correct	probability_close	probability_wrong
[0,20]	0	0	100
(20,40]	78.57	21.43	0
(40,60]	82.14	17.86	0
(60,80]	100	0	0
(80,100]	100	0	0

Guidelines:

• There should be no bins that contain no samples, as the accuracy of identification cannot be assessed in this case.
• Each bin should contain multiple samples to allow an estimate of the accuracy of identification for the bin
• There is a trade-off between too few bins and too many bins
  • Too few bins: loss of resolution
  • Too many bins: some bins may be unrepresentative due to the low number of samples, or even contain no samples at all
• Increasing the number of validation samples may allow to also increase the number of bins and therefore the granularity of confidence assessments

Example:
The below figures shows confidence estimates of the same validation data depending on whether support is divided into 3, 5 or 7 bins. Note that there are very few samples falling into the first (leftmost) bin.
The figure with 5 bins includes samples in all bins and contains more detailed information than the figure with 3 bins. While the figure with 7 bins contains even more detailed information, there are no samples in the first bin, and confidence can therefore not be estimated for this bin.
In this case, estimating confidence using 5 bins might be the preferred choice for a maximum granularity in the confidence assessments whilst being able to assess confidence for all bins.

Next step

Upon completion of Method calibration, continue to 4. Sample identification.