3. Method validation

4. Estimate confidence in identification

This step calculates the percentage of test samples with correct, close or wrong identification depending on the percentage of genes supporting the identification.

First, the top identification for each sample is retrieved.
Second, the top identifications are assigned to "bins" depending on the percentage of genes supporting the top identification, and for each bin the percentage of samples with correct, close and wrong identifications is calculated.

• The range of support is first divided into equal-sized bins
• The top identifications are assigned to bins depending on the percentage of genes supporting the identification
• For each bin, the percentage of correct, close and wrong identifications is estimated
  By default, confidence estimates are produced for between 1 and 10 bins (the number of bins can be adjusted freely). For example, if the number of bins is set to 5, five categories of 0-20%, >20-40%, >40-60%, >60-80% and >80-100% are created. A top identification supported by 37% of genes would be assigned the category >20-40%.

For each number of bins, a plot showing support for each bin is produced.

Based on these results, the optimal number of bins for the dataset needs to be specified.

Guidelines:

• Each bin should contain multiple samples to allow an estimate of the accuracy of identification for the bin
• There is a trade-off between too few bins and too many bins
  • Too few bins: loss of resolution
  • Too many bins: some bins may be unrepresentative due to the low number of samples, or even contain no samples at all
• Increasing the number of test samples may allow to also increase the number of bins and therefore the granularity of confidence assessments

Example:
The below figures shows confidence estimates of the same test data depending on whether support is divided into 3, 5 or 7 bins. Note that there are very few samples falling into the first (leftmost) bin.
The figure with 5 bins includes samples in all bins and contains more detailed information than the figure with 3 bins. While the figure with 7 bins contains even more detailed information, there are no samples in the first bin, and confidence can therefore not be estimated for this bin.
In this case, estimating confidence using 5 bins might be the preferred choice for a maximum granularity in the confidence assessments whilst being able to assess confidence for all bins.

Output: validation file

Confidence estimates table calibration_confidence_support.csv:
This file contains the probability of the top identification being correct, close or wrong, depending on the percentage of genes supporting the identification.
Example file:

range_support	probability_correct	probability_close	probability_wrong
[0,20]	0	0	100
(20,40]	78.57	21.43	0
(40,60]	82.14	17.86	0
(60,80]	100	0	0
(80,100]	100	0	0