Bill's test of class. err. vs prior for model (with bootstrap) and for data

[abhranildas]

To see whether a normal model is adequate to explain data from a classification experiment, we want to calculate the likelihood of seeing that classification (error rates) under a normal model, i.e. whether they are within the errorbar of the expected error rates. This test is more relevant for a classification task than a normality test of the individual distributions.
Likelihood of observed confusion matrix can be expressed as product of binomials over the independent elements.
For multiple classes, you can vary each of the priors to get different families of boundaries.

Perhaps a plot of the two distributions, with a family of boundaries going from one to the other. Nope, we are doing it in 6D.

• Write in the paper about how there are multiple criteria (linear, quadratic) and multiple families of them (shift the linear classifier from one normal to the other, or shift the quadratic by changing the prior ratio, or shift the quadratic by changing the variance ratio). The way we change it with the prior ratio is a sensible one that is a higher-dimensional extension of shifting it from one to the other in 2D by changing the prior odds. Also change the separation of the means. These are the two main things that could be happening with a subject: change of d', and change of criterion. Other sensible criteria are ones that ignore certain cues or covariances.

• Write in the paper about how these tests can be done even for high-D, when the normality of data cannot be easily seen. This could reveal structure.

• You could also compute the likelihood of the entire confusion matrix. But plotting p11 and p22 gives you more insight. Eg, we can see that p(2|2) is failing more, which is consistent with how we see that dist 2 is less normal.

• p(a|a) instead of p_aa etc.

• This could be done for multi-class as well, by varying the prior and covariance scales, for each normals.