Evaluation
The official metric for evaluating the multi-class classification is the multi-class generalization of the Matthews correlation coefficient (MCC). This generalization called the R_K statistic (for K different classes) and defined in terms of a KxK confusion matrix C.
The R_K statistic is in essence a correlation coefficient between the observed and predicted binary classifications for (for K different classes); it returns a value between −1 and +1. A coefficient of +1 represents a perfect prediction, 0 no better than random prediction and a value <0 indicates disagreement between prediction and observation (the lower negative value corresponds to the stronger disagreement). The minimum negative value of the R_K statistic will be between -1 and 0 depending on the true distribution. The maximum value is always +1.
For the performance sub-task the evaluation will be performed based on time and detection performance running all submitted code on the same machine. Details about the machine will be released asap.
*** Important note ***
The training and test datasets provided contain a number of images can be classified into several classes simultaneously, for example image 25169200.jpg in the test set contains both instrument and polyp. As a general rule in such the cases, we expect a single-label classification based on the importance of the object types. Here, presence of the instrument in the frame means that doctor already spotted the polyp and doing some treatment procedures. On the to-be-implemented-in-future analysis steps instrument type detection, its segmentation and masking will be performed before lesion or landmark type detection, thus instrument is more important class for the moment and it should be classifier output. We define object types importance as following (for all the classes provided, from most important to less important):
- out-of-patient
- instruments
- dyed-lifted-polyps
- dyed-resection-margins
- polyps
- esophagitis
- ulcerative-colitis
- retroflex-rectum
- retroflex-stomach
- normal-cecum
- normal-pylorus
- normal-z-line
- stool-plenty
- stool-inclusions
- colon-clear
- blurry-nothing
However, if your classifier can provide output for several classes per one image, you can add several output lines per one image in the runs submission files. First, we will evaluate all the participants using single-label rule (your first detection will be used as your classifier output). Next, we will separately evaluate only participants provided multi-label classification output.
For the automatic reporting subtask the evaluation will be performed by three medical experts. Details about the requirements are announced with the release of the test videos (see schedule). Expected is a rather open process beside of some minimal requirements (for example, create a report showing each detected disease in the video only one time). The visual design and look can be seen as part of the challenge (how to present the results in the best way to the medical experts).