Submissions

To submit please zip the submission files and send the zip archive via regular e-mail to the task organizers Pål Halvorsen (paalh at simula.no), Steven Hicks (steven at simula.no) and Michael Riegler (Michael at simula.no).

The subtasks are the following:

• Detection subtask - id: detection. The classification has to be achieved using visual information. The goal is to maximize the algorithm's performance in terms of detection accuracy. The used amount of training data will also be taken into account.

• Efficient detection - id: speed. The classification should be achieved as fast as possible in terms of data processing speed with the use of the visual information and any computation speed-up techniques. The goal is to find a balance between the algorithm's performance in terms of detection accuracy and the performance in terms of data processing speed, while keeping in mind that the problem area requires real-time processing and has lack of data. There are no constraints on hardware and software resources participants can use for this submission, but we kindly ask the participants to stay within the boundaries of conventional hardware widely available on the market (please avoid GPU-enabled mining farms and powerful supercomputers).

• Efficient detection (the same hardware for all the participants) - id: hardware. The classification is to be achieved as fast as possible in terms of data processing speed with the use of the visual information and any computation speed-up techniques. The code and the compilation instructions should be provided to the task organizers instead of the classification results. All submitted solutions will be tested on the same hardware: [TBA].

• Report generation - id: report. Automatically create a text-report for a medical doctor with the detection results. [A list of requirements will be provided to the participants together with the video dataset].

Submission format

Submissions should be submitted in separated files with the following names, depending on the subtask:

 gc19bio_GroupName_SubmissionName_SubmissionId_TrainingSamplesUsed.txt

GroupName: the name of the participating group, maximum 12 symbols (example: colonwizards)

SubmissionName: the id of the submission, maximum 9 symbols, one of the following ids: detection, speed, hardware or report (example: detection).

SubmissionId: the id of the submission, maximum 15 symbols (example: ver1)

TrainingSamplesUsed: the number of samples in all the classes used to train the classification algorithm. This value will be used to rank the reported performance of runs in case of equally good results showed by different teams. The lower value is better.

Examples: gc19bio_colonwizards_detection_ver1_2000.txt

For all submissions regarding the detection, the files should report the multi-class classification of each image together with a confidence value / interestingness level / rank and (for the efficient detection subtask only) image processing time, according to the following format:

Detection subtask (comma-separated):

imagename,[classification decision, one of class names from the following list: `blurry-nothing`, `colon-clear`, `dyed-lifted-polyps`, `dyed-resection-margins`, `esophagitis`, `instruments`, `normal-cecum`, `normal-pylorus`, `normal-z-line`, `out-of-patient`, `polyps`, `retroflex-rectum`, `retroflex-stomach`, `stool-inclusions`, `stool-plenty` or `ulcerative-colitis`],[confidence value / interestingness level / rank, floating point value]

Efficient detection subtask (comma-separated):

imagename,[classification decision, one of class names from the following list: `blurry-nothing`, `colon-clear`, `dyed-lifted-polyps`, `dyed-resection-margins`, `esophagitis`, `instruments`, `normal-cecum`, `normal-pylorus`, `normal-z-line`, `out-of-patient`, `polyps`, `retroflex-rectum`, `retroflex-stomach`, `stool-inclusions`, `stool-plenty` or `ulcerative-colitis`],[confidence value / interestingness level / rank, floating point value],[image processing time in seconds, floating point value]

Efficient detection (the same hardware for all the participants) subtask: A Docker image containing your code. More details can be found here.

Report generation (Experimental) subtask: One report per video.

Ranking

In all the submissions the confidence value / interestingness level / rank value refers to a rank orders, expressing an expectation that some classified instances are more likely to be positive / correctly classified than others. The rank value is expected to be a floating point value within [0.0;1.0] interval deciding for any pair of instances whether the first is more likely (>), equally likely (=), or less likely (<) to be positive / correctly classified than the second. If your algorithm does not provide any ranking for the classified instances, you must use 1.0 as the rank value.

Processing Speed

In all the Efficient detection submissions the image processing time in seconds value refers to a time interval from the moment when image has been completely loaded into memory to the moment of the final decision about the image class made by the classification algorithm. In other words, the full amount of time required to classify a particular image minus time spent to image loading and classification results saving. This image processing time referrers to the classification of the images from the test dataset only, you do not need to include algorithm training time.

Example for the detection subtask:

0014d89e-5afa-4186-99f9-4400c8c58a18.jpg,polyp,0.831

ae68bef5-8ca4-42fd-a275-d34ee4ea8c14_00000074.jpg,dyed-resection-margins,0.1234

...

Example for the efficient detection subtask:

0014d89e-5afa-4186-99f9-4400c8c58a18.jpg,polyp,0.831,2.09

ae68bef5-8ca4-42fd-a275-d34ee4ea8c14_00000074.jpg,dyed-resection-margins,0.1234,0.529

...

Example for the efficient detection (the same hardware for all the participants) subtask:

A Docker image in the format described in this repository.

Example for the report generation (experimental) subtask:

One report per video.