A Framework for Comparing and Selecting from Several Segmentation Methods in Absence of Ground Truth labels
Find the bioRxiv pre-print here!
And our poster from the HTAN Summer Meeting 2022 here!
Our method derives a consensus-based ground truth segmentation by combining and ablating the results from multiple candidate segmentations. The process is probabilistic, interpreting each segmented object as a spatial probability distribution of a true segmentation. Probability maps are combined and and resulting "ground-truth" map is derived, which is then used to iteratively weight the input segmentations and improve the overall agreement across methods as represented by the ground-truth result. The below figure illustrates this process in detail:
The 'Segmentation Evaluation Methods Breakdown.ipynb' notebook gives a step-by-step breakdown of the process used in the accompanying paper.
The eval.py script does all the work here. This depends on a config yaml file in this case just called config.yaml. The data inputs are different segmentation masks for the same samples coming from different segmentation methods. The idea is to develop an ensemble based approach to approximate a "ground truth" label for the segmentation against which each individual mask can be evaluated.
If you open the config file you'll see a list of methods and a specified data directory. The masks for each method are assumed to live under ${data_dir}/${method} and across methods each mask should share a commond name reflective of the underlying data sample it can be associated with. For an example see this directory on exacloud: ``/home/groups/ChangLab/dataset/HMS-TMA-TNP/DATA-03292022`. This is an example of actual inputs / outputs from running this script.
Two other important parameters in the config file are num_agree and radii. The former corresponds to the number of methods that must agree on a pixel (cell or not) to be called a true positive. The later is a hyper-parameter that indicates the amount of dilation about the centroid used to construct the probability segmentation masks.
In the eval.py script you'll see three command line flags that can be passed. To run the entire script you should pass all three. If you run them separately they must be computed in serial.