Skip to content

Commit

Permalink
docs finalized
Browse files Browse the repository at this point in the history
  • Loading branch information
György Kovács committed Oct 17, 2023
1 parent e063bb4 commit 1ffcc74
Show file tree
Hide file tree
Showing 2 changed files with 5 additions and 13 deletions.
2 changes: 1 addition & 1 deletion README.rst
Expand Up @@ -569,7 +569,7 @@ Experts in various fields are kindly invited to contribute further test bundles
Retinal vessel segmentation
---------------------------

One such field is retinal vessel segmentation [RV]_, where the authors have the option to include or exclude certain parts of the images (the pixels outside the Field-of-View), making the reported scores incomparable. To facilitate the meaningful comparison, evaluation and interpretation of reported scores, we provide two functions to check the internal consistency of scores reported for the DRIVE retinal vessel segmentation dataset.
The segmentation of the vasculature in retinal images [RV]_ gained enormous interest in the recent decades. Typically, the authors have the option to include or exclude certain parts of the images (the pixels outside the Field-of-View), making the reported scores incomparable. (For more details see [RV]_.) To facilitate the meaningful comparison, evaluation and interpretation of reported scores, we provide two functions to check the internal consistency of scores reported for the DRIVE retinal vessel segmentation dataset.

The first function enables the testing of performance scores reported for specific test images. Two tests are executed, one assuming the use of the FoV mask (excluding the pixels outside the FoV) and the other assuming the neglect of the FoV mask (including the pixels outside the FoV). As the following example illustrates, one simply provides the scores and specifies the image (whether it is from the 'test' or 'train' subset and the image identifier) and the consistency results with the two assumptions are returned.

Expand Down
16 changes: 4 additions & 12 deletions docs/02b_retina.rst
@@ -1,17 +1,9 @@
Retinal vessel segmentation
===========================
---------------------------

The segmentation of the vasculature in retinal images gained enormous attention in the recent decade. Recently, using the tools implemented in this package, it was shown in [RV]_ that the authors use at least two different evaluation techniques, leading to incomparable performance scores and skewing the field. Retinal images have valueable image content only in the circular Field-of-View (FoV) region:
The segmentation of the vasculature in retinal images [RV]_ gained enormous interest in the recent decades. Typically, the authors have the option to include or exclude certain parts of the images (the pixels outside the Field-of-View), making the reported scores incomparable. (For more details see [RV]_.) To facilitate the meaningful comparison, evaluation and interpretation of reported scores, we provide two functions to check the internal consistency of scores reported for the DRIVE retinal vessel segmentation dataset.

.. image:: https://rumc-gcorg-p-public.s3.amazonaws.com/i/2020/01/21/75d803ef.png

When a binary segmentation of the vasculature is evaluated, some authors do calculate the performance scores only in the FoV region, others involve also the pixels outside the FoV region, which increases the number of true negatives (and consequently, accuracy and specificity) enormously. The functionalities implemented in this package are suitable to distinguish the two kinds of evaluations based on the reported performance scores.

The package contains the statistics of the images of the DRIVE dataset, and provides the functionality of checking the consistency of reported scores with the assumptions of using the FoV pixels only or using all pixels for evaluation, for both image level figures and score aggregated for all images of the DRIVE dataset.

Again, we highlight, that the techniques in the package detect inconsistency with certainty. If the use of the FoV pixels only was found to be inconsistent with a certain set of scores, the user can conclude that these scores are not comparable with other scores which are calculated in the FoV region.

The first function enables the testing of performance scores reported for certain test images, the two tests executed assume the use of the FoV mask (excluding the pixels outside the FoV) and the neglection of the FoV mask (including the pixels outside the FoV). As the following example shows, one simply supplies the scores and specifies the images (whether it is from the 'test' or 'train' subset and the identifier of the image) and gets back if inconsistency is identified with any of the two assumptions.
The first function enables the testing of performance scores reported for specific test images. Two tests are executed, one assuming the use of the FoV mask (excluding the pixels outside the FoV) and the other assuming the neglect of the FoV mask (including the pixels outside the FoV). As the following example illustrates, one simply provides the scores and specifies the image (whether it is from the 'test' or 'train' subset and the image identifier) and the consistency results with the two assumptions are returned.

.. code-block:: Python
Expand All @@ -23,7 +15,7 @@ The first function enables the testing of performance scores reported for certai
identifier='01')
# {'fov_inconsistency': True, 'no_fov_inconsistency': True}
The interpretation of these results is that the reported scores are inconsistent with any of the reasonable evaluation methodolgoies.
The interpretation of these results is that the reported scores are inconsistent with any of the reasonable evaluation methodologies.

A similar functionality is provided for the aggregated scores calculated on the DRIVE images, in this case the two assumptions of using the pixels outside the FoV is extended with two assumptions on the way of aggregation.

Expand Down

0 comments on commit 1ffcc74

Please sign in to comment.