Improve sct_label_vertebrae using a deep learning-based approach

[joshuacwnewton]

NB: This issue has been migrated from the spinalcordtoolbox repository: spinalcordtoolbox/spinalcordtoolbox#3793

Context

The sct_label_vertebrae function currently works by using C2-C3 disc detector, then using a template-matching, similarity measure-based approach to identify discs. In other words, it uses traditional computer vision techniques, and does not rely on deep learning.

This approach has various shortcomings (e.g. spinalcordtoolbox/spinalcordtoolbox#2237). So, in the past, there have been two separate efforts to improve the sct_label_vertebrae function by adding deep learning-based approaches:

• 2020 (Countception - @lrouhier)
• 2021 (HourglassNet - @rezazad68)

In both cases, momentum stalled at:

• The "evaluation" stage (we were unable to validate whether the new approaches were an improvement over the existing approach)
• The "integration" stage (we were unable to take the research scrips and integrate them into the spinalcordtoolbox codebase)

2020 (Countception)

• Initial project planning board: https://github.com/spinalcordtoolbox/spinalcordtoolbox/projects/10
  • Vertebral labelling - Deep learning  spinalcordtoolbox/spinalcordtoolbox#2438
  • Vertbral labelling-DL : Retrieve Spine generic discs coordinates spinalcordtoolbox/spinalcordtoolbox#2482
  • Vertebral labelling-DL: Retrieve spine generic centerline coordinates  spinalcordtoolbox/spinalcordtoolbox#2483
  • Vertebral labelling-DL: Statistical analysis spinalcordtoolbox/spinalcordtoolbox#2484
  • Vertebral labelling-DL: Create pipeline for SCT spinalcordtoolbox/spinalcordtoolbox#2485
  • Vertebral labelling-DL : Generate 3D image  spinalcordtoolbox/spinalcordtoolbox#2486
• Development work (ivadomed): Add deep learning vertebral labeling code ivadomed/ivadomed#271
• Trained models: https://github.com/ivadomed/model_find_disc_t1, https://github.com/ivadomed/model_find_disc_t2
• Integration of approach into SCT: Add new DL-based method (Countception) for C2-C3 disc detection and template matching  spinalcordtoolbox/spinalcordtoolbox#2679
• Validation: https://github.com/sct-pipeline/vertebral-labeling-validation

2021 (HourglassNet)

• Research and development was originally done in https://github.com/rezazad68/Deep-Intervertebral-Disc-Labeling
• Integration of the approach was originally done in Intervertebral disc labeling using pose estimation ivadomed/ivadomed#787

Follow-up

In July 2021, both the Countception and Hourglass PRs/repos were in an unworkable state. So, I (@joshuacwnewton) took some time to clean up merge conflicts, as well as making sure that the validation scripts actually worked:

• Replaced original ivadomed HourglassNet PR #787 with a new PR #852 to remove conflicts and fix issues.
• Retrained the HourglassNet model to fix straight vs. curved input data issues: https://github.com/ivadomed/model_label_intervertebral-disc_t1-t2_hourglass-net
• Updated SCT PR #2679 so that it could be used for both the Countception and HourglassNet approaches
• Updated https://github.com/sct-pipeline/vertebral-labeling-validation so that it could be used to compare the Countception and HourglassNet approaches
• Began work to compare the two approaches in this comment
• However, the HourglassNet approach still needed additional tweaking before it could be fairly compared to the Countception-based approach.

Future work

Presently, it is still unclear whether either of these approaches are viable enough to be worth including in SCT.

So, the future work would involve a plan that looks something like:

1. Continue to develop the HourglassNet approach in Intervertebral disc labeling using pose estimation ivadomed/ivadomed#852.
2. Follow the instructions in https://github.com/sct-pipeline/vertebral-labeling-validation to install and run inference for both approaches.
3. Design a procedure to validate the approaches and compare them to the existing sct_label_vertebrae approach.
4. Integrate any successful approaches into SCT.