Home
See https://github.com/rg2/DeepFluoroLabeling-IPCAI2020 for complete details on obtaining the dataset and HDF5 file layouts.
The full resolution, non-preprocessed, dataset (ipcai_2020_full_res_data.h5) is used by these tools and is available here.
The CT volume, CT volume segmentation, and 3D anatomical landmarks may be extracted using the xreg-ipcai-extract-3d-data tool.
The specimen ID string is used to select which 3D data to extract.
For example, the following command will extract the 3D data for specimen 1:
xreg-ipcai-extract-3d-data ipcai_2020_full_res_data.h5 17-1882 spec_1_vol.nii.gz spec_1_vol_seg.nii.gz spec_1_3d_lands.fcsv
Examination of the extracted volumes and landmarks may conducted using 3D Slicer as shown by the screenshot below:
2D projections are extracted using the xreg-ipcai-extract-proj-data tool.
As shown by the command below, the specimen ID and projection index are used to select the 2D data to extract:
xreg-ipcai-extract-proj-data ipcai_2020_full_res_data.h5 17-1882 96 proj_96.h5
The projection may be previewed using the xreg-remap-tile-proj-data tool, with the following command creating the projection preview below:
xreg-remap-tile-proj-data proj_96.h5 proj_96_01.png -d 0.25
The xreg-ipcai-global-pelvis-regi tool will attempt to register the pelvis to projection 96 without any 2D annotations or manually adjusted initialization.
The first positional argument must be either 1 or 2 and is used to specify the global registration strategy.
See the IPCAI/IJCARS paper for the details of each approach.
A registration using method is run using the following command:
xreg-ipcai-global-pelvis-regi -v 1 spec_1_vol.nii.gz spec_1_vol_seg.nii.gz spec_1_3d_lands.fcsv proj_96.h5 proj_96_global_pelvis_method_1_regi.h5 proj_96_global_pelvis_method_1_debug.h5
The registration may be verified using the xreg-regi2d3d-replay tool as demonstrated by the following command:
xreg-regi2d3d-replay proj_96_global_pelvis_method_1_debug.h5 --proj-ds 0.5 --video-fps 10
This should create two movies (edges.mp4 and mov.mp4) similar to the below animations:
The edge overlay demonstrates that the pelvis registration is sufficient for further refinement and additional femur registration. Had the registration been insufficient, the second method could be invoked and replay movies created with the following commands:
xreg-ipcai-global-pelvis-regi -v 2 spec_1_vol.nii.gz spec_1_vol_seg.nii.gz spec_1_3d_lands.fcsv proj_96.h5 proj_96_global_pelvis_method_2_regi.h5 proj_96_global_pelvis_method_2_debug.h5
xreg-regi2d3d-replay proj_96_global_pelvis_method_2_debug.h5 --proj-ds 0.5 --video-fps 10
The replay tool should produce animations similar to those below, which also indicate a sufficient pelvis registration to continue:
The processing device (e.g. GPU) should have at minimum 6 GB of memory, in order to run the xreg-ipcai-global-pelvis-regi tool with parameters matching the IPCAI/IJCARS paper.
Next, the xreg-ipcai-global-femurs-pelvis-regi tool is used to register the femurs and refine the pelvis registration.
The previously computed pelvis registration is used as initialization.
The below command will perform this registration (using the pelvis registration from method 1 as input):
xreg-ipcai-global-femurs-pelvis-regi -v spec_1_vol.nii.gz spec_1_vol_seg.nii.gz spec_1_3d_lands.fcsv proj_96.h5 proj_96_global_pelvis_method_1_regi.h5 proj_96_global_pelvis_regi.h5 proj_96_global_left_femur_regi.h5 proj_96_global_right_femur_regi.h5 proj_96_global_debug.h5
As before, the registration may be verified using the replay tool:
xreg-regi2d3d-replay proj_96_global_debug.h5 --proj-ds 0.5 --video-fps 10
Movies similar to the following animations should be produced:
According to the edges movie, the registration appears to be sufficient for the pelvis and left femur in order to project 3D annotations into 2D. The right femur is not visible and the global registration estimate should be marked as invalid.
Now that the poses of the pelvis and left femur are known with respect to this projection, we are ready to project the 3D annotations into 2D using the xreg-ipcai-proj-3d-labels-into-2d tool.
The following command will create segmentations and preprocess the projection according to the parameters used in the IPCAI/IJCARS paper:
xreg-ipcai-proj-3d-labels-into-2d -v spec_1_vol_seg.nii.gz spec_1_3d_lands.fcsv proj_96.h5 proj_96_global_pelvis_regi.h5 proj_96_global_left_femur_regi.h5 proj_96_global_right_femur_regi.h5 proj_96_seg.h5 -d 0.125 --preproc -i proj_96_proc.h5 --pat-up
Specifically, the projection is cropped by 50 pixels about its border, log transformed, and then downsampled 8x in each dimension. The output projection files are all oriented to have the patient "up." The following commands demonstrate using the remap tool to preview the data and should yield images similar to those below:
xreg-remap-tile-proj-data proj_96_proc.h5 proj_96_proc.png -o -b 1 --overlay-land-radius 5 --overlay-land-thickness 2
xreg-remap-tile-proj-data proj_96_seg.h5 proj_96_seg.png -o -b 1 --overlay-land-radius 5 --overlay-land-thickness 2 --seg-u8
A few hundred of these annotations should allow you to train a neural network similar to this: https://github.com/rg2/DeepFluoroLabeling-IPCAI2020/tree/master/train_test_code
At this point, a neural network should have been trained using the ground truth annotations produced in the previous section. The following examples assume that the training/testing process found here was followed with the following files produced:
-
spec_1_test.h5: Estimated segmentations and landmark heatmaps for all projection data of specimen 1 -
spec_1_lands.csv: Estimated landmark locations for all projection data of specimen 1 Although segmentations and landmark annotations are available for every projection, the demonstrations in this section will stay focused on projection 96.
The following command will extract the estimated segmentation into PNG format.
python extract_seg_from_nn_h5.py spec_1_test.h5 96 proj_96_est_seg.png
Note that this segmentation has dimensions that match the cropping and downsampling applied prior to projecting the 3D labels into 2D, and when the registration pipeline used later must match these dimensions during processing, or account for them in some other way.
The segmentation may be more easily visualized by loading into 3D Slicer:
The following command will extract the estimated landmarks into FCSV format:
python extract_fcsv_from_nn_csv.py spec_1_lands.csv 1 96 0.125 proj_96_est_lands.fcsv
The extracted FCSV may be inserted into the original projection file using the xreg-add-lands-to-proj-data tool, and then remapped for verification:
xreg-add-lands-to-proj-data proj_96.h5 proj_96_est_lands.fcsv
xreg-remap-tile-proj-data proj_96.h5 proj_96_w_lands.png -o -d 0.25
As only two landmarks were detected, classic approaches which rely on finding a PnP solution may not perform well.
Three intraoperative registration strategies will be evaluated using the xreg-ipcai-intraop-pelvis-femurs-regi tool:
- Naive unconstrained intensity-only registration with the pelvis initialized at nominal AP pose,
- Attempt to solve the PnP problem to find the initial pelvis pose and then run an intensity-based registration,
- Combined intensity-based and landmark-based registration.
Further details on these methods are provided in the IPCAI/IJCARS paper.
The first positional argument of the xreg-ipcai-intraop-pelvis-femurs-regi specifies the method to use.
The following command will perform the registration using method 1:
xreg-ipcai-intraop-pelvis-femurs-regi -v 1 spec_1_vol.nii.gz spec_1_vol_seg.nii.gz spec_1_3d_lands.fcsv proj_96.h5 proj_96_est_seg.png proj_96_intraop_pelvis_regi_m1.h5 proj_96_intraop_left_femur_regi_m1.h5 proj_96_intraop_right_femur_regi_m1.h5 proj_96_intraop_left_femur_rel_pose_m1.h5 proj_96_intraop_right_femur_rel_pose_m1.h5 proj_96_intraop_debug_m1.h5
The replay tool may be used:
xreg-regi2d3d-replay --proj-ds 0.5 --video-fps 10 proj_96_intraop_debug_m1.h5
to create movies similar to the following animations:
It is clear that this approach has failed.
The following command will perform the registration using method 2:
xreg-ipcai-intraop-pelvis-femurs-regi -v 2 spec_1_vol.nii.gz spec_1_vol_seg.nii.gz spec_1_3d_lands.fcsv proj_96.h5 proj_96_est_seg.png proj_96_intraop_pelvis_regi_m2.h5 proj_96_intraop_left_femur_regi_m2.h5 proj_96_intraop_right_femur_regi_m2.h5 proj_96_intraop_left_femur_rel_pose_m2.h5 proj_96_intraop_right_femur_rel_pose_m2.h5 proj_96_intraop_debug_m2.h5
The replay tool may be used:
xreg-regi2d3d-replay --proj-ds 0.5 --video-fps 10 proj_96_intraop_debug_m2.h5
to create movies similar to the following animations:
It is clear that this approach has failed.
The following command will perform the registration using method 3:
xreg-ipcai-intraop-pelvis-femurs-regi -v 3 spec_1_vol.nii.gz spec_1_vol_seg.nii.gz spec_1_3d_lands.fcsv proj_96.h5 proj_96_est_seg.png proj_96_intraop_pelvis_regi_m3.h5 proj_96_intraop_left_femur_regi_m3.h5 proj_96_intraop_right_femur_regi_m3.h5 proj_96_intraop_left_femur_rel_pose_m3.h5 proj_96_intraop_right_femur_rel_pose_m3.h5 proj_96_intraop_debug_m3.h5
The replay tool may be used:
xreg-regi2d3d-replay --proj-ds 0.5 --video-fps 10 proj_96_intraop_debug_m3.h5 -l
to create movies similar to the following animations:
Unlike the previous two approaches, this method is able to successfully register both the pelvis and the left femur.
The proj_96_intraop_left_femur_rel_pose_m3.h5 file is a relative pose of the left femur with respect to the pelvis and may be used to create a 3D visualization of bones using the draw bones tool.
First, a surface mesh of the pelvis must be created by using the labels of each hemipelvis since the draw bones tool only supports one label per object.
The following command will create such a mesh using the xreg-create-mesh tool:
xreg-create-mesh spec_1_vol_seg.nii.gz spec_1_pelvis_mesh.h5 1 2
The draw bones tool may be used to create the interactive 3D visualization with the following command:
xreg-pao-draw-bones --bg-color 1 1 1 spec_1_vol_seg.nii.gz spec_1_3d_lands.fcsv left --no-frag --hide-contra-femur --femur-not-rel-to-frag --femur-only-xform proj_96_intraop_left_femur_rel_pose_m3.h5 --pelvis-mesh spec_1_pelvis_mesh.h5
The tool should produce views similar to the following:
