Is ivadomed --segment --no-patch also applicable to 3D?

[jcohenadad]

From the doc:

The --no-patch specifies the description for 2D kernels, but is that also applicable to 3D?

• If not: short term: we should make it explicit. Long term: we should implement this feature for 3D kernels.
• If so: we should update the doc.

[jcohenadad]

Note: I am designing a segment_image CLI (#1254), and it looks like the --no-patch is not working for 3D data, ie: predictions are only created at the center of the image. However the output message is confusing to me:

2023-01-03 15:56:19.843 | WARNING  | ivadomed.inference:segment_volume:428 - The 'no_patch' option is provided but the model has no 'length_2D' and 'stride_2D' parameters in its configuration file 'model_seg_lesion_mp2rage.json'. 2D patching is ignored, the segmentation 'is done on the entire image without patches.

Terminal output

julien-macbook:~/temp/model_seg_basel $ python ~/code/ivadomed/ivadomed/scripts/segment_image.py 
2023-01-03 15:56:19.689 | INFO     | ivadomed.utils:define_device:160 - Cuda is not available.
2023-01-03 15:56:19.689 | INFO     | ivadomed.utils:define_device:161 - Working on cpu.
2023-01-03 15:56:19.843 | WARNING  | ivadomed.inference:segment_volume:409 - fname_roi has not been specified, then the entire volume is processed.
2023-01-03 15:56:19.843 | WARNING  | ivadomed.inference:segment_volume:428 - The 'no_patch' option is provided but the model has no 'length_2D' and 'stride_2D' parameters in its configuration file 'model_seg_lesion_mp2rage.json'. 2D patching is ignored, the segmentation 'is done on the entire image without patches.
2023-01-03 15:56:22.497 | INFO     | ivadomed.inference:segment_volume:464 - Loaded 49 axial volumes of shape [32, 32, 16].
2023-01-03 15:56:22.523 | DEBUG    | ivadomed.inference:get_preds:87 - Likely ONNX model detected at: /Users/julien/temp/rosenberg_nvme/model_seg_lesion_mp2rage_20230102_145722/model_seg_lesion_mp2rage/model_seg_lesion_mp2rage.onnx
2023-01-03 15:56:22.524 | DEBUG    | ivadomed.inference:get_preds:88 - Conduct ONNX model inference... 
2023-01-03 15:56:24.303 | DEBUG    | ivadomed.inference:get_preds:91 - Sending predictions to CPU
2023-01-03 15:56:24.444 | DEBUG    | ivadomed.inference:get_preds:87 - Likely ONNX model detected at: /Users/julien/temp/rosenberg_nvme/model_seg_lesion_mp2rage_20230102_145722/model_seg_lesion_mp2rage/model_seg_lesion_mp2rage.onnx
2023-01-03 15:56:24.445 | DEBUG    | ivadomed.inference:get_preds:88 - Conduct ONNX model inference... 
2023-01-03 15:56:24.693 | DEBUG    | ivadomed.inference:get_preds:91 - Sending predictions to CPU
2023-01-03 15:56:24.705 | DEBUG    | ivadomed.inference:get_preds:87 - Likely ONNX model detected at: /Users/julien/temp/rosenberg_nvme/model_seg_lesion_mp2rage_20230102_145722/model_seg_lesion_mp2rage/model_seg_lesion_mp2rage.onnx
2023-01-03 15:56:24.705 | DEBUG    | ivadomed.inference:get_preds:88 - Conduct ONNX model inference... 
2023-01-03 15:56:24.968 | DEBUG    | ivadomed.inference:get_preds:91 - Sending predictions to CPU
2023-01-03 15:56:24.971 | DEBUG    | ivadomed.inference:get_preds:87 - Likely ONNX model detected at: /Users/julien/temp/rosenberg_nvme/model_seg_lesion_mp2rage_20230102_145722/model_seg_lesion_mp2rage/model_seg_lesion_mp2rage.onnx
2023-01-03 15:56:24.971 | DEBUG    | ivadomed.inference:get_preds:88 - Conduct ONNX model inference... 
2023-01-03 15:56:25.037 | DEBUG    | ivadomed.inference:get_preds:91 - Sending predictions to CPU

EDIT 20230103_160513: Actually, the problem of predictions appearing at the center might be coming from the config file, which specifies a centercrop:

    "transformation": {
        "Resample": {"hspace": 1, "wspace": 1, "dspace": 1},
        "CenterCrop": {"size": [32, 32, 80]},
        "NormalizeInstance": {"applied_to": ["im"]}
    }

So, in this case, how can we best apply a model without the CenterCrop restriction? Should we modify the JSON file and remove the CenterCrop field?

EDIT 20230103_162901: Yup, that did the trick. Even though the inference is veeeeery slow (motivating #1259):

julien-macbook:~/temp/model_seg_basel $ python ~/code/ivadomed/ivadomed/scripts/segment_image.py         
2023-01-03 16:10:24.609 | INFO     | ivadomed.utils:define_device:160 - Cuda is not available.
2023-01-03 16:10:24.609 | INFO     | ivadomed.utils:define_device:161 - Working on cpu.
2023-01-03 16:10:24.787 | WARNING  | ivadomed.inference:segment_volume:409 - fname_roi has not been specified, then the entire volume is processed.
2023-01-03 16:10:24.787 | WARNING  | ivadomed.inference:segment_volume:428 - The 'no_patch' option is provided but the model has no 'length_2D' and 'stride_2D' parameters in its configuration file 'model_seg_lesion_mp2rage.json'. 2D patching is ignored, the segmentation 'is done on the entire image without patches.
2023-01-03 16:10:32.085 | INFO     | ivadomed.inference:segment_volume:464 - Loaded 7303505 axial volumes of shape [32, 32, 16].  ---> 😱 

[jcohenadad]

But wait... the number of patches is confusing. Here, it loads 269335 axial volumes of 32x32x16 size:

(ivadomed_venv) julien-macbook:~/temp/model_seg_basel $ python ~/code/ivadomed/ivadomed/scripts/segment_image.py
2023-01-03 17:44:35.499 | INFO     | ivadomed.utils:define_device:160 - Cuda is not available.
2023-01-03 17:44:35.499 | INFO     | ivadomed.utils:define_device:161 - Working on cpu.
2023-01-03 17:44:35.641 | WARNING  | ivadomed.inference:segment_volume:409 - fname_roi has not been specified, then the entire volume is processed.
2023-01-03 17:44:35.641 | WARNING  | ivadomed.inference:segment_volume:428 - The 'no_patch' option is provided but the model has no 'length_2D' and 'stride_2D' parameters in its configuration file 'model_seg_lesion_mp2rage.json'. 2D patching is ignored, the segmentation 'is done on the entire image without patches.
2023-01-03 17:44:36.012 | INFO     | ivadomed.inference:segment_volume:464 - Loaded 269335 axial volumes of shape [32, 32, 16].

However, the input image is 90x98x86:

julien-macbook:~/temp/model_seg_basel $ sct_image -i /Users/julien/temp/model_seg_basel/data_processed_lesionseg_small/sub-P025/anat/sub-P025_UNIT1.nii.gz  -header

--
Spinal Cord Toolbox (git-master-3cfab1dbff6a1467ca2fad03d070eb2762dfa5e1)

sct_image -i /Users/julien/temp/model_seg_basel/data_processed_lesionseg_small/sub-P025/anat/sub-P025_UNIT1.nii.gz -header
--

sizeof_hdr	348
data_type	UINT16
dim		[3, 90, 98, 86, 1, 1, 1, 1]

So, how is the sliding performed? And with what stride?

[mariehbourget]

Hi @jcohenadad!
A lot of good questions, I'll try to address all of them and we'll see if any fix are needed afterwards.

The --no-patch specifies the description for 2D kernels, but is that also applicable to 3D?

• If not: short term: we should make it explicit. Long term: we should implement this feature for 3D kernels.
• If so: we should update the doc.

As specified, it is only applied for 2D kernels. IIUC, you would like us to add a note tot he doc saying "not applicable in 3D" to make it more explicit?
As for implementing it in 3D, the initial test showed some errors (ref: #1049 (comment)) and I don't think anyone has worked on this since.

Note: I am designing a segment_image CLI (#1254), and it looks like the --no-patch is not working for 3D data, ie: predictions are only created at the center of the image. However the output message is confusing to me:

2023-01-03 15:56:19.843 | WARNING  | ivadomed.inference:segment_volume:428 - The 'no_patch' option is provided but the model has no 'length_2D' and 'stride_2D' parameters in its configuration file 'model_seg_lesion_mp2rage.json'. 2D patching is ignored, the segmentation 'is done on the entire image without patches.

Good catch! This message is very confusing indeed when using a 3D model, it should only appears for 2D models. We should have a different error message when the model is 3D to make it clearer that the --no-patch option is 2D only.

EDIT 20230103_160513: Actually, the problem of predictions appearing at the center might be coming from the config file, which specifies a centercrop:

    "transformation": {
        "Resample": {"hspace": 1, "wspace": 1, "dspace": 1},
        "CenterCrop": {"size": [32, 32, 80]},
        "NormalizeInstance": {"applied_to": ["im"]}
    }

So, in this case, how can we best apply a model without the CenterCrop restriction? Should we modify the JSON file and remove the CenterCrop field?

Yes, AFAIK, the CenterCrop from the model config file is applied to the segment command and the only way to bypass it is to remove it from the config.

But wait... the number of patches is confusing. Here, it loads 269335 axial volumes of 32x32x16 size:

(ivadomed_venv) julien-macbook:~/temp/model_seg_basel $ python ~/code/ivadomed/ivadomed/scripts/segment_image.py
2023-01-03 17:44:35.499 | INFO     | ivadomed.utils:define_device:160 - Cuda is not available.
2023-01-03 17:44:35.499 | INFO     | ivadomed.utils:define_device:161 - Working on cpu.
2023-01-03 17:44:35.641 | WARNING  | ivadomed.inference:segment_volume:409 - fname_roi has not been specified, then the entire volume is processed.
2023-01-03 17:44:35.641 | WARNING  | ivadomed.inference:segment_volume:428 - The 'no_patch' option is provided but the model has no 'length_2D' and 'stride_2D' parameters in its configuration file 'model_seg_lesion_mp2rage.json'. 2D patching is ignored, the segmentation 'is done on the entire image without patches.
2023-01-03 17:44:36.012 | INFO     | ivadomed.inference:segment_volume:464 - Loaded 269335 axial volumes of shape [32, 32, 16].

As mentioned before, "2d patches" are not applied in 3D. However, 3D training and segmentation is always done by subvolumes ("3d patches"). So here you have 269335 subvolumes to segment to complete the entire volume prediction.

I'm wondering about the wording though... I'm not too familiar with 3D but is the expression "axial volumes" makes sense here? Should the orientation be specified or not?

However, the input image is 90x98x86:

julien-macbook:~/temp/model_seg_basel $ sct_image -i /Users/julien/temp/model_seg_basel/data_processed_lesionseg_small/sub-P025/anat/sub-P025_UNIT1.nii.gz  -header

--
Spinal Cord Toolbox (git-master-3cfab1dbff6a1467ca2fad03d070eb2762dfa5e1)

sct_image -i /Users/julien/temp/model_seg_basel/data_processed_lesionseg_small/sub-P025/anat/sub-P025_UNIT1.nii.gz -header
--

sizeof_hdr	348
data_type	UINT16
dim		[3, 90, 98, 86, 1, 1, 1, 1]

So, how is the sliding performed? And with what stride?

Yeah I agree, the number 269335 seems oddly high.
The striding is suppose to come from the config file as well, param stride_3D in Modified3DUNet.
What is the stride in your config?

[jcohenadad]

The striding is suppose to come from the config file as well, param stride_3D in Modified3DUNet.
What is the stride in your config?

ah! excellent point, thank you. The stride is 1x1x1:

        "stride_3D": [
            1,
            1,
            1
        ],

I used it because I got confused about the definition of a stride (#1248). I understand now that this is an inappropriate stride size 😉. In fact, I just found out by going through the debugger in inference.py and realizing that the z index was only incremented by one (which now makes sense):

As specified, it is only applied for 2D kernels. IIUC, you would like us to add a note tot he doc saying "not applicable in 3D" to make it more explicit?

Yes, as a short term solution. However in the long term we definitely want the no-patch option for 3D (see comment below).

As for implementing it in 3D, the initial test showed some errors (ref: #1049 (comment)) and I don't think anyone has worked on this since.

Hum, OK. Let's level up the priority on this then, given that many of us are playing with 3D models.

[jcohenadad]

I'm wondering about the wording though... I'm not too familiar with 3D but is the expression "axial volumes" makes sense here? Should the orientation be specified or not?

Thank you for bringing that up. It does make sense to talk about orientation for 3D data (also for 2D), but I'm not sure if specifying the orientation is necessary. I'm also not sure if implicating the orientation that much in the code makes sense. I feel that the orientation stuff should be treated at a higher level. For example, here, the code assumes that 'axial' refers to the last dimension of a 3D array:

ivadomed/ivadomed/loader/bids3d_dataset.py

Line 24 in c61ab28

slice_axis (int): Indicates the axis used to extract slices: "axial": 2, "sagittal": 0, "coronal": 1.

However, there is also the possibility to specify an orientation in the JSON file:

ivadomed/ivadomed/config/config.json

Line 36 in 199eba6

"slice_axis": "axial",

So, does this mean that a real axial slice depends on the value of "slice_axis"? I think this should deserve its own issue discussion (👉 #1261)