New multiclass model for segmenting the spinal cord and gray matter on 7T data

[jcohenadad]

Motivation for the feature

7T researchers cannot use SCT's segmentation models as is because features of the data are too different than 3T data used to train the model. Moreover, it would be desired to have a multi-class model (SC + GM) instead of two separate models.

Description of the feature

Researchers from the CRMBM in Marseille (Nilser Laines, Arnaud Le Troter and Virginie Callot) trained a model which would be beneficial for SCT users. The purpose of this issue is to document for the Marseille team (and everybody else wanting to chip in) the steps involved for training such a model within the ivadomed framework. The benefits is that the model will then be compatible with SCT's sct_deepseg command.

[charleygros]

We have a tutorial for "how to train a model, single class, on toy dataset", here.

I'm wondering how we could build on top of this, make it friendly to Marseille folks.
Should we make a tutorial for "how to train a model, multi class, on toy dataset"? --> would be quite similar to the existing one.
Anyway, may be good to chat about this during next ivadomed meeting so that we understand better their needs!

[arnaudletroter]

Hi Julien and Charley,

Thanks you for opening a discussion on this topic.
As you know, T2s images at 7T, compared to 3T, have some particularities in terms of contrast, resolution and artifacts.

Your CNN model included in SCT works very well for GM segmentation (sct_deepseg_gm command), but the SC delineation (sct_deepseg_sc command) suffers from the 7T contrast and seems to have room for improvement.
To try to compare the existing model optimised for 3T images, we have re-trained our full 7T dataset (containing about a thousand images, annotated by 2 raters), with your MCS framework (https://github.com/neuropoly/multiclass-segmentation), including controls and patients at different levels of resolution (from 0.17mm to 0.3mm iso) and preliminary results seems to be improved (on a small testing dataset), using a two-class GM/WM training and not two separate single-class trainings.

We would like to share this model with you and see if it overfits our data or not ?

If you are interested, as suggested by Julien, we can redo a set of training with ivadomed toolbox and following your instructions.

On our side, we played on the data augmentation part and in particular with the torchio module (https://github.com/fepegar/torchio), very interesting tools to simulate realistic spatial transformations, and MRI artefact effects (like mvt, ghosting, etc...). We could also share this experience with you.

Next step is whether we should do a multi-class training, or already follow your ivadomed tutorial adapted for one class (on the WM class in priority).
What do you think about ?

Below is an archive containing a sample of 3 slices merged into a volume, our MCS model (.ft format), and the results of the SCT and MCS 7T segmentations.
https://amubox.univ-amu.fr/s/2MgwwY8wWGS9wdH

I hope to continue the discussion with you and Nilser, and thanks for all your tools that help us a lot

Best
Arnaud

[jcohenadad]

If you are interested, as suggested by Julien, we can redo a set of training with ivadomed toolbox and following your instructions.

That would be awesome indeed! The multiclass-segmentation repository is fairly old and is no more maintained (I just archived it and added a disclaimer to use ivadomed instead). Some of the benefits of using ivadomed include:

• bleeding edge architectures are readily accessible,
• BIDS-compatible dataset can be input,
• code is actively maintained and cross-platform/OS tested with continuous integration,
• ivadomed is imported in SCT (and hence any model trained under ivadomed can readily be used inside SCT).

On our side, we played on the data augmentation part and in particular with the torchio module (https://github.com/fepegar/torchio), very interesting tools to simulate realistic spatial transformations, and MRI artefact effects (like mvt, ghosting, etc...). We could also share this experience with you.

This is a great library indeed! We are also using it in some projects (eg: MS lesion segmentation).

Next step is whether we should do a multi-class training, or already follow your ivadomed tutorial adapted for one class (on the WM class in priority). What do you think about ?

Great question. The multi-class approach seems relevant. The closest project I can think of is the WM/GM multiclass segmentation from ex vivo high res data. The repository includes a config file. One cool feature of this project is that it uses SoftSeg, which accounts for partial volume effect on the estimated segmentation.

However, it seems that your multiclass (GM/SC) approach includes voxels that are contained within two output classes (GM and SC). This is the reason we went for a multiclass WM/GM model (ie: the sum of all classes equals one, for each voxel in the image).

[Nilser3]

Hello Julien et Charley,
Thanks you for opening a discussion on this topic.

Today I did a Single_Class training for SC and GM (attached result. GT: Green, DL_Seg: Red),
but I would like you to help me adjust some parameters (Attached config_file.json and version_info.log).

1. How to specify the Data Training and Data Validation? because I understand that ivadomed makes a relation (0.6 / 0.2). but it seems to me in a random way / I would like to define the Train and Valid subjects.

2. Test, I have in the pred_masks/ folder two files: "sub-31_T2star_pred_painted" and "sub-31_T2star_pred" and I would like to know the difference between these images (because they look the same to me).

3. Can we adjust the resolution of the input for training ?, as we could do in MCS.

(I'm starting to use IVADOMED, so sorry for some such basic errors).

Best

Nilser

version_info.log

config_file.txt

[charleygros]

Hi @Nilser3,

It was great to meet you yesterday and thank you for following-up on this.

• How to specify the Data Training and Data Validation? because I understand that ivadomed makes a relation (0.6 / 0.2). but it seems to me in a random way / I would like to define the Train and Valid subjects.

Yes, you are correct: this is a random split, defined by a random_seed.
If you would like to use a specific split of subjects, you could use this fname_split parameter, which points towards a file containing your Train/Valid/Test split. Please let us know if you need further help with this.

• Test, I have in the pred_masks/ folder two files: "sub-31_T2star_pred_painted" and "sub-31_T2star_pred" and I would like to know the difference between these images (because they look the same to me).

Yes. For your application you can ignore _painted file --> it assign different values to connected objects depending on TruePositive / FP etc.
So yes, you should use _pred.nii.gz file I reckon.

• Can we adjust the resolution of the input for training ?, as we could do in MCS.

Absolutely, you could add a transformation called Resample as preprocessing so that all input data will have the same resolution. See documentation here and example here. Again, please let us know if you need further help with this :-)

[charleygros]

Looking at the config file you shared with us @Nilser3, the issue may be coming from the transformations (maybe). In particular:

        "Resample": {
            "wspace": 0.75,
            "hspace": 0.75,
            "dspace": 1
        },

--> This will resample your input data to 0.75x0.75x1 mm3 --> which is a poor resolution compared to your initial data. I would recommend you to change this with a finer resolution, eg the finer resolution encountered in your dataset.

        "CenterCrop": {
            "size": [
                128,
                128
            ]
        },

Based on the resolution you are choosing (see above), this may change: Is the cord fully included (ie not partially cropped) when you do a center-wise cropping of 128x128 vox2? This will need to be checked.

As a general advice for the optimisation of the preprocessing params as well as the DataAugmentation params, I encourage you to use this script which gives you a easy was to visualize Data Augmentation transformations.

Hope it helps :-)

[Nilser3]

Hi @charleygros

Thanks for your prompt responses,

I had much better results manipulating the resolution and was also able to do a Multiclass Segmentation. (Attached image)

Now I am working on the .joblib files to cuztomize the Train / Valid / Test. (If they had any Tuto about it, it would help me)

I still have some questions.

1. How can I disable the Transformations / Data augmentation?

Because me and @arnaudletroter think that if we deactivate this package, we can load into "Train Datasets" the True Train datasets + Externally Augmented Train datasets.

2. Sorry for asking you a more theoretical question, but is it advisable apply Data Augmentation in Validation Data?

Finally, I am quite happy to work with such a comfortable and intuitive framework! Ivadomed , Plus their scripts are great.

Best

Nilser

[charleygros]

Thank you for your feedback @Nilser3 and your kind words! Appreciated

Also, I'm glad you got better results.

I can't help you with the joblib until Tuesday, but maybe @mariehbourget could?

No Data Augmentation: you just need to remove the transformations from the config file (eg RandomAffine), while keeping the Resampling, Crop, NormaliseInstance and NumoyToTensor.

DA on validation: that's an interesting question. It is common not to apply DA on Validation set in order to have a realistic independent dataset as reference. Whereas DA on the training set is used to avoid overfitting. I would suggest not to DA the validation set.

[mariehbourget]

Hi @Nilser3,
Thank you for your feedback!

Regarding the .joblib file for the split, I’m happy to help.
Here is an example of how to generate the file using the joblib library:

import joblib

train_lst = ["sub-31_T2star.nii.gz", "sub-32_T2star.nii.gz", "sub-33_T2star.nii.gz", "sub-34_T2star.nii.gz"] # Your list of filenames to include in the training set
valid_lst = ["sub-35_T2star.nii.gz", "sub-36_T2star.nii.gz"] # Your list of filenames to include in the validation set
test_lst = ["sub-37_T2star.nii.gz", "sub-38_T2star.nii.gz"] # Your list of filenames to include in the test set

split_path = "path_to_split_joblib/split_datasets.joblib" # Your output path for the joblib file

split_dct = {'train': train_lst, 'valid': valid_lst, 'test': test_lst}
joblib.dump(split_dct, split_path)

So you will have to specify two things:

1. The filenames of the subjects you want to include in each of the train, valid and test list.
2. An output path for the .joblib file.

Once the .joblib file is generated, add its path to the fname_split parameter in the training config file.

Please let us know if you have any questions or concerns about this procedure.

Best,
Marie-Hélène

[Nilser3]

Thanks for your reply @charleygros
Thanks for your help @mariehbourget. I was able to launch my trainings, from .joblibs

I did a Multiclass and Singleclass(x2) training , with the same parameters, (attached image) and although the results are not the same, they are already much better.

Red: Singleclass(x2)
Green: Multiclass

Thanks for your help, I will continue with the optimization.

[jcohenadad]

@arnaudletroter @Nilser3 @charleygros I have additional thoughts about the multiclass WM/GM vs. single class SC and single GM.

If we provide users with a WM/GM multiclass model, users would need to sum both masks to obtain the SC segmentation. However, this scenario makes it more difficult to manually correct the GM mask in case the segmentation is not perfect (and as we know, 100% robust method don't exist). For example, if users manually adjust the GM prediction, then if they want to obtain the SC segmentation they cannot simply sum the WM and GM masks because there will be overlap (or missing voxels) at the interface between the WM and the GM mask. So this requires to also manually adjust the WM prediction.

Whereas if users segment the SC and the GM mask (as is currently the recommended workflow in SCT), the WM mask would be obtained by subtracting the GM from the SC mask. So if they manually adjust the GM mask (or the SC mask), they can still obtain the WM mask using a simple subtraction, without worrying about missing/added voxels at the interface between the two tissues.

[Nilser3]

Exactly @jcohenadad

When we initially started working with @arnaudletroter in the Multiclass_Segmentation network, we had to train WM and GM, and finally add them together to obtain the SC.

But now at IVADOMED I have been able to train SC and GM on the same model, and I agree with you, it would allow us to have a more precise WM.

Now I am analyzing statistics, to see if a Multiclass SC/GM model is =, better or worse than a Singleclass_Model(x2), trained under the same parameters and datasets of course.

[Nilser3]

Hello @charleygros @jcohenadad !

I would like to share with you the results of the Multiclass and Singleclass comparison, trained with the same parameters. Of course!

Also, I have a question. When we apply Data Augmentation, is a composition made of RandomAffine and ElasticTransform, which are applied to an image, to obtain a augmented image? In other words, for every X images in Train we will have X augmented images?.

Thanks a lot!
Best

Nilser

[jcohenadad]

Thank you for sharing your results in #841 (comment) @Nilser3. I will let @charleygros answer your question, but I have a few comments about this comparison:

• Does the multiclass and singleclass use the exact same participant split (train/eval/test)?
• Does it use the same initialization weights? I assume it does not because the architecture is probably different (2class vs. 1class), which brings me to another question: in order to make sure that this comparison is robust, it would be interesting to perform cross-validation (eg: re-train with different seeds and participant split and then average results across each training experiment). That way, we will be able to assess wether any possible difference between multiclass and singleclass is indeed real and not cause by the randomization.

[Nilser3]

Thanks to @mariehbourget I have been able to define the (train / valid / test), so for both Train I have used the same .joblib file.
We make this definition based on the heterogeneity of the data.

However, the idea of cross validation is very interesting, I will apply it right away.

[charleygros]

* Does it use the same initialization weights? I assume it does not because the architecture is probably different (2class vs. 1class), which brings me to another question: in order to make sure that this comparison is robust, it would be interesting to perform cross-validation (eg: re-train with different seeds and participant split and then average results across each training experiment). That way, we will be able to assess wether any possible difference between multiclass and singleclass is indeed real and not cause by the randomization.

I second this idea of multiple trainings with different train / valid / test subsets. Would be valuable analysis

[charleygros]

Also, I have a question. When we apply Data Augmentation, is a composition made of RandomAffine and ElasticTransform, which are applied to an image, to obtain a augmented image? In other words, for every X images in Train we will have X augmented images?.

For each epoch, the X training samples are augmented with different random parameters. In pseudo code it would be something like:

for i_epoch in range(0, max_epoch -1):
    list_train_sample_augmented = []
    for train_sample in list_train_sample:
        # Random affine transformation
        random_translation_param, random_rotation_param = get_random_affine_param()
        train_sample_affine = apply_affine(train_sample,  random_translation_param, random_rotation_param)
        # Random elastic transformation
        random_elastic_param = get_random_elastic_param()
        train_sample_affine_elastic = apply_elastic(train_sample_affine,  random_elastic_param)
        # Add the augmented sample to the list
        list_train_sample_augmented.append(train_sample_affine_elastic)

    # Train model with DA train data
    model.train(list_train_sample_augmented)

Hope it helps!

[Nilser3]

Hi everyone,

Thanks to the very valuable contributions of @jcohenadad , @charleygros and @mariehbourget , we were able to obtain a 7T multiclass segmentation model (SC / GM) that has very satisfactory results. Now we would like to share this model with the community, and if possible make it compatible with SCT's sct_deepseeg command.

Thank you !!

[jcohenadad]

Now we would like to share this model with the community, and if possible make it compatible with SCT's sct_deepseeg command.

amazing! i suggest to start by creating a repos under ivadomed org where the model will be located (as done for other models), and add a link to this model in SCT's deepseg function. @kousu @joshuacwnewton @taowa could you please help? thanks!!

[joshuacwnewton]

i suggest to start by creating a repos under ivadomed org where the model will be located (as done for other models),

I don't think @Nilser3 has the permissions to create a new repository under the @ivadomed org. But, I would be happy to create the repo on their behalf, and then grant them individual access to that repo. 🙂

add a link to this model in SCT's deepseg function. @kousu @joshuacwnewton @taowa could you please help? thanks!!

This is quite easy to do! Here is an example of a recent pull request that added a model to sct_deepseg: spinalcordtoolbox/spinalcordtoolbox#3224

So, I would recommend:

1. Forking the spinalcordtoolbox repository, then cloning it locally
2. Creating a new local branch for your changes
3. Adding the new model to spinalcordtoolbox/deepseg/models.py (similar to how it was done in the PR linked above)
4. Pushing the changes to a remote branch on your fork, then creating a PR to the main SCT repo with your changes

Then I would be happy to test out and review your changes. :)

[joshuacwnewton]

I've created the https://github.com/ivadomed/model_seg_gm-wm_t2star_7T_unet3d-multiclass repository, and sent out an invite to @Nilser3 that would give Maintain access to that repository. 🙂

[Nilser3]

Hi everyone!!

Thanks @joshuacwnewton I have been able to upload our model in the https://github.com/ivadomed/model_seg_gm-wm_t2star_7T_unet3d-multiclass repository and creates a PR and I modified spinalcordtoolbox/deepseg/models.py that as fork (I'm really sorry because I am new to Git).

However, I still can't get my .zip file from my repository to be a direct download for the models installation.

Thanks very much for your help

[joshuacwnewton]

Thanks @joshuacwnewton I have been able to upload our model in the https://github.com/ivadomed/model_seg_gm-wm_t2star_7T_unet3d-multiclass repository and creates a PR and I modified spinalcordtoolbox/deepseg/models.py that as fork

Thank you so much for the contribution! This is an excellent start. 😄

(I'm really sorry because I am new to Git). However, I still can't get my .zip file from my repository to be a direct download for the models installation.

Git can be tricky to work with, even for us, so no worries whatsoever! We're grateful to have your contribution, and happy to help wherever we can, so feel free to ask questions. 😄

To get a zip file as a direct download, you just need to:

1. First remove the current .zip file from the repository. (It is actually not needed!)
2. Create a "Release" from GitHub's release page, titled e.g. r20210929. Once the release is created, GitHub will automatically package everything inside the repository into a .zip for you. 🙂

Here is an example release -- you can see the "Source code" zip downloads attached by GitHub under "Assets".

[Nilser3]

Thank you so much @joshuacwnewton

our release is ok in https://github.com/ivadomed/model_seg_gm-wm_t2star_7T_unet3d-multiclass/archive/refs/tags/v1.0.zip

now I can't rename this repo https://github.com/ivadomed/model_seg_gm-wm_t2star_7T_unet3d-multiclass

on the recommendation of @jcohenadad
spinalcordtoolbox/spinalcordtoolbox#3527 (comment)

could you (please) change it for /model_seg_gm-sc_t2star_7t_unet2d-multiclass please

Thanks a lot!

[joshuacwnewton]

could you (please) change it for /model_seg_gm-sc_t2star_7t_unet2d-multiclass please

My apologies! I thought "Maintain" privileges would let you do that. Got it -- I've renamed the repo!

FYI -- I also renamed the release to match the date-based convention used for other models. (r20210930)

The new link to the .zip is: https://github.com/ivadomed/model_seg_gm-wm_t2star_7t_unet3d-multiclass/archive/r20210930.zip