This project aim to assess the benifit of multi-task learning in classification of renal lesion based on MRI images and clinical data. There is three task on which our models will be trained: the classification of the malignancy, the subtype and the grade of renal tumour. The classification of the subtype and the grade can only be done on malignant tumour.
Please take note that the dataset has not been push on github since it include private data and it is really heavy. For any information about the data that has been used, please contact Alexandre Ayotte (Alexandre.Ayotte2@USherbrooke.ca) or Martin Vallières (Martin.Vallieres@USherbrooke.ca)
The present package is written in Python 3.8. In order to run it in full capacity, the user should have a Nvidia GPU with CUDA 11.1 installed. Morever, the following package are required to execute our code.
- antspy
- h5py
- matplotlib
- monai
- nibabel
- pandas
- pytorch
- scikit-learn
- torchsummary
- tqdm
- comet_ml
- tensorboard
- Normalization
- MRIimage Class
- Patient Class
- Transfer header from images to ROI
- Normalize and crop 3D images and ROI
- Normalize and crop 2D images and ROI
- Classification
- RenalDataset
- Abstract Trainer
- NeuralNet
- Single-Task Learning
- ResNet2D
- ResNet3D
- SingleTaskTrainer
- Add tensorboard
- Prepare Multi-Task Learning
- MultiTaskTrainer
- MultiLevelResNet3D
- SharedNet
- Uncertainty loss
- Conditional probability
- Cumulate gradient
- Add hparam tracking with comet.ml
- Add Hybrid ResNet
- Add MTAN
- Compute Radiomics
- Experiment Phase 1
- Block type
- Hybrid ResNet
- Grouped Convolution
- Merged Mask
- Saving criterion
- Experiment Phase 2
- MTL Loss (H-S)
- Split layer (H-S)
- Pretraining (S-S)
- Sharing Modules Position (S-S)
- MTAN depth (MTAN)
- MTAN width (MTAN)
- Attention Module (MTAN)
- Prepare Radiomics Experiments
- Compute radiomics
- Adapt MultiTaskTrainer
- Add Learn-To-Branch (LTB) Model
- Experiment Phase 3
- Auxiliary tasks
- Base Test (LTB)
- Multiple Layer Type (LTB)
- Experiment Phase 4
- Radiomics in self-supervising
- With one set of radiomics
- All sets of radiomics
- Radiomics in MultiTask Learning
- One task + ine radiomics set
- All tasks + one radiomics set
- One task + all radiomics sets
- All tasks + all radiomics sets
- Radiomics in self-supervising
- Write paper
- Bonus
- Add Fully Adaptive Features Sharing (FAFS) Model
- Test FAFS
.
├── Classification
│ ├── DataManager # Dataset and data visualisation related files.
│ │ ├── BrainDataset.py # BrainDataset class (inherit from HDF5Dataset).
│ │ ├── DataAugView2D.py # Visualisation of the data augmentation on 2D images.
│ │ ├── DataAugView3D.py # Visualisation of the data augmentation on 3D images.
│ │ ├── DatasetBuilder.py # Build and split the training, validation and testing set.
│ │ ├── HDF5Dataset.py # HDF5Dataset class.
│ │ ├── RenalDataset.py # RenalDataset class (inherit from HDF5Dataset).
│ │ ├── Sampler.py # Sampler class. Used to create split for BrainDataset.
│ │ └── reject_list.txt # List of patient id to reject in this project (RenalDataset).
│ │
│ ├── Model # Neural network model related files.
│ │ ├── Block.py # Commun ResNet, PreResNet and Attention block.
│ │ ├── CapsNet2D.py # CapsNet2D class.
│ │ ├── CapsuleBlock.py # CapsNet related block and function.
│ │ ├── HardSharedResNet.py # HardSharing ResNet3D class.
│ │ ├── LTBResNet.py # Learn-To-Branch ResNet3D class.
│ │ ├── Module.py # Commun module used to build NeuralNetwork.
│ │ ├── MTAN.py # The MTAN class (MultiTask Attention Network).
│ │ ├── NeuralNet.py # Abstract class of all neural network classes (except ResNet2D).
│ │ ├── ResNet.py # ResNet3D class.
│ │ ├── ResNet_2D.py # ResNet2D model class.
│ │ └── SharedNet.py # SharedNet model class.
│ │
│ ├── Trainer # Neural Network training related files.
│ │ ├── MultiTaskTrainer.py # Training class for Multi-Task Learning.
│ │ ├── SingleTaskTrainer.py # Training class for Single-Task Learning.
│ │ ├── Trainer.py # Abstract class of all trainer classes.
│ │ └── Utils.py # Utils function for the trainers.
│ │
│ ├── ArgParser.py # ArgParser used by all Main files.
│ │── comet_api_key.txt # Contain the needed api key to use comet.ml
│ ├── Constant.py # Several class of constant and enum.
│ ├── Main.py # Main script for all experimentation.
│ ├── ModelCreation.py # Script that create the different neural network.
│ ├── TrainerCreation.py # Script that intialize the trainer before training.
│ └── Utils.py # Utils function for the main files.
│
├── Normalization # Normalization and preprocessing of MRI images.
│ ├── MRI_image.py # The MRIimage class.
│ ├── Normalize2D.py # Used to create a h5py dataset of 2D normalized images.
│ ├── Normalize.py # Used to normalize the nifti images of a list of patient.
│ ├── Patient.py # The Patient class.
│ ├── RoiStats.py # Script used to compute statistics about the ROI.
│ ├── TransferHeader.py # Script used to transfer metadata between nifti files.
│ ├── Transfer_in_hdf5.py # Transfer nifti files and clinical data into hdf5.
│ ├── Utils.py # Commun and useful function in normalization process.
│ └── VisualizeNormalization.py # Script used to visualize the normalization process.
│
└── README.md
- Alexandre Ayotte (Ms student in medical imaging, University of Sherbrooke, GRIIS)
- Martin Vallieres (Pr, Computer science departement, University of Sherbrooke, GRIIS)