MIDL2020

This is the implementation of the Classification of Epithelial Ovarian Carcinoma Whole-Slide Pathology Images Using Deep Transfer Learning. Citation information:

@misc{wang2020classification,
    title={Classification of Epithelial Ovarian Carcinoma Whole-Slide Pathology Images Using Deep Transfer Learning},
    author={Yiping Wang and David Farnell and Hossein Farahani and Mitchell Nursey and Basile Tessier-Cloutier and Steven J. M. Jones and David G. Huntsman and C. Blake Gilks and Ali Bashashati},
    year={2020},
    eprint={2005.10957},
    archivePrefix={arXiv},
    primaryClass={eess.IV}
}

Presentation and Slides

Reviews

Our work is inspired by ProGAN and fast.ai.

Prerequisites

• Linux or macOS
• Python 3.5.2
• PyTorch 1.3.0+cu92
• scikit-learn 0.22.1
• NVIDIA GPU + CUDA CuDNN

Get Started

Installation

• Install the required packages

  • pip install torch==1.3.0+cu92 torchvision==0.4.1+cu92 -f https://download.pytorch.org/whl/torch_stable.html
  • pip install -r requirements.txt

• Clone this repo

mkdir wsi_classification
cd wsi_classification
git clone https://github.com/AIMLab-UBC/MIDL2020
cd MIDL2020

Repo Structure

• patch_level.py is the patch-level classifier entry. Inside this file, function train initialize models, create data loader, optimize model weights, log training information, etc. evaluate simply loads the trained model and apply the model on validation or testing sets.

• slide_level.py is the slide-level classifier entry. Inside this file, function main initialize models, create a data loader, optimize model weights, and compute the 6-fold cross-validation.

• config.py is the program that reads arguments from the user. Therefore, you can custom any hyperparameters or settings through config.py

• models is the sub-module that contains implementation involves models.

  • Inside models sub-module, models/base_model.py is the template class for models/models.py. It defines various expected behaviours of a model, such as forward, optimize_weights, load_state, save_state, etc. Any models should inherit BaseModel.

  • Inside models sub-module, models/models.py is the models/networks.py interface. It initializes models/networks.py and optimizes the weights of models/networks.py. Different models in models/models.py are summarized in the following table.

    Models	Usage
    CountBasedFusionModel	Slide-Level model wrapper. It assumes an input of N*C matrix, where N represents the number of slides, C represents the number of classes.
    DeepModel	Patch-Level model wrapper. It is a simple wrapper for out-of-box deep learning models. It takes one patch for each forward pass. It follows the standard deep learning training protocol.

  • Inside models sub-module, models/networks.py is the implementation of various networks. It simply defines network architecture and forward functions. Keep it as simple as possible.

• data is the sub-module that contains data loader, preprocess, and post-process functions.

  • Inside the data sub-module, data/base_dataset.py is the template class for other data loaders. It simply defines data loader behaviour, and assign config.py settings into the current data loader. Moreover, it also modifies the patch ids if requires, such as change multi-scale model scales, etc.

  • Inside the data sub-module, data/patch_dataset.py is the main patch images or patch features data loader.

    Dataset	Usage
    SubtypePatchDataset	It loads patch images from H5 files and applies preprocess steps.

  • Inside the data sub-module, data/create_patient_groups.py is the helper script to split the dataset by patients.

• utils is the sub-module that contains simple but useful function snippet.

  • Inside utils sub-module, utils/utils.py contains a wide range of useful small functions. Contributors should also add useful functions in here, and add docstring. Moreover, functions in here should be self-explained and as general as possible.

  • Inside utils sub-module, utils/subtype_enum.py defines enum for classes.

Patch extraction

First of all, the enum in utils/subtype_enum.py should be defined.

Afterwards, we extract the 1024 * 1024 patches and then downsampled to 512 * 512 and 256 * 256 using the extract_patches.py script. This script not only extract patches but also store the patches into an H5 file for easy data transfer and management. However, we use our data annotation file so the annotation parse and check portion need to change for other datasets.

We store the patches in the h5 files who has the format subtype_name/slide_id/patch_locaton_x_y and use .txt files to store the data entry ids.

Patch-level: train, validation and test

The following bash script is used to invoke training, validation and test:

#!/bin/bash
chmod 775 ./patch_level.py
echo 'Two-stage model using split A'

echo 'Stage 1 - Patch Size 256 * 256 Training'
./patch_level.py  --deep_model DeepModel --deep_classifier two_stage --model_name_prefix split_a --use_pretrained --lr 0.0002 --batch_size 64 --epoch 20 --rep_intv 250 --use_equalized_batch --n_eval_samples 2000 --is_multiscale_expert --expert_magnification 256 --dataset_dir /projects/ovcare/classification/ywang/midl_dataset/1024_resize --preload_image_file_name 1024_resize.h5 --train_ids_file_name patch_ids/1_2_train_3_eval_train_ids.txt  --val_ids_file_name patch_ids/1_2_train_3_eval_eval_0_ids.txt --log_dir /projects/ovcare/classification/ywang/project_log/1024_resize_log/ --save_dir /projects/ovcare/classification/ywang/project_save/1024_resize_save/
echo 'Stage 1 - Patch Size 256 * 256 Validation'
./patch_level.py  --mode Validation --deep_model DeepModel --deep_classifier two_stage --model_name_prefix split_a --use_pretrained --lr 0.0002 --batch_size 64 --epoch 20 --rep_intv 250 --use_equalized_batch --n_eval_samples 2000 --is_multiscale_expert --expert_magnification 256 --dataset_dir /projects/ovcare/classification/ywang/midl_dataset/1024_resize --preload_image_file_name 1024_resize.h5 --train_ids_file_name patch_ids/1_2_train_3_eval_train_ids.txt  --val_ids_file_name patch_ids/1_2_train_3_eval_eval_0_ids.txt --log_dir /projects/ovcare/classification/ywang/project_log/1024_resize_log/ --save_dir /projects/ovcare/classification/ywang/project_save/1024_resize_save/

echo 'Stage 2 - Patch Size 512 * 512 Training'
./patch_level.py  --deep_model DeepModel --deep_classifier two_stage --model_name_prefix split_a --use_pretrained --lr 0.0002 --batch_size 32 --epoch 20 --rep_intv 250 --use_equalized_batch --n_eval_samples 2000 --is_multiscale_expert --expert_magnification 512 --dataset_dir /projects/ovcare/classification/ywang/midl_dataset/1024_resize --preload_image_file_name 1024_resize.h5 --train_ids_file_name patch_ids/1_2_train_3_eval_train_ids.txt  --val_ids_file_name patch_ids/1_2_train_3_eval_eval_0_ids.txt --log_dir /projects/ovcare/classification/ywang/project_log/1024_resize_log/ --save_dir /projects/ovcare/classification/ywang/project_save/1024_resize_save/
echo 'Stage 2 - Patch Size 512 * 512 Validation'
./patch_level.py  --mode Validation --deep_model DeepModel --deep_classifier two_stage --model_name_prefix split_a --use_pretrained --lr 0.0002 --batch_size 32 --epoch 20 --rep_intv 250 --use_equalized_batch --n_eval_samples 2000 --is_multiscale_expert --expert_magnification 512 --dataset_dir /projects/ovcare/classification/ywang/midl_dataset/1024_resize --preload_image_file_name 1024_resize.h5 --train_ids_file_name patch_ids/1_2_train_3_eval_train_ids.txt  --val_ids_file_name patch_ids/1_2_train_3_eval_eval_0_ids.txt --log_dir /projects/ovcare/classification/ywang/project_log/1024_resize_log/ --save_dir /projects/ovcare/classification/ywang/project_save/1024_resize_save/

Slide-level: train and test

We train Random Forests using 6-fold cross-validation on the results of six patch-level test set.

After changing the path to the six patch-level results in the slide_level.py, simply run python3 slide_level.py and it will output the slide-level results as well as save the trained model.

Our results

We include our patch-level and slide-level results in ./results/.

Datasets and Detailed Results

The epithelial ovarian carcinoma whole-slide pathology images used in this study will be available upon institutional review board approvals at a later date. We will update this page once the dataset is availablle.

Our dataset has the following distribution in terms of patients, slides, and 1024 * 1024 tumor patches:

Data Type	CC	LGSC	EC	MC	HGSC	Total
Patient	32	14	28	9	76	159
Slide	53	29	55	11	157	305
Patch	16.49%	16.31%	12.93%	10.96%	43.31%	161516

We first randomly divided the datasets by patients into three groups. We denote these three groups as Group 1, Group 2, and Group 3.

Group 1 has the following distributions:

Data Type	CC	LGSC	EC	MC	HGSC	Total
Patient	11	5	10	3	26	55
Slide	20	8	14	4	54	100
Patch	15.23%	10.43%	11.44%	13.93%	48.98%	56034

Group 2 has the following distributions:

Data Type	CC	LGSC	EC	MC	HGSC	Total
Patient	11	5	10	3	26	55
Slide	16	8	26	3	60	113
Patch	12.30%	17.80%	15.51%	7.09%	47.30%	64855

Group 3 has the following distributions:

Data Type	CC	LGSC	EC	MC	HGSC	Total
Patient	10	4	8	3	24	49
Slide	17	13	15	4	43	92
Patch	24.91%	22.05%	10.89%	13.03%	29.12%	40627

Slide-level and Patch-Level Results

For slide-level classification, we only use the patch-level test set results (as shown in the above figure) to build the input matrix to train random forests, and we report the 6-fold cross-validation slide-level results.

Split	CC	LGSC	EC	MC	HGSC	Weighted Accuracy	Kappa	AUC	F1 Score	Average Accuracy
baseline 6-fold cross-validation	83.02%	65.52%	54.55%	54.55%	80.25%	73.77%	0.5993	0.9391	0.6855	67.58%
stage-1 6-fold cross-validation	79.25%	79.31%	61.82%	54.55%	85.99%	78.69%	0.6730	0.9375	0.7414	72.18%
stage-2 6-fold cross-validation	86.79%	100.00%	74.55%	81.82%	90.45%	87.54%	0.8106	0.9641	0.8718	86.72%

For patch-level classification, we employ a 3-fold cross-validation scheme with a tweak. We use two of three patient groups as the training set and divide the remaining group equally by patient into two subgroups, one of the subgroups will be used as validation or test set. Therefore, we eventually have 6 different training, validation and test set.

Split A Distribution and Patch-level Classifier Test Results

Training set

Data Type	CC	LGSC	EC	MC	HGSC	Total
Patient	22	10	20	6	52	110
Slide	36	16	40	7	114	213
Patch	13.66%	14.39%	13.62%	10.26%	48.08%	120889

Validation set

Data Type	CC	LGSC	EC	MC	HGSC	Total
Patient	5	2	4	2	12	25
Slide	7	3	9	3	19	41
Patch	17.31%	27.79%	7.85%	31.73%	15.33%	14594

Test set

Data Type	CC	LGSC	EC	MC	HGSC	Total
Patient	5	2	4	1	12	24
Slide	10	10	6	1	24	51
Patch	29.18%	18.83%	12.59%	2.55%	36.85%	26033

Baseline, Stage-1 and Stage-2 test results

Model	CC	LGSC	EC	MC	HGSC	Weighted Accuracy	Kappa	AUC	F1 Score	Average Accuracy
Baseline	99.22%	89.60%	73.89%	100.00%	75.05%	85.33%	0.8015	0.9739	0.8546	87.55%
Stage-1	99.42%	79.05%	63.58%	99.85%	74.70%	81.97%	0.7543	0.9651	0.8105	83.32%
Stage-2	99.50%	77.34%	72.70%	99.55%	72.64%	82.05%	0.7568	0.9658	0.8243	84.34%

Split B Distribution and Patch-level Classifier Test Results

Training set

Data Type	CC	LGSC	EC	MC	HGSC	Total
Patient	22	10	20	6	52	110
Slide	36	16	40	7	114	213
Patch	13.66%	14.39%	13.62%	10.26%	48.08%	120889

Validation set

Data Type	CC	LGSC	EC	MC	HGSC	Total
Patient	5	2	4	1	12	24
Slide	10	10	6	1	24	51
Patch	29.18%	18.83%	12.59%	2.55%	36.85%	26033

Test set

Data Type	CC	LGSC	EC	MC	HGSC	Total
Patient	5	2	4	2	12	25
Slide	7	3	9	3	19	41
Patch	17.31%	27.79%	7.85%	31.73%	15.33%	14594

Baseline, Stage-1 and Stage-2 test results

Model	CC	LGSC	EC	MC	HGSC	Weighted Accuracy	Kappa	AUC	F1 Score	Average Accuracy
Baseline	89.90%	76.58%	81.05%	20.89%	58.78%	58.84%	0.4986	0.8969	0.5698	65.44%
Stage-1	89.59%	74.04%	75.72%	44.58%	50.38%	63.89%	0.5521	0.8997	0.6122	66.86%
Stage-2	86.26%	64.32%	63.23%	37.93%	60.84%	59.13%	0.4965	0.8823	0.5711	62.52%

Split C Distribution and Patch-level Classifier Test Results

Training set

Data Type	CC	LGSC	EC	MC	HGSC	Total
Patient	21	9	18	6	50	104
Slide	37	21	29	8	97	192
Patch	19.30%	15.31%	11.21%	13.55%	40.63%	96661

Validation set

Data Type	CC	LGSC	EC	MC	HGSC	Total
Patient	6	3	5	2	13	29
Slide	7	3	13	2	25	50
Patch	6.87%	12.82%	12.82%	12.84%	54.64%	18990

Test set

Data Type	CC	LGSC	EC	MC	HGSC	Total
Patient	5	2	5	1	13	26
Slide	9	5	13	1	35	63
Patch	14.55%	19.86%	16.62%	4.71%	44.26%	45865

Baseline, Stage-1 and Stage-2 test results

Model	CC	LGSC	EC	MC	HGSC	Weighted Accuracy	Kappa	AUC	F1 Score	Average Accuracy
Baseline	95.11%	54.81%	80.17%	96.76%	63.07%	70.52%	0.6035	0.9335	0.7300	77.99%
Stage-1	84.44%	32.51%	77.52%	97.45%	81.99%	72.51%	0.6132	0.9276	0.7162	74.78%
Stage-2	97.06%	58.49%	78.55%	97.64%	68.81%	73.84%	0.6452	0.9507	0.7532	80.11%

Split D Distribution and Patch-level Classifier Test Results

Training set

Data Type	CC	LGSC	EC	MC	HGSC	Total
Patient	21	9	18	6	50	104
Slide	37	21	29	8	97	192
Patch	19.30%	15.31%	11.21%	13.55%	40.63%	96661

Validation set

Data Type	CC	LGSC	EC	MC	HGSC	Total
Patient	5	2	5	1	13	26
Slide	9	5	13	1	35	63
Patch	14.55%	19.86%	16.62%	4.71%	44.26%	45865

Test set

Data Type	CC	LGSC	EC	MC	HGSC	Total
Patient	6	3	5	2	13	29
Slide	7	3	13	2	25	50
Patch	6.87%	12.82%	12.82%	12.84%	54.64%	18990

Baseline, Stage-1 and Stage-2 test results

Model	CC	LGSC	EC	MC	HGSC	Weighted Accuracy	Kappa	AUC	F1 Score	Average Accuracy
Baseline	64.29%	83.37%	48.71%	99.84%	80.76%	78.30%	0.6790	0.9473	0.7212	75.39%
Stage-1	76.55%	79.10%	40.04%	98.24%	87.14%	80.77%	0.7056	0.9423	0.7383	76.21%
Stage-2	65.06%	84.11%	33.76%	95.41%	88.33%	80.10%	0.6881	0.9277	0.7261	73.33%

Split E Distribution and Patch-level Classifier Test Results

Training set

Data Type	CC	LGSC	EC	MC	HGSC	Total
Patient	21	9	18	6	50	104
Slide	33	21	41	7	103	205
Patch	17.16%	19.44%	13.73%	9.38%	40.30%	105482

Validation set

Data Type	CC	LGSC	EC	MC	HGSC	Total
Patient	6	3	5	2	13	29
Slide	11	3	7	3	27	51
Patch	12.62%	10.19%	11.73%	39.90%	25.57%	16690

Test set

Data Type	CC	LGSC	EC	MC	HGSC	Total
Patient	5	2	5	1	13	26
Slide	9	5	7	1	27	49
Patch	16.33%	10.54%	11.32%	2.91%	58.91%	39344

Baseline, Stage-1 and Stage-2 test results

Model	CC	LGSC	EC	MC	HGSC	Weighted Accuracy	Kappa	AUC	F1 Score	Average Accuracy
Baseline	79.29%	88.15%	26.77%	99.74%	65.01%	66.47%	0.5003	0.9149	0.6336	71.79%
Stage-1	61.00%	95.37%	58.06%	98.78%	68.85%	70.01%	0.5490	0.9371	0.7058	76.41%
Stage-2	70.92%	90.98%	47.80%	97.81%	66.74%	68.74%	0.5285	0.9156	0.7067	74.85%

Split F Distribution and Patch-level Classifier Test Results

Training set

Data Type	CC	LGSC	EC	MC	HGSC	Total
Patient	21	9	18	6	50	104
Slide	33	21	41	7	103	205
Patch	17.16%	19.44%	13.73%	9.38%	40.30%	105482

Validation set

Data Type	CC	LGSC	EC	MC	HGSC	Total
Patient	5	2	5	1	13	26
Slide	9	5	7	1	27	49
Patch	16.33%	10.54%	11.32%	2.91%	58.91%	39344

Test set

Data Type	CC	LGSC	EC	MC	HGSC	Total
Patient	6	3	5	2	13	29
Slide	11	3	7	3	27	51
Patch	12.62%	10.19%	11.73%	39.90%	25.57%	16690

Baseline, Stage-1 and Stage-2 test results

Model	CC	LGSC	EC	MC	HGSC	Weighted Accuracy	Kappa	AUC	F1 Score	Average Accuracy
Baseline	94.78%	31.24%	65.25%	39.14%	51.64%	51.61%	0.3993	0.8751	0.5124	56.41%
Stage-1	95.58%	33.35%	90.04%	33.88%	58.13%	54.40%	0.4404	0.8893	0.5464	62.20%
Stage-2	95.96%	28.12%	53.65%	41.63%	75.05%	57.06%	0.4615	0.8287	0.5492	58.88%