Skip to content

Latest commit

 

History

History
75 lines (59 loc) · 4.37 KB

BENCHMARKS.md

File metadata and controls

75 lines (59 loc) · 4.37 KB
Raw

This file contains benchmark AUC performance for predicting each pathlogy on 20% of each dataset.

To reproduce these results use the script scripts/model_calibrate.py. This script will take a long time so it will cache the results so you can work with them after the compute is done. The script also calculates the calibration for a model but those outputs can be ignored for this benchmarking. Below is an example of running different models on the PadChest pc dataset and writing the output as markdown.

python model_calibrate.py pc resnet50-res512-all -mdtable
python model_calibrate.py pc chexpert -mdtable
python model_calibrate.py pc jfhealthcare -mdtable
python model_calibrate.py pc densenet121-res224-all -mdtable

Results updated: 07/23/2023

NIH ChestX-ray14

Model Name Atelectasis Cardiomegaly Consolidation Edema Effusion Emphysema Fibrosis Hernia Infiltration Mass Nodule Pleural_Thickening Pneumonia Pneumothorax
XRV-ResNet-resnet50-res512-all 0.78 0.90 0.78 0.89 0.86 0.88 0.76 0.92 0.69 0.82 0.76 0.76 0.71 0.84
XRV-DenseNet121-densenet121-res224-all 0.76 0.88 0.77 0.85 0.85 0.73 0.72 0.91 0.68 0.80 0.69 0.74 0.71 0.75
jfhealthcare-DenseNet121 0.76 0.85 0.78 0.87 0.87 - - - - - - - - -
CheXpert-DenseNet121-ensemble 0.80 0.88 0.79 0.88 0.87 - - - - - - - - -

Google

Model Name # Params Lung Opacity Fracture Nodule or mass Pneumothorax
XRV-ResNet-resnet50-res512-all 23,538,642 0.7 0.88 - 0.92
XRV-DenseNet121-densenet121-res224-all 6,966,034 0.92 0.74 - 0.85

RSNA

Model Name # Params Lung Opacity Pneumonia
XRV-ResNet-resnet50-res512-all 23,538,642 0.85 0.87
XRV-DenseNet121-densenet121-res224-all 6,966,034 0.88 0.86

SIIM

Model Name # Params Pneumothorax
XRV-ResNet-resnet50-res512-all 23,538,642 0.91
XRV-DenseNet121-densenet121-res224-all 6,966,034 0.79

PadChest

Model Name # Params Atelectasis Cardiomegaly Consolidation Edema Effusion Emphysema Fibrosis Fracture Hernia Infiltration Mass Nodule Pleural_Thickening Pneumonia Pneumothorax
XRV-ResNet-resnet50-res512-all 23,538,642 0.80 0.94 0.88 0.97 0.95 0.86 0.96 0.86 0.95 0.85 0.85 0.76 0.85 0.81 0.87
XRV-DenseNet121-densenet121-res224-all 6,966,034 0.77 0.93 0.88 0.97 0.95 0.87 0.94 0.70 0.96 0.85 0.85 0.69 0.79 0.82 0.81
jfhealthcare-DenseNet121 12,525,301 0.78 0.89 0.82 0.94 0.96 - - - - - - - - - -
CheXpert-DenseNet121-ensemble 0.82 0.92 0.88 0.97 0.97 - - - - - - - - - -

VinBrain

Model Name # Params Atelectasis Cardiomegaly Consolidation Effusion Infiltration Lung Opacity Pleural_Thickening Pneumothorax
XRV-ResNet-resnet50-res512-all 23,538,642 0.60 0.85 0.91 0.85 0.82 0.71 0.79 0.69
XRV-DenseNet121-densenet121-res224-all 6,966,034 0.67 0.90 0.93 0.87 0.86 0.85 0.84 0.93
jfhealthcare-DenseNet121 12,525,301 0.79 0.81 0.95 0.92 - - - -
CheXpert-DenseNet121-ensemble 0.74 0.89 0.97 0.93 - - - -

CheXpert

Model Name # Params Atelectasis Cardiomegaly Consolidation Edema Enlarged Cardiomediastinum Fracture Lung Lesion Lung Opacity Effusion Pleural Other Pneumonia Pneumothorax Support Devices
XRV-ResNet-resnet50-res512-all 23,538,642 0.63 0.84 0.74 0.79 0.5 0.58 0.50 0.71 0.81 - 0.67 0.61 -
XRV-DenseNet121-densenet121-res224-all 6,966,034 0.91 0.91 0.90 0.92 0.78 0.74 0.84 0.87 0.94 - 0.84 0.85 -
jfhealthcare-DenseNet121 12,525,301 0.91 0.89 0.91 0.90 - - - - 0.95 - - - -
CheXpert-DenseNet121-ensemble 0.93 0.91 0.91 0.92 - - - - 0.96 - - - -

MIMIC-CXR (CheXpert labels)

Model Name Atelectasis Cardiomegaly Consolidation Edema Enlarged Cardiomediastinum Fracture Lung Lesion Lung Opacity Effusion Pleural Other Pneumonia Pneumothorax Support Devices
XRV-DenseNet121-densenet121-res224-all 0.88 0.88 0.91 0.92 0.84 0.74 0.82 0.86 0.92 - 0.82 0.81 -
XRV-ResNet-resnet50-res512-all 0.77 0.78 0.88 0.87 0.50 0.66 0.5 0.77 0.9 - 0.77 0.72 -
jfhealthcare-DenseNet121 0.90 0.82 0.93 0.93 - - - - 0.94 - - - -
CheXpert-DenseNet121-ensemble 0.88 0.84 0.93 0.93 - - - - 0.94 - - - -