UMIE_datasets

🤩 About the Project

Warning: This project is currently in alpha stage and may be subject to major changes

This repository presents a suite of unified scripts to standardize, preprocess, and integrate 882,774 images from 20 open-source medical imaging datasets, spanning modalities such as X-ray, CT, and MR. The scripts allow for seamless and fast download of a diverse medical data set. We create a unified set of annotations allowing for merging the datasets together without mislabelling. Each dataset is preprocessed with a custom sklearn pipeline. The pipeline steps are reusable across the datasets. The code was designed so that preorocessing a new dataset is simple and requires only reusing the available pipeline steps with customization performed through setting the appropriate values of the pipeline params.

The labels and segmentation masks were unified to be compliant with RadLex ontology.

Datasets

uid	Dataset	Modality	TASK
0	KITS-23	CT	Classification/Segmentation
1	CoronaHack	XRAY	Classification
2	Alzheimers Dataset	MRI	Classification
3	Brain Tumor Classification	MRI	Classification
4	COVID-19 Detection X-Ray	XRAY	Classification
5	Finding and Measuring Lungs in CT Data	CT	Segmentation
6	Brain CT Images with Intracranial Hemorrhage Masks	CT	Classification
7	Liver and Liver Tumor Segmentation	CT	Classification, Segmentation
8	Brain MRI Images for Brain Tumor Detection	MRI	Classification
9	Knee Osteoarthritis Dataset with Severity Grading	XRAY	Classification
10	Brain Tumor Progression	MRI	Segmentation
11	Chest X-ray 14	XRAY	Classification
12	COCA- Coronary Calcium and chest CTs	CT	Segmentation
13	BrainMetShare	MRI	Segmentation
14	CT-ORG	CT	Segmentation
17	LIDC-IDRI	CT	Segmentation
18	CMMD	MG	Classification

Using the datasets

Installing requirements

uv sync

Creating the dataset

Due to the copyright restrictions of the source datasets, we can't share the files directly. To obtain the full dataset you have to download the source datasets yourself and run the preprocessing scripts.

0.KITS-23

KITS-23

1. Clone the KITS-23 repository.
2. Enter the KITS-23 directory and install the packages with pip.

   cd kits23
   pip3 install -e .

3. Run the following command to download the data to the dataset/ folder.

   kits23_download_data
   

4. Fill in the source_path and target_path KITS-23Pipeline() in config/runner_config.py. e.g.

    KITS23Pipeline(
         path_args={
             "source_path": "kits23/dataset",  # Path to the dataset directory in KITS23 repo
             "target_path": TARGET_PATH,
             "labels_path": "kits23/dataset/kits23.json",  # Path to kits23.json
         },
         dataset_args=dataset_config.KITS23
     ),

1. Xray CoronaHack -Chest X-Ray-Dataset

Xray CoronaHack -Chest X-Ray-Dataset

1. Go to CoronaHack page on Kaggle.
2. Login to your Kaggle account.
3. Download the data.
4. Extract archive.zip.
5. Fill in the source_path to the location of the archive folder in CoronaHackPipeline() in config/runner_config.py.

2. Alzheimer's Dataset

Alzheimer's Dataset (4 class of Images)

1. Go to Alzheimer's Dataset page on Kaggle.
2. Login to your Kaggle account.
3. Download the data.
4. Extract archive.zip.
5. Fill in the source_path to the location of the archive folder in AlzheimersPipeline() in config/runner_config.py.

3. Brain Tumor Classification (MRI)

3. Brain Tumor Classification (MRI)

1. Go to Brain Tumor Classification page on Kaggle.
2. Login to your Kaggle account.
3. Download the data.
4. Extract archive.zip.
5. Fill in the source_path to the location of the archive folder in BrainTumorClassificationPipeline() in config/runner_config.py.

4. COVID-19 Detection X-Ray

COVID-19 Detection X-Ray

1. Go to COVID-19 Detection X-Ray page on Kaggle.
2. Login to your Kaggle account.
3. Download the data.
4. Extract archive.zip.
5. REMOVE TrainData folder. We do not want augmented data at this stage.
6. Fill in the source_path to the location of the archive folder in COVID19DetectionPipeline() in config/runner_config.py.

5. Finding and Measuring Lungs in CT Dat

Finding and Measuring Lungs in CT Data

1. Go to Finding and Measuring Lungs in CT Data page on Kaggle.
2. Login to your Kaggle account.
3. Download the data.
4. Extract archive.zip.
5. Fill in the source_path to the location of the archive/2d_images folder in FindingAndMeasuringLungsPipeline() in config/runner_config.py. Fill in masks_path with the location of the archive/2d_masks folder.

6. Brain CT Images with Intracranial Hemorrhage Masks

Brain CT Images with Intracranial Hemorrhage Masks

1. Go to Brain With Intracranial Hemorrhage page on Kaggle.
2. Login to your Kaggle account.
3. Download the data.
4. Extract archive.zip.
5. Fill in the source_path to the location of the archive folder in BrainWithIntracranialHemorrhagePipeline() in config/runner_config.py. Fill in masks_path with the same path as the source_path.

7. Liver and Liver Tumor Segmentation (LITS)

Liver and Liver Tumor Segmentation (LITS)

1. Go to Liver and Liver Tumor Segmentation.
2. Login to your Kaggle account.
3. Download the data.
4. Extract archive.zip.
5. Fill in the source_path to the location of the archive folder in COVID19DetectionPipeline() in config/runner_config.py. Fill in masks_path too.

8. Brain MRI Images for Brain Tumor Detection

Brain MRI Images for Brain Tumor Detection

1. Go to Brain MRI Images for Brain Tumor Detection page on Kaggle.
2. Login to your Kaggle account.
3. Download the data.
4. Extract archive.zip.
5. Fill in the source_path to the location of the archive folder in BrainTumorDetectionPipeline() in config/runner_config.py.

9. Knee Osteoarthrithis Dataset with Severity Grading

Knee Osteoarthrithis Dataset with Severity Grading

1. Go to Knee Osteoarthritis Dataset with Severity Grading.
2. Login to your Kaggle account.
3. Download the data.
4. Extract archive.zip.
5. Fill in the source_path to the location of the archive folder in COVID19DetectionPipeline() in config/runner_config.py.

10. Brain-Tumor-Progression

10. Brain-Tumor-Progression UPDATE: Some data in this collection contains images that could potentially be used to reconstruct a human face. The data collection is removed.

1. Go to Brain Tumor Progression dataset from the cancer imaging archive.

11. Chest X-ray 14

Chest X-ray 14

1. Go to Chest X-ray 14.
2. Create an account.
3. Download the images folder and DataEntry2017_v2020.csv.

12. COCA- Coronary Calcium and chest CTs

COCA- Coronary Calcium and chest CTs

1. Go to COCA- Coronary Calcium and chest CTs.
2. Log in or sign up for a Stanford AIMI account.
3. Fill in your contact details.
4. Download the data with azcopy.
5. Fill in the source_path with the location of the cocacoronarycalciumandchestcts-2/Gated_release_final/patient folder. Fill in masks_path with cocacoronarycalciumandchestcts-2/Gated_release_final/calcium_xml xml file.

13. BrainMetShare

BrainMetShare

1. Go to BrainMetShare.
2. Log in or sign up for a Stanford AIMI account.
3. Fill in your contact details.
4. Download the data with azcopy.

14. CT-ORG

CT-ORG

1. Go to CT-ORG page on Cancer imaging archive.
2. Download the data.
3. Extract PKG - CT-ORG.
4. Fill in the source_path to the location of the OrganSegmentations folder in CtOrgPipeline() in config/runner_config.py. Fill in masks_path with the same path as the source_path.

17. LIDC-IDRI

LIDC-IDRI

1. Go to LIDC-IDRI.
2. Download "Images" using NBIA Data Retriever, and "Radiologist Annotations/Segmentations".
3. Extract LIDC-XML-only.zip.
4. Fill in the source_path in CmmdPipeline() in config/runner_config.py with the location of the manifest-{xxxxxxxxxxxxx}/LIDC-IDRI directory.
5. Fill in the masks_path in CmmdPipeline() in config/runner_config.py with the location of the LIDC-XML-only/ directory.

   Data Citation
   Armato III, S. G., McLennan, G., Bidaut, L., McNitt-Gray, M. F., Meyer, C. R., Reeves, A. P., Zhao, B., Aberle, D. R., Henschke, C. I., Hoffman, E. A., Kazerooni, E. A., MacMahon, H., Van Beek, E. J. R., Yankelevitz, D., Biancardi, A. M., Bland, P. H., Brown, M. S., Engelmann, R. M., Laderach, G. E., Max, D., Pais, R. C. , Qing, D. P. Y. , Roberts, R. Y., Smith, A. R., Starkey, A., Batra, P., Caligiuri, P., Farooqi, A., Gladish, G. W., Jude, C. M., Munden, R. F., Petkovska, I., Quint, L. E., Schwartz, L. H., Sundaram, B., Dodd, L. E., Fenimore, C., Gur, D., Petrick, N., Freymann, J., Kirby, J., Hughes, B., Casteele, A. V., Gupte, S., Sallam, M., Heath, M. D., Kuhn, M. H., Dharaiya, E., Burns, R., Fryd, D. S., Salganicoff, M., Anand, V., Shreter, U., Vastagh, S., Croft, B. Y., Clarke, L. P. (2015). Data From LIDC-IDRI [Data set]. The Cancer Imaging Archive. https://doi.org/10.7937/K9/TCIA.2015.LO9QL9SX

18. CMMD - The Chinese Mammography Database

CMMD

1. Go to CMMD.
2. Download .tcia file from Data Access table.
3. Download NBIA Data Retriver to be able to download images.
4. Download CMMD_clinicaldata_revision.xlsx from Data Access table for labels information.
5. Fill in the source_path in CmmdPipeline() in config/runner_config.py with the location of the manifest-{xxxxxxxxxxxxx}/CMMD folder.
6. Fill in the labels_path in CmmdPipeline() in config/runner_config.py with the location of the CMMD_clinicaldata_revision.xlsx file.

   Data Citation
   ui, Chunyan; Li Li; Cai, Hongmin; Fan, Zhihao; Zhang, Ling; Dan, Tingting; Li, Jiao; Wang, Jinghua. (2021) The Chinese Mammography Database (CMMD): An online mammography database with biopsy confirmed types for machine diagnosis of breast. The Cancer Imaging Archive. DOI: https://doi.org/10.7937/tcia.eqde-4b16

To preprocess the dataset that is not among the above, search the preprocessing folder. It contains the reusable steps for changing imaging formats, extracting masks, creating file trees, etc. Go to the config file to check which masks and label encodings are available. Append new labels and mask encodings if needed.

Optional pipeline steps (data quality, preprocessing, metadata, formats, infra & export)

Beyond the core conversion steps, the pipeline ships a set of optional, opt-in steps that plug into any BasePipeline via its steps tuple. They are additive: when not configured they are a no-op, and they never change UMIE ids, the {uid}_{name}/{phase}/Images|Masks/ layout, or existing JSONL fields. Configuration is supplied per-pipeline through new sub-configs on PipelineArgs (quality=, preprocessing=, metadata=, format_conversion=, export=). Analysis steps write their findings to a reports/ folder next to Images/Masks.

Step	Purpose	Config
DetectDuplicates	perceptual-hash duplicate / near-duplicate & cross-dataset overlap report	quality
DetectCorruptImages	flag unreadable / truncated / blank / undersized images	quality
CheckMaskQuality	mask↔image dim match, in-vocabulary colors, empty-mask report	quality
ValidateDicomMetadata	verify required DICOM tags before conversion	quality
ApplyWindowing	named CT HU window presets (lung/bone/brain/…)	preprocessing
ApplyClahe	optional CLAHE contrast enhancement (images only)	preprocessing
NormalizeSpacing	resample NIfTI volumes to a target voxel spacing	preprocessing
ResizeImages	resize to a standard size (pad/crop/letterbox/stretch); masks nearest-neighbour	preprocessing
StandardizeBitDepth	standardize 8/16-bit sources	preprocessing
AutocropBorders	crop uniform black borders; mask cropped identically	preprocessing
ExtractDicomMetadata	extract de-identified DICOM tags into the JSONL	metadata
CreateSplits	reproducible patient/study-level train/val/test splits	metadata
AddProvenance	additive license / source-attribution fields (config/provenance.py)	metadata
ConvertDicomSeg	DICOM-SEG / RTSTRUCT → UMIE masks	format_conversion
ConvertBboxToMask	COCO / YOLO / VOC bounding boxes → UMIE masks	format_conversion
MergeMasks	merge single-structure masks into one multi-class mask	format_conversion
CreateManifest	SHA-256 manifest of outputs + a verify mode	export
SkipProcessed	incremental processing: skip already-converted files	export
ExportHuggingFace	write an HF datasets Arrow dataset (with splits) + auto dataset card	export

Cross-cutting utilities: utils/distribution_report.py (cross-dataset label/modality/imbalance report from the JSONL) and utils/parallel.py (deterministic, order-preserving parallel map).

Overall the dataset should have 882,774 images in .png format

• CT - 500k+
• X-Ray - 250k+
• MRI - 100k+

🎯 Roadmap

• dcm
• jpg
• nii
• tif
• Shared radlex ontology
• Huggingface datasets
• Data dashboards

👋 Contributors

🤝 Contact

Barbara Klaudel

TheLion.AI

Development

Pre-commits

Install pre-commits https://pre-commit.com/#installation

If you are using VS-code install the extention https://marketplace.visualstudio.com/items?itemName=MarkLarah.pre-commit-vscode

To make a dry-run of the pre-commits to see if your code passes run

pre-commit run --all-files

Adding python packages

Dependencies are handled by the uv package manager, to add a new dependency run

uv add <package_name>

Debugging

To modify and debug the app, development in containers can be useful .

Testing

run_tests.sh