Oral Image Generation Pipeline

This repository implements a comprehensive pipeline for generating high-quality synthetic oral images for dental research and clinical applications. The pipeline consists of three main stages: normal oral image generation, decayed oral image generation, and super-resolution enhancement.

Project Overview

Our methodology follows a three-stage approach to generate realistic oral images:

1. Normal Oral Image Generation using Semantic Diffusion Models (SDM)
2. Decayed Oral Image Generation using SANTA (Shortest Path Regularization)
3. Super-Resolution Enhancement using Real-ESRGAN

Repository Structure

Oral-Image-Generation/
├── SDM/                    # Semantic Diffusion Model for normal oral image generation
├── SANTA/                  # SANTA model for healthy-to-decayed tooth translation
├── Real-ESRGAN/           # Real-ESRGAN for super-resolution enhancement
└── README.md              # This file

Pipeline Components

1. SDM (Semantic Diffusion Model)

Location: SDM/

Generates synthetic oral images containing only healthy teeth using a diffusion-based approach.

Key Features:

• Based on Denoising Diffusion Model (DDPM)
• Separate processing of noisy images and semantic masks
• Multi-layer spatially-adaptive normalization (SPADE)
• Individual models for five views: upper, lower, frontal, left, right
• Output resolution: 256×256

Usage:

Training (Single-GPU):

cd SDM/
python image_train.py \
    --data_dir /path/to/dental/dataset \
    --dataset_mode nia_full \
    --lr 2e-5 \
    --batch_size 4 \
    --image_size 256 \
    --num_classes 66 \
    --class_cond True \
    --no_instance True \
    --save_interval 1000

Training (Multi-GPU):

cd SDM/
export OPENAI_LOGDIR='OUTPUT/DENTAL-SDM-256CH'
mpiexec -n 8 python image_train.py \
    --data_dir /path/to/dental/dataset \
    --dataset_mode nia_full \
    --lr 2e-5 \
    --batch_size 4 \
    --attention_resolutions 32,16,8 \
    --diffusion_steps 1000 \
    --image_size 256 \
    --learn_sigma True \
    --noise_schedule linear \
    --num_channels 256 \
    --num_head_channels 64 \
    --num_res_blocks 2 \
    --resblock_updown True \
    --use_fp16 True \
    --use_scale_shift_norm True \
    --use_checkpoint True \
    --num_classes 66 \
    --class_cond True \
    --no_instance True

Sampling/Generation:

cd SDM/
python image_sample.py \
    --data_dir /path/to/dental/dataset \
    --dataset_mode nia_full \
    --attention_resolutions 32,16,8 \
    --diffusion_steps 1000 \
    --image_size 256 \
    --learn_sigma True \
    --noise_schedule linear \
    --num_channels 256 \
    --num_head_channels 64 \
    --num_res_blocks 2 \
    --resblock_updown True \
    --use_fp16 True \
    --use_scale_shift_norm True \
    --num_classes 66 \
    --class_cond True \
    --no_instance True \
    --batch_size 2 \
    --num_samples 10000 \
    --model_path OUTPUT/DENTAL-SDM/ema_0.9999_050000.pt \
    --results_path RESULTS/DENTAL-SDM \
    --s 1.5

2. SANTA (Shortest Path Regularization)

Location: SANTA/

Performs unpaired image-to-image translation to generate decayed teeth from healthy teeth.

Key Features:

• Unpaired domain translation using shared latent space
• Individual tooth-specific model training
• Tooth region cropping and integration pipeline
• Preserves semantic content during translation
• Generates localized dental caries

Main Script: SANTA/generate_decayed_oral.py

Usage:

Training Individual SANTA Models:

cd SANTA/
python train.py \
    --dataroot /path/to/dental/dataset \
    --direction AtoB \
    --lambda_path 0.1 \
    --n_epochs 100 \
    --n_epochs_decay 100

Generation of Decayed Oral Images:

cd SANTA/
python generate_decayed_oral.py \
    --input_dir /path/to/images \
    --mask_dir /path/to/masks \
    --output_dir /path/to/output \
    --model_dir /path/to/models \
    --tooth_labels tooth_11 tooth_16 tooth_26  # specify tooth labels

Key Functions:

• crop_tooth_region(): Crops target tooth region from oral image using tooth mask
• generate_decayed_tooth(): Generates decayed tooth image using SANTA model
• integrate_decayed_tooth(): Naturally integrates generated decayed tooth into original oral image

3. Real-ESRGAN (Super-Resolution)

Location: Real-ESRGAN/

Enhances the resolution of generated oral images for clinical applications.

Key Features:

• Pre-trained realesrgan-x4plus model
• 6× upscaling (256×256 → 1536×1536)
• Final resize to 1920×1080 using Lanczos interpolation
• No fine-tuning required
• Clinical-grade output quality

Main Script: Real-ESRGAN/super_resolution_pipeline.py

Usage:

cd Real-ESRGAN/
python super_resolution_pipeline.py

Key Papers and References

• SDM: Semantic Image Synthesis via Diffusion Models
• SANTA: Unpaired Image-to-Image Translation With Shortest Path Regularization (CVPR2023)
• Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data (ICCVW 2021)

Third-Party Models & Code

This repository leverages the following open-source implementations for research purposes. Copyrights and licenses remain with the original authors. Please consult each repository’s LICENSE and documentation before redistribution or commercial use.

1) Semantic Image Synthesis via Diffusion Models (SDM)

• Repository: https://github.com/WeilunWang/semantic-diffusion-model
• Paper: https://arxiv.org/abs/2207.00050
• License: No explicit LICENSE file is present in the repository as of 2025‑08‑30. Please confirm terms with the authors before redistribution.
• We used it for: semantic layout–conditioned generation of healthy oral images.
• Our modifications: dataset loader adjustments for dental views; training script options adapted for five standardized views (upper, lower, frontal, left, right).

2) SANTA — Unpaired Image‑to‑Image Translation with Shortest Path Regularization (CVPR 2023)

• Repository: https://github.com/Mid-Push/santa
• Paper: https://openaccess.thecvf.com/content/CVPR2023/papers/Xie_Unpaired_Image-to-Image_Translation_With_Shortest_Path_Regularization_CVPR_2023_paper.pdf
• License: MIT License
• We used it for: unpaired healthy→decayed tooth translation while preserving semantic structure.
• Our modifications: tooth‑region cropping/integration pipeline; per‑tooth models; mask‑guided inference.

3) Real‑ESRGAN — Real‑World Blind Super‑Resolution with Pure Synthetic Data (ICCVW 2021)

• Repository: https://github.com/xinntao/Real-ESRGAN
• Paper: https://arxiv.org/abs/2107.10833
• License: BSD‑3‑Clause
• We used it for: super‑resolution enhancement of synthesized oral images (256×256 → 1536×1536 → 1920×1080).
• Our modifications: batch inference wrapper and default output sizing for clinical viewing.

Attribution & Compliance

• When publishing results derived from this pipeline, include proper attribution to all three upstream projects and comply with their licenses (e.g., retaining copyright notices and including license texts where required).
• Redistribution of any modified upstream code must follow the respective license terms. In particular, verify SDM’s redistribution terms due to the absence of an explicit license file at the time of writing.

Requirements

Each component has its own requirements. Please refer to the individual README files in each directory:

• SDM/README.md
• SANTA/README.md
• Real-ESRGAN/README.md

Getting Started

1. Setup Environment: Install dependencies for each component
2. Prepare Data: Collect tooth images and semantic masks for each view
3. Train Models: Train SDM and SANTA models according to your dataset
4. Run Pipeline: Execute the three-stage pipeline to generate final oral images

Citation

If you use this pipeline in your research, please cite the following works in addition to acknowledging this implementation.

BibTeX

@article{wang2022semantic,
  title={Semantic Image Synthesis via Diffusion Models},
  author={Wang, Weilun and Bao, Jianmin and Zhou, Wengang and Chen, Dongdong and Chen, Dong and Yuan, Lu and Li, Houqiang},
  journal={arXiv preprint arXiv:2207.00050},
  year={2022}
}

@inproceedings{xie2023unpaired,
  title={Unpaired Image-to-Image Translation With Shortest Path Regularization},
  author={Xie, Shaoan and Xu, Yanwu and Gong, Mingming and Zhang, Kun},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={10177--10187},
  year={2023}
}

@inproceedings{wang2021realesrgan,
  author    = {Xintao Wang and Liangbin Xie and Chao Dong and Ying Shan},
  title     = {Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data},
  booktitle = {International Conference on Computer Vision Workshops (ICCVW)},
  year      = {2021}
}

License

Please refer to individual license files in each component directory.