Joint CT & Anatomical Mask Generation

Official implementation of GuideGen in GuideGen: A Text-guided Framework for Joint CT Volume and Anatomical structure Generation, submitted to MICCAI 2024, and sadly, rejected, so I am not updating the contents in this repo for now, until the updated version of our work is released, hopefully by the end of this year.

Code

Note that you may need to first familiarize yourself with the methodology of CCDM and LDM, since these modules are modified and combined in our paper to cope with our goal of joint CT & anatomical mask generation.

Sample training & evaluation code for CCDM

cd ./ccdm
# Training
python ddpm_train.py ./params.yml <exp_name>
# Evaluation
python ddpm_eval.py ./params_eval.yml <exp_name>

Sample training and evaluation code for LDM

cd ./ldm
# Training
python main.py --base ./configs/latent-diffusion/<cfg_name> -t --gpus 0, (--resume_from_checkpoint <ckpt_path>)
# Evaluation
python sample_diffusion.py -r <ckpt_path> --inputs <stage_1_generated_mask_dir> --batch_size 1

Teasers

Overall pipeline for GuideGen. At inference time, given a text condition (white), the volumetric mask sampler (yellow) outputs a corresponding segmentation mask for major abdominal organs and tumor site. This mask is upsampled and sliced before passing into the conditional image generator (blue) to generate the CT volume autoregressively.

Comparative Results with other Methods on our private dataset. Colored regions on the right represent different organ masks and green area marked in each figure represents generated tumor site. Green, red and blue boxes on generated masks represent synthesized tumor masks that is well-positioned, misplaced or missing with respect to the text condition (which is exhibited as real tumor locations in the Original Anatomies).

Insights

• FVD may not be good evaluation metric for medical imaging, a similar insight has been substantiated by "XReal: Realistic Anatomy and Pathology-Aware X-ray Generation via Controllable Diffusion Model"
• Direct cross-attn of texts may not be so helpful when they are the only guidance for image generation, need to think of other ways, especially when they are not articulating themselves
• ...