Explicit Temporal Embedding in Deep Generative Latent Models for Longitudinal Medical Image Synthesis

Official PyTorch implementation of the paper "Explicit Temporal Embedding in Deep Generative Latent Models for Longitudinal Medical Image Synthesis".

Abstract: Medical imaging plays a vital role in modern diagnostics and treatment. The temporal nature of disease or treatment progression often results in longitudinal data. Due to the cost and potential harm, acquiring large medical datasets necessary for deep learning can be difficult. To address this, we use the recent advances in latent space-based image editing to propose a novel joint learning scheme to explicitly embed temporal dependencies in the latent space of GANs. This allows us to synthesize continuous, smooth, and high-quality longitudinal volumetric data with limited supervision.

Key Features

• Explicit Temporal Embedding: Jointly trains a GAN and a linear direction in the latent space to capture temporal evolution (e.g., breathing motion, tumor regression).
• Volumetric Synthesis: Capable of generating high-quality 3D longitudinal data ($128 \times 128 \times 128$ voxels).
• Continuous Generation: Unlike sequence-to-sequence models, this approach allows for smooth, continuous interpolation of time steps by shifting the latent code.
• Model Agnostic: The training procedure can be adapted to various latent variable-based GAN architectures (implemented here using an adapted SA-GAN).

Visuals

LIDC - x-Shift

4D CT - Breathing Motion

CBCT - Tumor Regression

Citation

@article{schoen2023explicit,
  title={Explicit Temporal Embedding in Deep Generative Latent Models for Longitudinal Medical Image Synthesis},
  author={Sch{\"o}n, Julian and Selvan, Raghavendra and Nyg{\aa}rd, Lotte and Vogelius, Ivan Richter and Petersen, Jens},
  journal={arXiv preprint arXiv:2301.05465},
  year={2023}
}