GenDSA

Large-scale Pretrained Multi-Frame Generative Model Enables Real-Time Low-Dose DSA Imaging

Huangxuan Zhao^{1 🏷️ 📧},Ziyang Xu^{2 🏷️},Linxia Wu^{1 🏷️}, Lei Chen^{1 🏷️}, Ziwei Cui², Jinqiang Ma¹, Tao Sun¹, Yu Lei¹, Nan Wang³, Hongyao Hu⁴, Yiqing Tan⁵, Wei Lu⁶, Wenzhong Yang⁷, Kaibing Liao⁸, Gaojun Teng⁹, Xiaoyun Liang¹⁰, Yi Li¹⁰, Congcong Feng¹¹, Tong Nie¹, Xiaoyu Han¹, P.Matthijs van der Sluijs¹², Charles B.L.M. Majoie¹³, Wim H. van Zwam¹⁴, Yun Feng¹⁵, Theo van Walsum¹¹, Aad van der Lugt¹¹, Wenyu Liu², Xuefeng Kan^{1 📧}, Ruisheng Su^{11 📧}, Weihua Zhang^{9 📧}, Xinggang Wang^{2 📧}, Chuansheng Zheng^{1 📧}

(^🏷️) equal contribution, (^📧) corresponding author.

¹ Department of Radiology, Union Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan, China.
² Institute of AI, School of Electronic Information and Communications, Huazhong University of Science and Technology, Wuhan, China.
³ Department of Radiology, Tongji Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan, China.
⁴ Department of Interventional Radiology, Renmin Hospital of Wuhan University, Wuhan, China.
⁵ Department of Radiology, Tongren Hospital of Wuhan University (Wuhan Third Hospital), Wuhan University, Wuhan, China.
⁶ Department of Interventional Radiology, Zhongnan Hospital of Wuhan University, Wuhan, China.
⁷ Department of Radiology, Maternal and Child Health Hospital of Hubei Province, Wuhan, China.
⁸ Department of Radiology, Hubei Integrated Traditional Chinese and Western Medicine Hospital, Wuhan, China.
⁹ Department of Radiology, Zhongda Hospital, Medical School, Southeast University, Nanjing, China.
¹⁰ Institute of Research and Clinical Innovations, Neusoft Medical Systems, Co., Ltd, Shanghai, China.
¹¹ CV Systems Research and Development Department, Neusoft Medical Systems, Co., Ltd, Shenyang, China.
¹² Department of Radiology & Nuclear Medicine, Erasmus MC, University Medical Center Rotterdam, The Netherlands.
¹³ Department of Radiology and Nuclear Medicine, Amsterdam University Medical Centers, location AMC, Amsterdam, The Netherlands.
¹⁴ Department of Radiology and Nuclear Medicine, Cardiovascular Research Institute Maastricht, Maastricht University Medical Center, Maastricht, The Netherlands.
¹⁵ Center for Biological Imaging, Institute of Biophysics, Chinese Academy of Sciences, Beijing, China.

News

• September 4, 2024: Paper is online and available now. Enjoy!

• July 5, 2024: GenDSA is received by Med (Cell Press journal)!

• June 5, 2024: We released a portion of the 3D vascular and 3D non vascular datasets. (link here)

• March 27, 2024: We released our inference code. Paper/Project pages are coming soon. Please stay tuned!

Abstract

Digital subtraction angiography (DSA) devices have been commonly used in hundreds of different interventional procedures in various parts of the body, requiring multiple scans of the patient in a single procedure, which was high radiation damage to doctors and patients. Inspired by generative artificial intelligence techniques, this study proposed a large-scale pretrained multi-frame generative model-based real-time and low-dose DSA imaging system (GenDSA). Suitable for most DSA scanning protocols, GenDSA could reduce the DSA frame rate (i.e., radiation dose) to 1/3 and generates video that was virtually identical to clinically available protocols. GenDSA was pre-trained, fine-tuned and tested on ten million of images from 35 hospitals. Objective quantitative metrics (PSNR=36.83, SSIM=0.911, generated times=0.07s/frame) demonstrated that the GenDSA’s performance surpassed that of state-of-the-art algorithms in the field of image frame generation. Subjective ratings and statistical results from five doctors showed that the generated videos reached a comparable level to the full-sampled videos, both in terms of overall quality (4.905 vs 4.935) and lesion assessment (4.825 vs 4.860), which fully demonstrated the potential of GenDSA for clinical applications.

Environment Setups

• python 3.8
• cudatoolkit 11.2.1
• cudnn 8.1.0.77
• See 'requirements_Ours.txt' for Python libraries required

conda create -n GenDSA python=3.8
conda activate GenDSA
conda install cudatoolkit=11.2.1 cudnn=8.1.0.77
pip install torch==1.9.0+cu111 torchvision==0.10.0+cu111 -f https://download.pytorch.org/whl/torch_stable.html
# cd /xx/xx/GenDSA
pip install -r GenDSA_env.txt

Model Checkpoints

Download the zip of model checkpoints (key:mqfd5s), decompress and put all pkl files into ../GenDSA/weights/checkpoints.

Our Dataset and Inference Cases

We released a portion of the 3D vascular and non vascular datasets, including the results of our model inference. (Google Drive)

Inference Demo

Run the following commands to generate single/multi-frame interpolation:

• Single-frame interpolation

python Simple_Interpolator.py \
--model_path ./weights/checkpoints/3D-vas-Inf1.pkl \
--frame1 ./demo_images/DSA_a.png \
--frame2 ./demo_images/DSA_b.png \
--inter_frames 1

• Two-frame interpolation

python Simple_Interpolator.py \
--model_path ./weights/checkpoints/3D-vas-Inf2.pkl \
--frame1 ./demo_images/DSA_a.png \
--frame2 ./demo_images/DSA_b.png \
--inter_frames 2

• Three-frame interpolation

python Simple_Interpolator.py \
--model_path ./weights/checkpoints/3D-vas-Inf3.pkl \
--frame1 ./demo_images/DSA_a.png \
--frame2 ./demo_images/DSA_b.png \
--inter_frames 3

You can also use other checkpoints to generate 1~3 frame interpolation for your 2D/3D Head/Abdomen/Thorax/Pelvic/Periph images. Please promote and cite our work if you find it helpful. Enjoy!