MSAT

Source code for the paper "A Medical Semantic-Assisted Transformer for Radiographic Report Generation"

Figure: A overview of the proposed framework

Requirements

• Python 3
• CUDA 11
• tqdm
• easydict
• psutil
• ftfy
• regex
• tqdm
• PyTorch=1.7.1
• torchvision

Data preparation

1. Extract clip features: Download the images and our preprocessed annoatation.json file.

python tools/extract_clip_feature.py --annotation mimic_cxr/annotation.json --save_path ./data/feature/mimic_clip16_att_512

2. Convert reports to tokens and save it to data/mimic folder using the following script. Or download from here

python tools/build_vocab.py --annotation mimic_cxr/annotation.json --save_path data/mimic --radgraph data/mimic/MIMIC-CXR_graphs.json

3. Download metric package from here and unzip it into MSAT folder.

Training

Train MSAT model

python main.py --folder experiments/V1

Train MSAT model using reinforcement learning

python main.py --folder experiments/V1_rl --resume experiments/V1/snapshot/{best_model}.pth

Acknowledgements

Thanks the contribution of image-captioning, self-critical.pytorch and awesome PyTorch team.