Vector Field Transformer

[MIDL 2022, MIA 2023] Domain Generalization for Retinal Vessel Segmentation via Hessian-based Vector Field

---

• The paper is available here

Introduction

Blessed by the vast amount of data, the learning-based methods have achieved remarkable performance in almost all tasks in computer vision and medical image analysis. Although these deep models can simulate highly nonlinear mapping functions, they are not robust with regard to domain shift of input data. This is one of the major concerns that impedes the large scale deployment of deep models in medical images since they have inherent variation in data distribution due to lack of imaging standardization. Therefore, a domain generalization (DG) method is needed to alleviate this problem. In this work, our main contributions are in three folds:

• data augmentation: A full-resolution variational auto-encoder (f-VAE) network that generates synthetic latent images
• domain alignment: A Hessian-based vector field that serves as an aligned image space that delineates the morphology of vessels
• model architecture: A novel paralleled transformer blocks that helps to learn local features in different scales

The overall pipeline of the work is shown as following:

Data augmentation

We implement a full resolution variational auto-encoder (f-VAE) in which the latent space is set to have the same width and height with the input image. The architecture is illustrated by the image below:

The supervision is provided by the binary vessel map. Since there is no direct constraint on the latent representation, we observe that each time we re-train the f-VAE will result in latent images with different styles while the anatomical structure remains unchanged. The augment-2, augment-3 and augment-4 are the three synthesized versions while augment-1 is the CLAHE version (intensty inversed) of the raw input.

The code for the synthetic model is accessible augment code
The trained model is available at augment model

Domain alignment

Please cite our work:

  @inproceedings{hu2021domain,
  title={Domain Generalization for Retinal Vessel Segmentation with Vector Field Transformer},
  author={Hu, Dewei and Li, Hao and Liu, Han and Oguz, Ipek},
  booktitle={Medical Imaging with Deep Learning},
  year={2021}
}