Pix2Pix (GAN)😆👋- Diffusion based models, to see how to train a custom dataset look at https://github.com/openai/improved-diffusion
https://datasets.simula.no/hyper-kvasir/
We hold out 4971 unlabeled images and 200 polyps that we use in step 2 and 4 to compute FID.
-
We train our diffusion model on a large part of the unlabeled dataset in Hyperkvasir.
-
Generate samples and pick the model with best FID(Cosine 500K - 25.66).
-
Fine tune the model on 800 polyp images from Kvasir-SEG.
-
Pick the model with best FID score(0.3 dropout 26K - 80.55).
-
Compare our generated polyp images against Kvasir-SEG polyp images that are downsampled to 128x128 and center cropped.
Our best model generated polyps | Real polyp images from Kvasir-SEG |
---|---|
![]() |
![]() |
Link to all our generated images from 026000 model: https://www.dropbox.com/sh/3iz0rirrn9k22f3/AABZn4EA7Lb3BW4ocb8ect2qa?dl=0
To interpolate between two images using your DDPM can you use the following flags in the guided-diffusion folder. Make sure you use the same flags as the model you trained, for example same amount attention_resolutions etc. --rescale_timesteps needs to be True. Interpolation should create 99 images a example of interpolation between two polyps is shown below with t intervals of [0,125,250,375,500,625,750,875,999]. Images from left to right are [reconstructed src1, λ=0.1,λ=0.2,...,λ=0.9,reconstructed src2]
MODEL_FLAGS="--attention_resolutions 32,16,8 --image_size 128 --num_channels 128 --num_heads 4 --num_res_blocks 2 --learn_sigma True --resblock_updown True --use_fp16 True --use_scale_shift_norm True"
DIFFUSION_FLAGS="--diffusion_steps 1000 --noise_schedule linear --rescale_learned_sigmas False --rescale_timesteps True --interpolate True --src1 images/img1.jpg --src2 images/img2.jpg"
python scripts/image_sample.py --model_path your/model/path.pt $MODEL_FLAGS $DIFFUSION_FLAGS
Dataset | IoU | mIoU | DSC | Precision | Recall |
---|---|---|---|---|---|
Baseline | 0.762 | 0.732 | 0.840 | 0.871 | 0.821 |
+800 | 0.785 | 0.766 | 0.857 | 0.913 | 0.826 |
Dataset | IoU | mIoU | DSC | Precision | Recall |
---|---|---|---|---|---|
Baseline | 0.351 | 0.470 | 0.408 | 0.583 | 0.709 |
+800 | 0.396 | 0.492 | 0.451 | 0.604 | 0.727 |
Dataset | IoU | mIoU | DSC | Precision | Recall |
---|---|---|---|---|---|
Baseline | 0.642 | 0.628 | 0.735 | 0.831 | 0.720 |
+800 | 0.654 | 0.66 | 0.738 | 0.869 | 0.733 |
Generated polyp images | Real polyp images from Kvasir-SEG |
---|---|
![]() ![]() |
![]() ![]() |
![]() ![]() |
![]() ![]() |
![]() ![]() |
![]() ![]() |