Feat ref cross attn unet [2306.08276]

[pnsuau]

This PR adds a new option for unet mha diffusion generator. This generator is based on https://arxiv.org/abs/2306.08276 work.

Usage :

python3 train.py  --G_netG unet_mha_ref_attn --data_dataset_mode self_supervised_labeled_mask_online_ref --model_type palette --alg_palette_conditioning " "  --alg_palette_cond_image_creation y_t

Notes:

• Reference generator for diffusion requires a dataset with reference images
• Reference generator cannot (yet) be combined with other types of conditioning, e.g. --alg_palette_conditioning or --alg_palette_cond_image_creation y_t

Typical results on VITON:

[Bycob]

"ref" stands for "reference image" right? Maybe add comments to point that out? And explain what a reference image is

[dingkwang]

Hi I am new to this project. I processed the VTON dataset following this guide. https://www.joligen.com/doc/tutorial_viton.html but it doesn't produce the reference image and the condition.txt. I think I just need to create them manually from the original viton dataset.

[beniz]

Take a look here instead: https://www.joligen.com/doc/training.html#ddpm-training-with-image-reference-conditioning

[dingkwang]

For the --data_dataset_mode, you should use self_supervised_labeled_mask_ref, instead of self_supervised_labeled_mask_online_ref

[beniz]

you should use self_supervised_labeled_mask_ref, instead of self_supervised_labeled_mask_online_ref

The online dataloader samples crops around the bboxes, which is useful on some datasets, including VITON, see #568 (comment) and #576 (comment)