Any plans for a diffusers version?

[tonyf]

Hey guys, this paper looks great. Really excited to see the full training code. Was curious-- do you had any plans to make a diffusers port?

[haoosz]

Yeah. We will make diffusers version after all the work is done. Thank you.

[tonyf]

Amazing! Looking forward to seeing it. Just curious-- is there an expected timeline for the diffusers version? Debating whether to implement it myself

[haoosz]

Sorry, but I am occupied by the following work and might not work on the diffuser version right now. I will work on the diffuser version in August. If it is too late for you, I am very glad you can implement by yourself. Thank you!

[okaris]

I am working on this.

[garychan22]

I have finished the diffusers version but simply feeding the reference image to the frozen unet and doing the otsu is low-speed, which is weird. hahaha

[okaris]

@garychan22 I've also recently finished it and have been working on getting the hyperparams to fit my needs. otsu itself is the bottleneck, the point of having it is to escape the need of preprocessing, but if you are already doing that a manually supplied mask could also help and speed it up. Other than that this repo is not taking advantage of higher performance attention processors, which you can't use for the attention calculations where you need to extract the scores. But it's possible to use xformers or pytorch's scaled_dot_product_attention for faster calculations.

Were you able to replicate the results exactly like the samples here?

[okaris]

Also if you would like to submit a PR, here is my issue: huggingface/diffusers#3719

[garychan22]

@garychan22 I've also recently finished it and have been working on getting the hyperparams to fit my needs. otsu itself is the bottleneck, the point of having it is to escape the need of preprocessing, but if you are already doing that a manually supplied mask could also help and speed it up. Other than that this repo is not taking advantage of higher performance attention processors, which you can't use for the attention calculations where you need to extract the scores. But it's possible to use xformers or pytorch's scaled_dot_product_attention for faster calculations.

Were you able to replicate the results exactly like the samples here?

Thanks for the useful tips here! For now, I have not replicated the similar results as this repo and I will keep working on this.

Moreover, I have been training my own blip-diffusion, finding that better results to dreambooth can be achieved within one-minute fine-tuning, which is awesome. Hope to replicate the results as shown in the paper and release the pre-trained model to the hub soon.