how to conditional diffusion with two classes label

[Johnson-yue]

No description provided.

[anton-l]

Hi @Johnson-yue,
We don't have any examples for class-conditional diffusion just yet, but you already can train your own model for that!

UNet2DConditionModel's forward() function exposes an encoder_hidden_states parameter: https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/unet_2d_condition.py#L118
So you can encode the class labels in your preferred way (for example, using torch.nn.Embeddings and a couple of Linear layers) and pass them to the UNet during training and inference 🙂

[sean-halpin]

@anton-l Even though it is possible to provide the encoder_hidden_states while training, the DDPMPipeline has no mechanism to provide encoder_hidden_states during inference.