should y_embedding in CaptionEmbedder class be fixed as the embedding of null text when training PixArt-alpha?

[Gus-Guo]

should y_embedding in CaptionEmbedder class be fixed as the embedding of null text when training PixArt-alpha from scratch? instead of randomly initialized.

[lawrence-cj]

yes

[Gus-Guo]

@lawrence-cj Got it. But why inference codes in pixart-alpha use model.y_embedder.y_embedding as the null text embedding instead of using real null text embedding produced by T5 encoder, which is adopted by PixArt-sigma inference code? I check the model.y_embedder.y_embedding and found it is not the same as the real null text embedding produced by T5 encoder.

[lawrence-cj]

It's a historical problem. Later in sigma this problem is solve, but it won't influence much. BTW, the inference code in alpha is kinda outdated. The app.py and so on can pass the negative prompt into T5.

[Gus-Guo]

@lawrence-cj Got it. which inference code do you recommend if I want to infer model using pixart-alpha ckpt, e.g. the provided FID732 ckpt?