Some minor questions regarding the design

[XavierXiao]

Thanks for updating the implementation frequently! The codebase looks much nicer than when I first looked at it. I took a closer look on the details and would like to ask some questions on specific design choice. Of course I understand that not every specific design comes with a reason, but I would like to know if you have reference/intuition on these things:

1. It seems like in the main generator, self attention comes before cross attention, while in the upsampler, cross attention comes before self attention.
2. This line has a residual connection, but it is already done inside the Transformer class. Same here. Is this something new in transformer literature?
3. Here in the discriminator, the residual is added after attention block. Does it make more sense to add it right after two conv blocks, since the attention block has its own residual connection?
4. Very tiny issue. The definition in this is unused.

[lucidrains]

@XavierXiao thank you for the code review! this is more helpful than you could know 🙏

could you see if the latest commit addresses all the issues?

[XavierXiao]

Yes! Looks good to me. I will let you know if I further spot something.