rgb info in the generator is not collected in the correct way?

[XavierXiao]

Hi! Thanks for the implementation, it is great! One possible issue I noticed is that the rgb images in each generator block is not collected correctly. If I understand correctly, in this line, the rgbs should collect rgb rather than layer_rgb?

[lucidrains]

@XavierXiao oh yes 🤦‍♂️ thank you!

[XavierXiao]

Great! Thanks! BTW, I see that you upload some codes for upsampler, however it looks like it is a simple Unet architecture with ResNet blocks. Do you plan to build the unet_upsampler with gigaGAN blocks recently?

[lucidrains]

@XavierXiao i'm kind of confused by what they used for upsampling

they have no architectural diagram, and only said it was a traditional unet, so that's what i'm going to start off with, paired with the GigaGAN discriminator. i was going to modify the unet to also output the rgb, like the style/gigagan generator

[lucidrains]

@XavierXiao open to pointers and suggestions, if you have any insights

[XavierXiao]

Thanks! Yeah I agree the info in the paper is insufficient especially for the upsampler part. But my two cents on the potential architecture:

1. Since in table A.2, the configuration of upsampler model has mapping network, w dimension, etc, I think definitely there is latent variable in the upsampler, similar to the base model. So the mapping network with concatenation of noise and global text embedding as input, as well as the adaptive filter with style modulation should be there. However, in the main text, it mentioned "residual blocks", so I guess it is more like a resnet block, but with conv layers replaced by adaptive kernel conv.
2. Since for text-to-image upsampler, the multi-scale loss is also True, there should be multi-scale outputs (i.e., toRGB layers). It makes no sense to have toRGB layers on resolutions lower than the input, so I guess you can have toRGB layers on the last few Unet blocks where the resolution is larger or equal to the input resolution.

[lucidrains]

@XavierXiao yes what you say makes sense Xavier! will build it exactly how you said!

[lucidrains]

@XavierXiao b22ecfc let me know if we are on the same page after this commit

[XavierXiao]

Great! Looks good to me! Only inconsistency to paper that I noticed is that the paper mentioned (in 3.4) the Unet is asymmetric (i.e., 3 down blocks and 6 up blocks for a 8x up-sampler), and the skip connection of the unet is, of course, only at matched resolutions of up and down blocks. Here your Unet seems to be symmetric. Not sure if it matters though.