Do you have any plans to export a pytorch version?

[MultiPath]

Hi, I am not too familiar with tensorflow...
If there are no such plans currently, do you have quick pointers to:

1. the GANsformer model, especially where and how you deal with the latents (based on your paper, you split the latents?)
2. what kind of optimizers are you using? and how do you implemented it? Is it similar to what we did in NLP (warmup, etc);
3. did you ever tried using the standard feedforward after your duplex attention layer instead of 3x3? Did it still work?

Thanks again for your kind attention!
Best,

[ygjwd12345]

Pytorch version, please.

[dorarad]

Hi, thank you for the interest!
Yea I think it will be great idea to have a pytorch version and hope to work on it after the neurips deadline in a few weeks.

For the specific questions:

1. All the GANsformer model details can be found in the network.py file. The parts the relate to initializing the latents the attention and transformer layer here and the latent initialization happens at the loss file here based on the model inputs format that is specified here.
2. The optimization is specified in the network.py file and follows exactly StyleGAN2 code exactly.
3. You mean avoiding convolution completely? Yes I did do such experiments. Got a bit lower quality but generally works ok-ish. I think the advantage of the bipartite transformer structure that we use here vs. standard transformers is that we have linear attention from a few global latent variables to the image, while a simple transformer has (quadratic) self-attention between all pixels to all pixels (or all spatial features to themselves) (which also prevents standard transformers from scaling for high-res images). As a result, our transformer can model regional or global aspects of the region, but it is more limited in its ability to modulate direct interactions between specific pairs of pixels/features, which is why retaining the convolution here is useful to complement this aspect.

[MultiPath]

Thanks very much for your kind attention and looking forward to the pytorch version!

Will check the codes you referred!

Thanks for the explanation of the convolution parts! Good to know it also works.

The last question i forgot to ask: how is the inference speed of your model compared to StyleGAN2 in a similar size?
Will transformer affect the speed? In my view, duplex attention still has a lot of computation due to the image size.

Thanks again

[dorarad]

Sure thing happy to help!

StyleGAN2 takes 0.45s to produces a 4-images batch, while Simplex and Duplex Transformers’ speeds are 0.48s and 0.53s per batch respectively, so not a significant overhead!

The EM process in the Duplex attention is amortized over the multiple layers of the generator as it is gradually increasing the features resolution, such that each layer adds one iteration only, allowing for efficient computation that avoids a large overhead compared to Simplex attention.

[yzcv]

Hi, @dorarad

May I ask have you developed the PyTorch version of gansformer? I look forward to your PyTorch version and the checkpoint.

[dorarad]

Hi, yea it's in progress and I hope to release it soon (over the next couple of weeks)!

[yzcv]

Hi, yea it's in progress and I hope to release it soon (over the next couple of weeks)!

Thanks so much for your prompt reply. I will keep an eye on this repo and thanks for your work again. @dorarad

[MultiPath]

Hi any news on the Pytorch version? Thanks

[dorarad]

Apologies for the delay, was in a summer internship! It's top on my todo list and will update as soon as it's added!

[sb-nw]

Hi! Is there an estimated release date for the PyTorch version? I know you mentioned that you are working on it recently, but it would be great to have an estimate! Thanks! - Stunning work btw.

[dorarad]

Thanks so much for the kind words! :-) The reason I can't give yet an exact time estimation is that there are now the deadlines for the talk and camera ready for NeurIPS and I got a GANformer's follow-up work accepted that I'm keen to get public too so I plan to finish that asap and right afterwards get back to complete the PyTorch version!

[dorarad]

Hi Guys! Without further ado, happy to introduce the new pytorch implementation of the model!
It has matching interface to the original TF version. Will add readme shortly, please do let me know for any questions or suggestions!