🌟 New model addition: FNet

[cccntu]

🌟 New model addition: FNet

FNet is a highly efficient Transformer-like encoder architecture, wherein the self-attention sublayers have been wholly replaced by standard, unparameterized Fourier Transforms.

I would like to help adding this!

Open source status

• the model implementation is available: https://github.com/google-research/google-research/tree/master/f_net
• the model weights are available: https://github.com/google-research/google-research/tree/master/f_net
• who are the authors: (@ilyaeck @santiontanon) (Not sure, googled the authors' name + github, sorry if it's incorrect)

[NielsRogge]

Somebody is already working on this, see #12335

[cccntu]

Thanks @NielsRogge , weird that I didn't see it when I searched.

[gchhablani]

@cccntu I believe what you want for the JAX/Flax community week is a Flax model. It seems unlikely that I will finish the PR in the next week. Maybe, you can start working on the Flax model parallely?

Or, we can discuss over slack and then try to finish both.

@patil-suraj @patrickvonplaten wdyt? Is it easier to go from PyTorch to Flax? Or it doesn't matter at all? In case PT is needed, I am willing to spend my time next week on this and try to finish it.

[cccntu]

@gchhablani Yes! I would love to add the Flax part.
@patil-suraj @patrickvonplaten I have a few questions before I proceed:

• There is no license in the original repo, should I email the authors for permission for code and weights?
• How much of the original model code should I modify, other than wrapping it in huggingface/transformers classes?
  Should we refactor it for better weight alignment with pytorch code e.t.c?

Thanks!

[gchhablani]

Great @cccntu! Let's discuss over Slack.