Question: Scaling down number of random features depending on number of heads?

[Parskatt]

The theory for the paper gives a result which gives some guarantees for nb_features = O(dim*log(dim)).
When using multiple heads, e.g. dim = 512, heads = 8, you would get a lower dimensionality per head, is it then reasonable to scale the dimension of nb_features = O((dim/heads)*log(dim/heads)) ? Or is the variance too high when the number of features gets too low? Do you have any intuition for this, cause I'm feeling a bit unsure.

[lucidrains]

@Parskatt I think it makes sense for this to be dim / heads. For a standard 1024 dim and 8 heads, 128 * log(128) ~= 256

[lucidrains]

@Parskatt I was told this hyperparameter is pretty critical to good performance

[lucidrains]

@Parskatt Let us know what you find in your experiments!

[Parskatt]

I will continue my experiments and let you know later :)