Bug fix in original google-research implementation

[gulnazaki]

Hey there,

I've seen that a significant bug regarding the data_normalizer has been recently fixed in the original implementation in case you haven't checked it yet. I see it exists here too, since you ported the code.

google-research/google-research@b09ac83

[lucidrains]

@gulnazaki haha, I ported over their Jax code, so it should be fine :) that's their new tensorflow implementation

[lucidrains]

@gulnazaki thanks for letting me know!

btw, new follow-up paper for Performer! https://arxiv.org/abs/2012.11346

tldr: sorta-gradient checkpointing along the sequence dimension

[gulnazaki]

Oh yes, I am sorry I only had a quick look at it. I understand they fixed the tf implementation to match the one in jax.

Cool paper also, now the sky is the limit 😄