Collaborate on Implementation?

[calclavia]

I was planning on implementing this on Pytorch as well and started a repo https://github.com/calclavia/Performer-Pytorch
Implemented the kernel so far. If the author(s) of this repo wants to collaborate, would be happy to contribute.

[lucidrains]

@calclavia hey Henry! Yea, I want to build this! Do you have a plan for cutting down memory usage for the autoregressive case? Custom cuda like what epfl did?

[lucidrains]

@calclavia feel free to email me, happy to chat more :)

[lucidrains]

@calclavia I think I'm just going to go it alone and finish it tonight, I'm too tired to collaborate lol

sorry!

[lucidrains]

@calclavia we should keep in touch and exchange our findings testing this in the wild

[ketyi]

Go go guys! ;)

[lucidrains]

@calclavia committed and deployed my first draft! i'll keep digging into the jax code tomorrow to see what other essential details i'm missing

[lucidrains]

@calclavia turns out the CUDA code will definitely be necessary, since I got word that the number of random features is encouraged to be 256 or above. Let me know if you'd like to collaborate on this :)

[lucidrains]

@calclavia nevermind, a new paper has found a way to do this causal case without CUDA code. I'm going to wrap up the project today

[calclavia]

Sorry for the late response - was busier than I thought this week. Looking forward to your wrap up. :)