T5 Static Cache

[huseinzol05]

This enable to use torch.compile for T5 generation to enable faster generation.

Compiled static cache able to achieve 905.746397513013 tokens / sec while non-compiled got 259.57796367162115 tokens / sec on Small T5 60M parameters.

1. Notebook for non-compiled, https://github.com/mesolitica/t5-static-cache/blob/main/t5-static-cache-non-compile.ipynb
2. Notebook for compiled, https://github.com/mesolitica/t5-static-cache/blob/main/t5-static-cache.ipynb

Still work in progress

1. Current forked only work to use static cache, need to follow caching steps as Llama.
2. There are so many conditions need to fulfill first.
3. Only worked on Pytorch 2.4.0.dev20240508+cu121 version, not yet released as stable for custom function reduce-overhead torch compile.

[amyeroberts]

cc @ArthurZucker

[ArthurZucker]

Once this is ready feel free to ping either @fxmarty or @gante