[Feature Request]: Explore NVIDIA/TransformerEngine for speed/efficiency

[0xdevalias]

Is there an existing issue for this?

• I have searched the existing issues and checked the recent builds/commits

What would your feature do ?

• https://github.com/NVIDIA/TransformerEngine

  • A library for accelerating Transformer models on NVIDIA GPUs, including using 8-bit floating point (FP8) precision on Hopper GPUs, to provide better performance with lower memory utilization in both training and inference.

Proposed workflow

N/A

Additional information

Crossposted on:

• Explore NVIDIA/TransformerEngine for speed/efficiency huggingface/diffusers#1288

Other issues related to potential performance improvements:

• Stable Diffusion Meta AITemplate with >= 200% performance increase #1625
• [Feature Request]: Explore potential StableDiffusion speed benefits from implementing kernl (Up to 12X faster GPU inference) #4096
• [Feature Request]: colossalai integration #4606

[78Alpha]

Looks like the performance boost would only go to 40 series and above, as it relies on FP8 on the tensor cores. Even then, looks like it was locked out in the repo as all 40 series owners are reporting that it isn't currently working for them. And double for Windows.

Still, couldn't hurt for the boost but would also need an option for --no-FP8