Engine Rewrite

[anerli]

We are rewriting the entire inference engine in order to:

• Optimize decode and prefill speeds for consumer hardware on specific model architectures by writing our own kernels
• Enable dynamic memory scaling (scale memory up/down based on number of sessions and context length instead of fixed buffer)
• Improve batching performance and shared KV usage while maintaining maximum single-session performance
• Open doors to future engine-specific features and optimizations (for example: expert streaming)

This discussion is a space to give any feedback or ideas on the design as the new engine is developed.

The engine will heavily leverage TileLang in order to implement model-specific performant kernels once that can be automatically tuned and compiled for various backends.

The backends we plan to support include:

• CUDA (NVIDIA GPUs)
• ROCm/HIP (AMD GPUs)
• Metal (Apple Silicon)
• LLVM (CPU)

If you have inference-capable hardware of any kind - we will be looking for early testers to benchmark the new engine and work out any performance issues across different types of hardware.

[jcar01]

When will the new engine be released?