Inference

Achieve state of the art inference performance with modern accelerators on Kubernetes
https://github.com/llm-d/llm-d

Coral NPU is a hardware accelerator for ML inferencing
https://github.com/google-coral/coralnpu

dInfer: An Efficient Inference Framework for Diffusion Language Models
https://github.com/inclusionAI/dInfer

Domain specific architectures for AI inference
https://fleetwood.dev/posts/domain-specific-architectures

A TensorRT Toolbox for Optimized Large Language Model Inference
https://github.com/NVIDIA/TensorRT-LLM

Fast LLM Inference From Scratch
https://andrewkchan.dev/posts/yalm.html#section-3.5