Inference

The simplest way to serve AI/ML models in production
https://github.com/basetenlabs/truss

Lucebox optimization hub: hand-tuned LLM inference, built for specific consumer hardware
https://github.com/Luce-Org/lucebox-hub

HiSparse: Turbocharging Sparse Attention with Hierarchical Memory
https://www.lmsys.org/blog/2026-04-10-sglang-hisparse/

DFlash: Block Diffusion for Flash Speculative Decoding
https://github.com/z-lab/dflash

FlashInfer: Kernel Library for LLM Serving
https://github.com/flashinfer-ai/flashinfer

TriAttention: Efficient Long Reasoning with Trigonometric KV Compression
https://weianmao.github.io/tri-attention-project-page/

Modalities, a PyTorch-native framework for distributed and reproducible foundation model training
https://github.com/Modalities/modalities

Online Softmax Demystified: From PyTorch to FlashAttention in Triton
https://isztld.com/posts/online-softmax.html

How we made attention 45% faster on AWS Trainium
https://charleshong3.github.io/blog/autocomp_trainium_attention.html

MegaQwen: Hitting the Architectural Ceiling of Cooperative CUDA Megakernels
https://elliotarledge.com/blog/megaqwen

Flux 2 image generation model pure C inference
https://github.com/antirez/flux2.c

Challenges and Research Directions for Large Language Model Inference Hardware
https://www.arxiv.org/abs/2601.05047

NexaSDK lets you build the smartest and fastest on-device AI with minimum energy
https://github.com/NexaAI/nexa-sdk

Light Image Video Generation Inference Framework
https://github.com/ModelTC/LightX2V

Fast and Accurate Causal Parallel Decoding using Jacobi Forcing
https://arxiv.org/abs/2512.14681

The Anatomy of a Triton Attention Kernel
https://arxiv.org/abs/2511.11581

Continuous batching
https://huggingface.co/blog/continuous_batching

A lightweight vLLM implementation built from scratch.
https://github.com/GeeeekExplorer/nano-vllm

Achieve state of the art inference performance with modern accelerators on Kubernetes
https://github.com/llm-d/llm-d

Coral NPU is a hardware accelerator for ML inferencing
https://github.com/google-coral/coralnpu

dInfer: An Efficient Inference Framework for Diffusion Language Models
https://github.com/inclusionAI/dInfer

Domain specific architectures for AI inference
https://fleetwood.dev/posts/domain-specific-architectures

A TensorRT Toolbox for Optimized Large Language Model Inference
https://github.com/NVIDIA/TensorRT-LLM

Fast LLM Inference From Scratch
https://andrewkchan.dev/posts/yalm.html#section-3.5