Repository navigation
cakebench v0.1.0
Kernel benchmark modelled on CAKE (arXiv 2608.12629): a candidate kernel (hand-written or LLM-generated) must be correct against an fp32 oracle on every shape and input distribution before it earns a score; the score is the geometric-mean speedup over a torch baseline.
- 5 kernel families, 21 shapes in the full GPU portfolio: kmeans_assign, knn_search, tinygemm, kda_decode, kda_prefill
- Example hand-written torch kernels, all correct on one shared A100: kmeans_assign 1.69x, knn_search 1.13x, tinygemm 1.00x, kda_decode 1.03x, kda_prefill 1.06x (suite geomean 1.16x). These exercise the harness; no results for LLM-written GPU kernels are reported.
- Caveats: the Cake compiler is not public (this benches kernels, not the compiler); baselines are torch; paper numbers are B200, ours a shared A100 with noisy timings.
Video
cakebench_explainer.mp4 (attached; also in video/).
Install / run
pip install -e .[test]
cakebench list
cakebench suite --device cuda --fullHugging Face card with the recorded scores: https://huggingface.co/datasets/Bobliu/cakebench (no model weights; this is a benchmark harness).