Skip to content

v0.1.0

Latest

Choose a tag to compare

@Bobliuuu Bobliuuu released this 13 Nov 08:18

cakebench v0.1.0

Kernel benchmark modelled on CAKE (arXiv 2608.12629): a candidate kernel (hand-written or LLM-generated) must be correct against an fp32 oracle on every shape and input distribution before it earns a score; the score is the geometric-mean speedup over a torch baseline.

  • 5 kernel families, 21 shapes in the full GPU portfolio: kmeans_assign, knn_search, tinygemm, kda_decode, kda_prefill
  • Example hand-written torch kernels, all correct on one shared A100: kmeans_assign 1.69x, knn_search 1.13x, tinygemm 1.00x, kda_decode 1.03x, kda_prefill 1.06x (suite geomean 1.16x). These exercise the harness; no results for LLM-written GPU kernels are reported.
  • Caveats: the Cake compiler is not public (this benches kernels, not the compiler); baselines are torch; paper numbers are B200, ours a shared A100 with noisy timings.

Video

cakebench_explainer.mp4 (attached; also in video/).

Install / run

pip install -e .[test]
cakebench list
cakebench suite --device cuda --full

Hugging Face card with the recorded scores: https://huggingface.co/datasets/Bobliu/cakebench (no model weights; this is a benchmark harness).