Reading Triton benchmarks: kernel time, full API cost, and inference performance

[wbavon]

A fast Triton kernel can still sit inside a slow API call. Before changing the kernel, make sure the measurement answers the question you actually care about.

This FlagOS community note is a reading guide to FlagGems #4812, an open investigation into pointwise dispatch overhead. It summarizes published evidence; it does not introduce a new benchmark or claim the same behavior on every current build.

The original report documents an A100-PCIE-40GB, Triton 3.2.0 and PyTorch 2.6.0+cu124 setup. It compares device execution with eager API timing and investigates repeated specialization, cache-key construction and launch metadata work. Later comments discuss other environments and explicitly separate their measurement protocols. Keep those environments separate when interpreting the results.

Three measurements are useful:

Question	What to measure	What the result can tell you
Is the device kernel expensive?	Kernel durations and activity on a device profiler timeline	Device execution and the kernels actually launched
Is the operator API expensive?	A warmed, synchronized full call, with inputs prepared outside the timed region	The cost of the complete invocation under that protocol, including host preparation and synchronization
Does it improve inference?	A controlled serving workload, reporting throughput, TTFT and TPOT	Whether the change helps the workload users run

Several traps make these comparisons misleading:

1. Asynchronous launches. A CPU timer around an unsynchronized call can measure submission rather than completion. State where synchronization occurs and whether you time one call or amortize a batch.
2. Events are not automatically kernel-only time. Events around a high-level API can span device idle time while the host prepares more work. Use the device timeline to identify actual kernel execution. Avoid adding overlapping host and device intervals into a synthetic total.
3. Graph replay changes the experiment. CUDA Graph measurements can help isolate repeated launch overhead, but a graph-replay improvement does not establish the same result for eager execution. Benchmark the execution mode you intend to deploy.
4. Cold and warm paths differ. Record compilation, autotuning and first-call behavior separately from steady-state timing. Profiling can perturb the workload; use unprofiled runs for the performance comparison and traces for diagnosis.
5. Correctness includes metadata and behavior. Keep shape, dtype, strides, scalar arguments, output semantics, device and software revisions in the reproducer. Validate against an appropriate reference before comparing speed.

For the caching direction discussed in #4812, a useful review question is whether the cache stores reusable structural work or accidentally retains invocation-specific state. Shape/stride changes, dtype promotion, alignment, compilation options, autotuning, streams and graph capture need appropriate treatment. A faster path also needs evidence that the intended kernel actually used it.

Here is a compact template for comparing notes:

Operation / model:
Shape, dtype, strides and scalar arguments:
Device, driver, framework and compiler revisions:
Correctness reference and tolerance:
Cold or warmed? Eager or graph replay?
Timer, synchronization points and repetitions:
Device-kernel result:
Full-API result:
Serving result, if measured:
Public reproducer / trace:

Which distinction has mattered most in your own work: device execution, host dispatch or framework integration? Examples from different hardware are welcome. Please keep reproductions, proposed fixes and code reviews on the relevant GitHub issue so others can reproduce and build on them.

For ongoing cross-project discussion, FlagOS Community on Slack starts in #community, with optional #kernels-and-compilers and #models-and-serving channels. English and Chinese are welcome. The note and its source are public; joining is optional.