Add: benchmark BGEMM on host build graph - #1771
Merged
ChaoZheng109 merged 1 commit intoAug 11, 2026
Merged
Conversation
Port Case0 with the same parameters, signatures, and golden as the tensormap-and-ringbuffer benchmark. Reuse the existing orchestration and incore kernel sources directly so both runtimes exercise an identical compute graph.
📝 WalkthroughWalkthroughAdded a host-build-graph BGEMM benchmark for A2A3. The test generates tensor inputs, computes a golden result across ChangesHost-build-graph BGEMM benchmark
Estimated code review effort: 2 (Simple) | ~15 minutes Possibly related issues
Possibly related PRs
Sequence Diagram(s)sequenceDiagram
participant TestBenchmarkBgemmHostBuildGraph
participant SceneTestCase
participant BGEMMOrchestration
participant InCoreGemmAddKernels
TestBenchmarkBgemmHostBuildGraph->>SceneTestCase: run_module
SceneTestCase->>BGEMMOrchestration: execute orchestration metadata
BGEMMOrchestration->>InCoreGemmAddKernels: process GEMM and ADD operations
TestBenchmarkBgemmHostBuildGraph->>TestBenchmarkBgemmHostBuildGraph: compute golden output across grid_k
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (1)
examples/a2a3/host_build_graph/benchmark_bgemm/test_benchmark_bgemm.py (1)
25-59: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winDeclare benchmark configuration as class-level state.
Ruff reports RUF012 because
CALLABLEandCASESare mutable class attributes. AddClassVarannotations to declare that these are sharedSceneTestCasedefinitions. Use a targeted suppression only if the framework requires these attributes to be instance-mutable.Proposed fix
+from typing import ClassVar + ... - CALLABLE = { + CALLABLE: ClassVar = { ... - CASES = [ + CASES: ClassVar = [🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@examples/a2a3/host_build_graph/benchmark_bgemm/test_benchmark_bgemm.py` around lines 25 - 59, Annotate the mutable CALLABLE and CASES class attributes with ClassVar, preserving their existing SceneTestCase definitions and values. Add a targeted suppression only if the framework requires instance mutation of either attribute.Source: Linters/SAST tools
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@examples/a2a3/host_build_graph/benchmark_bgemm/test_benchmark_bgemm.py`:
- Around line 25-59: Annotate the mutable CALLABLE and CASES class attributes
with ClassVar, preserving their existing SceneTestCase definitions and values.
Add a targeted suppression only if the framework requires instance mutation of
either attribute.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 6e2e8eb3-e455-4ce9-977b-9314b568a5c9
📒 Files selected for processing (3)
examples/a2a3/host_build_graph/benchmark_bgemm/README.mdexamples/a2a3/host_build_graph/benchmark_bgemm/test_benchmark_bgemm.pyexamples/a2a3/tensormap_and_ringbuffer/benchmark_bgemm/kernels/orchestration/bgemm_orch.cpp
ChaoZheng109
approved these changes
Aug 11, 2026
This was referenced Aug 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
host_build_graphcounterpart ofbenchmark_bgemm::Case0.implementation unchanged.
so both runtimes exercise the same 500-GEMM + 500-ADD compute graph without
duplicating kernel code.
CasINOUTbecause it is a zero-initialized read-before-writeaccumulator across
grid_kiterations.This is one workload-sized part of #1727. The remaining workloads and the
benchmark tooling/skill changes are intentionally left for separate PRs.
Performance methodology
Measured on a2a3 hardware through
task-submit, using one locked NPU 7 atsimpler base revision
0fc77f21:TMR -> HBG, thenHBG -> TMR)Paired-batch geometric changes (95% confidence interval):
-23.53%to-17.07%). All 8 pairs werefaster with HBG; individual improvements were 12.43%-24.53%.
+42.28%to+85.26%). All 8 pairs wereslower with HBG; the median paired increase was 58.84%.
-14.87%to-6.88%).+61.84%to+83.20%).The mean host-total delta is +9.545 ms. Its phase attribution is:
An HBG info-level attribution run (kept separate because logging perturbs the
benchmark) measured
args_malloc_copyat 8.8 ms andprebuilt_runtime_arenaat 6.3 ms, withtotal_init_runtime_implat 15.2 ms.For comparison, steady TMR
bind.prebuiltwas about 0.005 ms. The HBGprebuilt-runtime arena explains most of the bind increase.
Therefore this migration shows no device-side regression: HBG device wall is
consistently about 20% faster for Case0. Steady host latency is higher, mainly
from bind/runtime-arena preparation and validation. The workload keeps the
same graph instead of masking that runtime cost; future #1727 benchmark tooling
should report host total and device wall as separate guard metrics.
Performance job:
task_20260810_192420_238922620377(NPU 7, exit 0).Testing
python test_benchmark_bgemm.py -p a2a3simtask-submiton NPU 4:task_20260810_191738_183514219385(exit 0)task-submiton NPU 7:task_20260810_192420_238922620377(exit 0)