Skip to content

bench(ops): add matmul micro-benchmarks (dense vs sparse operand)#868

Merged
FBumann merged 1 commit into
masterfrom
bench/kvl-matmul-idiom
Jul 24, 2026
Merged

bench(ops): add matmul micro-benchmarks (dense vs sparse operand)#868
FBumann merged 1 commit into
masterfrom
bench/kvl-matmul-idiom

Conversation

@FBumann

@FBumann FBumann commented Jul 24, 2026

Copy link
Copy Markdown
Collaborator

We are not yet tracking sparse matmul in the ops. To optimize it, im adding a baseline benchmark here.

Sorry for the churn. I wanted to keep it in the kvl benchmark, but It regressed with matmul, so decided to keep the existing one.

Note

The following content was generated by AI (Claude Code), prompted and reviewed by @FBumann.

The suite has no benchmark exercising @/dot: the kvl_cycles pattern — written to show what a sparse-aware matmul would win (#748) — builds its constraint via the expanded (flow * C).sum("branch"), which bypasses __matmul__ entirely, and ops.py has no contraction op. A sparse-aware kernel (#867) would land invisible to CI.

An earlier revision of this PR switched kvl_cycles itself to the @ operator, but that surfaced as a flat +31% memory "regression" across all severities: on the current kernel, Variable.__matmul__ goes through to_linexpr() (ones coefficients) plus a full-size multiply, allocating an extra (time × branch × cycle) float64 temporary that the expansion avoids. Real information about the @ idiom's current cost, but the wrong place to change measurement history — so instead this adds a dedicated matmul op group to ops.py, following the suite's own op-vs-pattern philosophy ("an op benchmark says which path got heavier"):

New ops carry no CodSpeed history, so no existing baseline moves and nothing is flagged. kvl_cycles is untouched. Landing this before #867 gives that PR a proper baseline: its CodSpeed run should show expr_matmul_sparse dropping sharply while expr_matmul_dense stays put.

🤖 Generated with Claude Code

@FBumann
FBumann marked this pull request as ready for review July 24, 2026 10:12
@FBumann
FBumann requested a review from FabianHofmann July 24, 2026 10:12
@codspeed-hq

codspeed-hq Bot commented Jul 24, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 173 untouched benchmarks
🆕 2 new benchmarks
⏩ 173 skipped benchmarks1

Performance Changes

Mode Benchmark BASE HEAD Efficiency
🆕 Memory test_op[expr_matmul_dense] N/A 39.7 MB N/A
🆕 Memory test_op[expr_matmul_sparse] N/A 39.7 MB N/A

Comparing bench/kvl-matmul-idiom (d38c0a4) with master (2afe5d0)

Open in CodSpeed

Footnotes

  1. 173 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

@FBumann
FBumann force-pushed the bench/kvl-matmul-idiom branch from f4b4f21 to a7232c0 Compare July 24, 2026 10:19
@FBumann FBumann changed the title bench(kvl_cycles): measure the @ operator itself, not its expansion bench(ops): add matmul micro-benchmarks (dense vs sparse operand) Jul 24, 2026
@FBumann
FBumann marked this pull request as draft July 24, 2026 10:23
@FBumann
FBumann force-pushed the bench/kvl-matmul-idiom branch from a7232c0 to 6ecf331 Compare July 24, 2026 10:23
The suite had no benchmark exercising @/dot: the kvl_cycles pattern
builds its constraint via the expanded (flow * C).sum('branch'), which
bypasses __matmul__ entirely, and ops.py had no contraction op. A
sparse-aware matmul kernel (#748/#867) would land invisible to CI.

Add a matmul op group contracting the profile's large dim against a
(1000 x 100) constant: expr_matmul_dense (fully nonzero — stays on the
dense kernel) and expr_matmul_sparse (incidence-shaped, ~3 nonzeros per
column — the case a sparse-aware kernel collapses). New ops carry no
CodSpeed history, so this changes no existing baseline; kvl_cycles is
left untouched.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@FBumann
FBumann force-pushed the bench/kvl-matmul-idiom branch from 6ecf331 to d38c0a4 Compare July 24, 2026 10:27
@FBumann
FBumann marked this pull request as ready for review July 24, 2026 10:29
@FBumann

FBumann commented Jul 24, 2026

Copy link
Copy Markdown
Collaborator Author

I will merge this as soon as tests finish

@FBumann
FBumann merged commit f268bff into master Jul 24, 2026
22 of 23 checks passed
@FBumann
FBumann deleted the bench/kvl-matmul-idiom branch July 24, 2026 12:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant