[repository-quality] Repository Quality Improvement Report - Benchmark Infrastructure Completeness & Regression Prevention

[github-actions[bot]]

🎯 Repository Quality Improvement Report - Benchmark Infrastructure Completeness & Regression Prevention

Analysis Date: 2026-07-10
Focus Area: Benchmark Infrastructure Completeness & Regression Prevention
Strategy Type: Reuse
Custom Area: No — reuse of 2026-06-11/2026-06-16 runs (highest 3-high-priority count in last 10); random value 94 → ≥90 triggers reuse

Executive Summary

The benchmark suite has grown to 97 functions across 23 test files, but allocation instrumentation remains at 19% adoption (18/97 have b.ReportAllocs()). The 79 functions missing this call produce structurally incomplete benchmark data: make bench with -benchmem reports allocations only when the benchmark itself opts in via b.ReportAllocs(). The daily-cli-performance workflow's regression detection is therefore blind to allocation regressions in 81% of the codebase's benchmarks.

Since the June 16 reuse run, the compiler_performance_benchmark_test.go file was correctly instrumented (all 7 functions have b.ResetTimer() + b.ReportAllocs()), but the 16 remaining benchmark files show zero adoption. Four benchmarks still use the deprecated for i := 0; i < b.N or for range b.N pattern instead of Go 1.24+ b.Loop(), missing automatic warm-up iteration exclusion.

The daily-cli-performance.md workflow monitors only 5 named benchmarks (BenchmarkCompileSimpleWorkflow, BenchmarkCompileComplexWorkflow, BenchmarkCompileMCPWorkflow, BenchmarkCompileMemoryUsage, BenchmarkParseWorkflow), leaving 92 other benchmarks unmonitored for trend regression. A make bench-regression Makefile target that runs benchstat over cached baselines is referenced in documentation but does not exist; benchstat is also not installed.

Full Analysis Report

Focus Area: Benchmark Infrastructure Completeness & Regression Prevention

Current State Assessment

Metrics Collected:

Metric	Value	Status
Total benchmark functions	97	⚠️
Benchmarks with b.ReportAllocs()	18 (19%)	❌
Benchmarks with b.ResetTimer()	8 (8%)	❌
Benchmarks using old b.N loop	4	⚠️
Benchmarks using b.Loop()	93	✅
benchstat installed	No	❌
Benchmarks monitored by daily CI workflow	5/97 (5%)	❌
Benchmark files with zero b.ReportAllocs()	16/23	❌

Files with most missing b.ReportAllocs() (top 6 by gap):

File	Missing / Total
pkg/workflow/expressions_benchmark_test.go	11/11
pkg/cli/logs_benchmark_test.go	10/10
pkg/stringutil/sanitize_test.go	8/8
pkg/workflow/processing_benchmark_test.go	7/7
pkg/workflow/compiler_benchmark_test.go	7/7
pkg/workflow/regex_benchmark_test.go	6/6

Files with old b.N loop pattern:

File	Location
pkg/stringutil/urls_test.go	line 130
pkg/workflow/js_comments_test.go	line 713
pkg/workflow/concurrency_validation_test.go	line 644
pkg/workflow/template_injection_validation_benchmark_test.go	line 28

Findings

Strengths

• compiler_performance_benchmark_test.go is fully instrumented with b.ResetTimer() + b.ReportAllocs() on all 7 functions.
• compile_security_benchmark_test.go has b.ReportAllocs() on all 7 functions.
• 93/97 benchmarks use the modern b.Loop() pattern.
• daily-cli-performance.md provides a regression baseline framework with repo-memory storage and Python trend analysis.
• make bench target uses -benchmem flag, which is a prerequisite for allocation measurement.

Areas for Improvement

• [HIGH] 79 benchmarks lack b.ReportAllocs(): -benchmem at the CLI level does not override the absence of b.ReportAllocs(); in the goccy/go-yaml-instrumented ecosystem, allocation counts are unreported without the in-benchmark call.
• [HIGH] 4 benchmarks use deprecated b.N iteration pattern instead of b.Loop(), missing the automatic warm-up exclusion that b.Loop() provides in Go 1.24+.
• [MEDIUM] 92 benchmarks are not tracked by the daily CI performance workflow; regressions in expressions, log parsing, YAML generation, and MCP-related code go undetected.
• [LOW] benchstat is documented in instructions and Makefile comments but not installed; make bench-compare outputs an advisory to use it but cannot run it.

Detailed Analysis

The -benchmem flag passed to go test enables allocation reporting at the test binary level, but individual benchmarks must call b.ReportAllocs() for their own allocation metrics to appear in output. Without it, allocs/op and B/op columns are suppressed per-benchmark even when -benchmem is passed. This means make bench produces meaningful allocation data only for the 18 instrumented benchmarks.

The highest-impact gap is in expressions_benchmark_test.go (11 benchmarks covering expression validation and parsing), logs_benchmark_test.go (10 benchmarks covering log parsing for all engines), and processing_benchmark_test.go (7 benchmarks covering tools/safe-outputs/permissions processing). These cover performance-critical paths where allocation regressions directly affect user-visible latency.

Setup-heavy benchmarks in compiler_benchmark_test.go create temporary directories and write files before b.Loop() but without b.ResetTimer(). The b.Loop() method in Go 1.24 does record a setup phase separately, but b.ResetTimer() before the loop is still the idiomatic way to exclude one-time setup allocation from the per-iteration reports.

---

🤖 Tasks for Copilot Agent

NOTE TO PLANNER AGENT: Split the following tasks into individual work items.

Improvement Tasks

Task 1: Add b.ReportAllocs() to All Benchmark Functions in expressions_benchmark_test.go and processing_benchmark_test.go

Priority: High
Estimated Effort: Small
Focus Area: Benchmark Infrastructure

Description: 18 benchmark functions in pkg/workflow/expressions_benchmark_test.go (11) and pkg/workflow/processing_benchmark_test.go (7) have no b.ReportAllocs() call. Add it as the first statement in each benchmark function body, before any setup code and before b.Loop(). This makes allocation data visible in make bench output and enables regression detection.

Acceptance Criteria:

• All 11 benchmarks in expressions_benchmark_test.go have b.ReportAllocs() as the first call.
• All 7 benchmarks in processing_benchmark_test.go have b.ReportAllocs() as the first call.
• go test -bench=. -benchmem -run=^$ ./pkg/workflow/ shows allocs/op and B/op columns for all affected benchmarks.
• No existing test or benchmark is broken.

Code Region: pkg/workflow/expressions_benchmark_test.go, pkg/workflow/processing_benchmark_test.go

In `pkg/workflow/expressions_benchmark_test.go` and `pkg/workflow/processing_benchmark_test.go`, add `b.ReportAllocs()` as the first statement inside each `func Benchmark...(b *testing.B)` body, immediately before any setup code. Do not modify the iteration loop or any other logic. Verify with `go test -bench=. -benchmem -run=^$ ./pkg/workflow/` that each affected benchmark now shows non-zero `allocs/op` and `B/op` columns.

---

Task 2: Add b.ReportAllocs() to All Benchmark Functions in logs_benchmark_test.go, compiler_benchmark_test.go, and regex_benchmark_test.go

Priority: High
Estimated Effort: Small
Focus Area: Benchmark Infrastructure

Description: 23 additional benchmark functions across pkg/cli/logs_benchmark_test.go (10), pkg/workflow/compiler_benchmark_test.go (7), and pkg/workflow/regex_benchmark_test.go (6) lack b.ReportAllocs(). For the compiler benchmarks that also have expensive setup (temp dir creation + file writes), add b.ResetTimer() immediately before b.Loop() to exclude setup time from the reported ns/op.

Acceptance Criteria:

• All 10 benchmarks in logs_benchmark_test.go have b.ReportAllocs().
• All 7 benchmarks in compiler_benchmark_test.go have b.ReportAllocs() and b.ResetTimer() before b.Loop().
• All 6 benchmarks in regex_benchmark_test.go have b.ReportAllocs().
• go test -bench=. -benchmem -run=^$ ./pkg/cli/ ./pkg/workflow/ confirms allocation data for all affected functions.

Code Region: pkg/cli/logs_benchmark_test.go, pkg/workflow/compiler_benchmark_test.go, pkg/workflow/regex_benchmark_test.go

In `pkg/cli/logs_benchmark_test.go`, `pkg/workflow/compiler_benchmark_test.go`, and `pkg/workflow/regex_benchmark_test.go`:
1. Add `b.ReportAllocs()` as the first statement in every `func Benchmark...` body.
2. For each benchmark in `compiler_benchmark_test.go` that creates a temp directory or writes files before the loop, also add `b.ResetTimer()` immediately before `for b.Loop() {` to exclude setup I/O from timing.
Verify with `go test -bench=. -benchmem -run=^$ ./pkg/cli/ ./pkg/workflow/`.

---

Task 3: Migrate Four b.N Loop Benchmarks to b.Loop() Pattern

Priority: Medium
Estimated Effort: Small
Focus Area: Benchmark Infrastructure

Description: Four benchmarks use the deprecated b.N iteration pattern instead of the Go 1.24 b.Loop() method. b.Loop() automatically excludes JIT warm-up iterations from timing and produces more stable results. Migrate all four.

File locations:

• pkg/stringutil/urls_test.go:130 — for range b.N {
• pkg/workflow/js_comments_test.go:713 — for range b.N {
• pkg/workflow/concurrency_validation_test.go:644 — for range b.N {
• pkg/workflow/template_injection_validation_benchmark_test.go:28 — for i := 0; i < b.N; i++ {

Acceptance Criteria:

• All four locations use for b.Loop() { with no loop variable.
• No loop-internal index variable referencing i remains (restructure if needed).
• go test -bench=. -benchmem -run=^$ ./pkg/stringutil/ ./pkg/workflow/ passes without errors.

Code Region: pkg/stringutil/urls_test.go, pkg/workflow/js_comments_test.go, pkg/workflow/concurrency_validation_test.go, pkg/workflow/template_injection_validation_benchmark_test.go

In the following four files, replace each old-style `b.N` loop with `b.Loop()`:
- `pkg/stringutil/urls_test.go` line 130: change `for range b.N {` to `for b.Loop() {`
- `pkg/workflow/js_comments_test.go` line 713: change `for range b.N {` to `for b.Loop() {`
- `pkg/workflow/concurrency_validation_test.go` line 644: change `for range b.N {` to `for b.Loop() {`
- `pkg/workflow/template_injection_validation_benchmark_test.go` line 28: change `for i := 0; i < b.N; i++ {` to `for b.Loop() {` (remove `i` variable if unused inside the loop body)

Also add `b.ReportAllocs()` as the first statement in each of these benchmark functions if not already present.

Verify all benchmarks pass with: `go test -bench=. -benchmem -run=^$ ./pkg/stringutil/ ./pkg/workflow/`

---

Task 4: Add b.ReportAllocs() to Remaining Benchmark Files (sanitize_test.go, stringutil_test.go, identifiers_test.go, envutil_test.go, frontmatter_benchmark_test.go, and others)

Priority: Medium
Estimated Effort: Small
Focus Area: Benchmark Infrastructure

Description: Complete the b.ReportAllocs() sweep for the remaining benchmark files: pkg/stringutil/sanitize_test.go (8), pkg/stringutil/stringutil_test.go (6), pkg/stringutil/identifiers_test.go (2), pkg/stringutil/urls_test.go (1), pkg/envutil/envutil_test.go (4), pkg/parser/frontmatter_benchmark_test.go (4), pkg/repoutil/repoutil_test.go (2), pkg/cli/commands_utils_test.go (2), pkg/workflow/mcp_benchmark_test.go (1), pkg/workflow/js_test.go (2), pkg/workflow/expression_parser_comprehensive_test.go (2), pkg/workflow/concurrency_validation_test.go (3), pkg/workflow/yaml_test.go (benchmarks if any).

Acceptance Criteria:

• grep -rn "func Benchmark" pkg/ --include="*_test.go" | wc -l equals grep -rn "b.ReportAllocs()" pkg/ --include="*_test.go" | wc -l (100% coverage).
• go test -bench=. -benchmem -run=^$ ./pkg/... shows allocs/op for all benchmarks.
• No test is broken.

Code Region: pkg/stringutil/, pkg/envutil/, pkg/parser/, pkg/repoutil/, pkg/cli/commands_utils_test.go, pkg/workflow/mcp_benchmark_test.go, pkg/workflow/js_test.go, pkg/workflow/expression_parser_comprehensive_test.go, pkg/workflow/concurrency_validation_test.go

Perform a sweep of all remaining benchmark files and add `b.ReportAllocs()` as the first statement in each `func Benchmark...(b *testing.B)` function that does not already have it. 

Target files (all missing b.ReportAllocs()):
- pkg/stringutil/sanitize_test.go — 8 benchmarks
- pkg/stringutil/stringutil_test.go — 6 benchmarks
- pkg/stringutil/identifiers_test.go — 2 benchmarks
- pkg/envutil/envutil_test.go — 4 benchmarks
- pkg/parser/frontmatter_benchmark_test.go — 4 benchmarks
- pkg/repoutil/repoutil_test.go — 2 benchmarks
- pkg/cli/commands_utils_test.go — 2 benchmarks
- pkg/workflow/mcp_benchmark_test.go — 1 benchmark
- pkg/workflow/js_test.go — 2 benchmarks
- pkg/workflow/expression_parser_comprehensive_test.go — 2 benchmarks
- pkg/workflow/concurrency_validation_test.go — 3 benchmarks (also migrate b.N to b.Loop per Task 3)

After changes, verify: `go test -bench=. -benchmem -run=^$ ./pkg/...` — confirm all benchmarks report allocs/op.

---

📊 Historical Context

Previous Focus Areas

Date	Focus Area	Type	Custom	Key Outcomes
2026-06-11	Benchmark Infrastructure Completeness & Regression Prevention	Custom	Y	93 benchmark functions with CI bench job but no automated regression detection; benchstat not installed
2026-06-16	Benchmark Infrastructure Completeness & Regression Prevention	Reuse	N	78/93 benchmarks (84%) still missing b.ReportAllocs(); benchstat still not installed
2026-07-08	Deprecated API Alias Proliferation & Zombie Callsite Cleanup	Custom	Y	3 GetGeminiAPITarget callsites; zombie no-op validation; stale TODO
2026-07-07	Environment Variable Access Fragmentation	Custom	Y	45 bare nolint suppressions; 6 env vars duplicated; envutil incomplete
2026-07-06	Context-Blind Sleep Gap	Custom	Y	4 functions with 7 time.Sleep calls; worst case 60 min uncancellable

---

🎯 Recommendations

Immediate Actions (This Week)

1. Add b.ReportAllocs() to expressions_benchmark_test.go (11) and processing_benchmark_test.go (7) — Priority: High
2. Migrate 4 b.N loops to b.Loop() — Priority: Medium

Short-term Actions (This Month)

1. Complete b.ReportAllocs() sweep across all 23 benchmark files — Priority: Medium
2. Add b.ResetTimer() to setup-heavy benchmarks in compiler_benchmark_test.go — Priority: Medium

Long-term Actions (This Quarter)

1. Install benchstat in CI and wire a make bench-regression target that compares current run against cached baseline — Priority: Low

---

📈 Success Metrics

• b.ReportAllocs() coverage: 18/97 (19%) → 97/97 (100%)
• b.Loop() adoption: 93/97 (96%) → 97/97 (100%)
• Benchmarks monitored by CI: 5/97 (5%) → 50+/97 (>50%)

---

Next Steps

1. Review and prioritise the tasks above
2. Assign tasks to Copilot coding agent via planner agent
3. Track progress on improvement items
4. Re-evaluate this focus area in 2 weeks

References:

• §29096103386

---

Generated by Repository Quality Improvement Agent

Generated by ⚡ Repository Quality Improvement Agent · 175.4 AIC · ⌖ 6.21 AIC · ⊞ 6.6K · ◷

• expires on Jul 11, 2026, 5:40 AM UTC-08:00

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-07-11T13:40:44.688Z.

Closed by Workflow