Skip to content

Add Go benchmarks matching the Python benchmark suite (currently 0 Go benchmarks exist) #71

@mrjf

Description

@mrjf

Summary

The repo has 5 Python benchmark files but zero Go benchmarks, so the migration cannot demonstrate the performance win that motivates rewriting in Go. Step 4 of the Python-to-Go Migration program (#3) calls for "run the same benchmark script against the Go implementation and record the result" — that step is currently unimplemented across the entire module.

Evidence

$ find . -name '*_test.go' -exec grep -l 'func Benchmark' {} \;   # → 0 results
$ grep -c 'func Benchmark' $(find internal cmd -name '*.go')      # → 0

The Python side has:

File Coverage
scripts/benchmark_manifest_ops.py manifest collision / sync_remove_files / cleanup_empty_parents
tests/benchmarks/test_perf_benchmarks.py core perf paths
tests/benchmarks/test_audit_benchmarks.py security audit
tests/benchmarks/test_git_and_compiler_benchmarks.py git ops + compilation
tests/benchmarks/test_security_and_resolver_benchmarks.py security gate + dep resolver

For each internal/ package the autoloop claims to have migrated (839 entries in migration-status.json with status `migrated`), there is no corresponding BenchmarkX to prove the Go version is faster, slower, or roughly the same as the Python original.

Why this matters

  1. The migration's stated win is performance. Without matched benchmarks, the claim is unverifiable.
  2. Regressions go undetected. Future autoloop iterations could ship a Go implementation that is slower than the Python it replaces, and nothing would catch it.
  3. `migration-status.json` is currently the only progress signal, and it overcounts dramatically (claims 1084%, see Python-to-Go Migration #3). A benchmark table with real before/after numbers would be a far more credible progress signal.

Concrete request

Add Go testing.B benchmarks under each migrated internal/<pkg>/ directory, matching the hot paths of the corresponding Python file:

internal/marketplace/builder/builder_bench_test.go    ← matches builder.py paths in test_perf_benchmarks.py
internal/install/install_bench_test.go                ← matches benchmark_manifest_ops.py
internal/security/gate/gate_bench_test.go             ← matches test_security_and_resolver_benchmarks.py
internal/deps/apmresolver/apmresolver_bench_test.go   ← matches resolver paths
... etc.

Each Go benchmark should:

  • Use the same input shapes as the Python equivalent (e.g. the 50 packages × 5 files synthetic set from benchmark_manifest_ops.py).
  • Be runnable via go test -bench=. -benchmem ./internal/....
  • Have its result recorded next to the Python baseline in benchmarks/migration-status.json (or a sibling file), so each migrated module has a python_ns_per_op / go_ns_per_op pair.

Suggested follow-up

A small harness script (scripts/compare_benchmarks.sh) that runs both sides and emits a markdown table would let the progress site (#20) show a real perf comparison instead of a misleading line-count percentage.

Relationship to #3

This is the missing other half of step 2 + step 4 of the Python-to-Go Migration program. The directive in #3 names benchmarks as a per-iteration deliverable but the loop has never actually produced any on the Go side — only on the Python side.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions