Skip to content

No Probe Cycle Benchmark or Perf Regression Test #32

@henry0816191

Description

@henry0816191

Problem

The two-frequency hot/cold probing strategy issues 1,600-2,000 HEAD requests per 30-minute cycle against the community-operated isocpp.org server. No benchmark or performance regression test exists for this cycle. The timing characteristics — total elapsed time, request distribution, concurrent connection count, error rates — exist only as runtime observations in production logs. There is no baseline measurement, no regression gate, and no way to detect whether a shift in cycle timing comes from a code change versus a server-side behavioral change. The concurrency cap (HTTP_CONCURRENCY=20) and overrun cooldown (POLL_OVERRUN_COOLDOWN_SECONDS=300) are tuned by feel rather than measurement.

Acceptance Criteria

  • Add a benchmarks/ directory with a pytest-benchmark (or standalone timing harness) that exercises the probe cycle against a mock HTTP server
  • Benchmark measures and records: total cycle wall-clock time, request count, peak concurrent connections, and per-request latency distribution (p50/p95/p99)
  • Establish a baseline JSON artifact (checked in or emitted as CI artifact) capturing current performance envelope with default settings
  • Add a CI job or pytest marker that runs the benchmark on PRs and flags regressions exceeding a configurable threshold (e.g., >20% wall-clock increase)
  • Document the performance envelope in a docs/probe-performance.md or README section: what's normal, what constitutes degradation, what thresholds warrant investigation

Implementation Notes

Use pytest-benchmark or a lightweight custom harness. The mock server should simulate isocpp.org's response patterns (200 with Last-Modified, 404, connection delays). Focus on ISOProber.probe_all() as the entry point. The HTTP_CONCURRENCY and POLL_INTERVAL_SECONDS settings should be parameterizable in the benchmark. This work also partially addresses Compound-5 (Unobservable Probe Volume) by creating a quantitative baseline that survives contributor handoff.

References

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions