ci: run integration + acceptance tests, stabilize a flaky perf gate

[ethan-scitix]

Type

• chore — CI, tooling, dependencies, config

Summary

CI ran only tests/unit, so the entire integration and acceptance layer went unverified on PRs. Two bugs shipped through that gap and had to be caught by hand during the 0.7.0 release (#55): a resume fixture that predated the version gate (#37), and a perf test still calling the pre-#7 Dataset.select().

• Add tests/integration + tests/acceptance to the matrix job, deselecting a new benchmark marker.
• The benchmark marker exists because tests/acceptance mixes two kinds of test. Nine are deterministic checks of the regression-detection helper. One (test_benchmark_scenarios) is a wall-clock throughput gate whose thresholds are calibrated on a dedicated box — long_output holds only 1.10× headroom there (0.771 measured vs 0.70 required), and the coverage tracer skews the very latency it measures. CI gains the nine; the gate stays a local tool that a plain pytest still runs.
• Make test_dependency_loading_overhead robust. It compares two ~20 ms hydrations as a ratio, so timing each arm once fed any transient stall straight into the metric. Each arm is now hydrated 5× and compared on its minimum — contention only ever adds time, never removes it.
• Enforce the gate in the release flow (.claude/commands/sieval-release.md). Since CI deselects benchmark, the release becomes the only place those thresholds get checked, so the step is now explicit. It also fixes a latent defect: the gates already ran there, but inside the batch the doc says to run in parallel — a wall-clock assertion sharing a box with ruff, ty, and a 2723-test pytest run measures that load, not the code, which is the very contention that produced the 876% reading. They are now held back with -m "not benchmark" and run alone afterwards, writing benchmark_summary.json to gitignored outputs/ so each release records a performance baseline.
• Document both markers in tests/README.md, including why a benchmark failure means "retry on an idle machine" rather than "regression".

Related Issues

Refs #55, #37, #7.

Test Plan

Automated

• Lint/format clean (ruff check && ruff format --check)
• Type check clean (ty check on both changed test modules: "All checks passed")
• The exact new CI command passes locally: 2723 passed, 1 deselected
• tests/performance: 66 passed, 5 deselected
• Plain local pytest tests/acceptance still collects all 10 (marker does not hide it locally)
• pytest -m benchmark selects exactly the 1 gate
• markdownlint clean on tests/README.md and .claude/commands/sieval-release.md
• Both new release-flow commands run as documented: the batch reports 2723 passed, 1 deselected; the serial gate reports 1 passed, 2794 deselected and writes the artifact without dirtying the tree

Manual

Flakiness fix verified empirically, not just reasoned about.

• Idle: 27.5% before → 25.1% after (same measurement, so the bound still means what it meant).
• Under the same concurrent-suite load that originally produced 876%: six consecutive runs now land in an 18–37% band, all passing.
• Estimator comparison in one process, 8 paired trials: single-shot spread 7–107% vs min-of-5 spread 2–16% — a ~7× reduction in spread, which is the property that stops the 300% bound being breached by noise.

Measured acceptance headroom on a dedicated box, the basis for excluding the gate from CI:

scenario	metric	threshold	measured	headroom
long_output	efficiency	0.70	0.771	1.10×
single_turn	efficiency	0.70	0.936	1.34×
high_concurrency	efficiency	0.65	0.859	1.32×
multi_iteration	efficiency	0.60	0.895	1.49×
framework_overhead	sps	500	1630	3.26×
resume_90pct	elapsed	5.0 s	0.199 s	27.9×

Checklist

Required (all PRs)

• PR title follows conventional format (type(scope): description)
• No internal paths, credentials, or personal info in committed files
• AI-generated code has AI-Generated Code - <model> (<provider>) in module docstring — n/a, no new modules
• No new upper-layer dependencies added to core/
• Deleted code verified — no remaining call sites depend on it (the removed timer_dep/timer_nodep locals were test-local)

Notes for reviewers

• CI wall-clock cost: the new command takes ~70s locally vs ~40s for tests/unit alone. No new install groups needed — everything tests/integration and tests/acceptance import (anyio, datasets, orjson, loguru in default; psutil, pytest in test) is already covered by INSTALL_GROUPS.
• addopts is deliberately left at -m "not stress", so benchmark tests keep running on a plain local pytest. Only the CI invocation deselects them.

🤖 Generated with Claude Code