validation: extract deterministic slice runner

[yisding]

Problem

validation/runner.py mixed four different orchestration domains with subprocess execution, environment policy, reliability artifact policy, and deterministic pytest-slice assembly. The 1,639-line module had nine direct complexity findings; run_validation_slice alone owned selector policy, command construction, filesystem layout, redaction, reliability evaluation, artifact projection, and final reporting.

Changes

• extract deterministic pytest lanes into validation/_slice_runner.py
• define each slice once through an immutable spec containing its selector, pytest arguments, and optional artifact behavior
• model resolved slice paths and evaluated outcomes explicitly
• move subprocess/test-root helpers into a downward-only runner support leaf
• move provider environment/secret policy into an independent environment leaf
• centralize shared reliability loading and budget failures for slice and latency lanes
• keep validation.runner as the existing facade while reducing it to 1,376 lines
• reduce direct runner complexity findings from nine to six; extracted modules pass complexity checks with ignores disabled

Validation

• 115 passed: complete validation guard plus focused slice contracts
• 55 passed: runner, latency, live, baseline, and slice regression set
• ruff format --check .
• ruff check . --no-cache
• changed-module mypy
• direct C901/PLR checks with per-file ignores disabled
• Import Linter: 3 contracts kept
• pre-commit on all changed files

References

• pytest exit-code contract: https://docs.pytest.org/en/stable/reference/exit-codes.html
• Python subprocess guidance: https://docs.python.org/3/library/subprocess.html

Summary by CodeRabbit

• New Features

  • Added deterministic validation slices with configurable test paths and commands.
  • Validation results now include standardized test outcomes, tool exit codes, and generated artifacts.
  • Added reliability budget checks that can flag invalid or out-of-budget validation results.
  • Added optional capture and reporting for WebRTC statistics.

• Bug Fixes

  • Validation failures now consistently report nonzero test exits.
  • Runtime secrets are redacted from validation output and generated reports.
  • Corrupted reliability artifacts are reported as structured validation failures.

[coderabbitai]

Warning

Review limit reached

@yisding, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 15 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 982bf044-11d7-40aa-b310-5ca4d6dcd44b

📥 Commits

Reviewing files that changed from the base of the PR and between b944fc1 and aef36d9.

📒 Files selected for processing (5)

• src/easycat/validation/_reliability_policy.py
• src/easycat/validation/_runner_support.py
• src/easycat/validation/_slice_runner.py
• src/easycat/validation/runner.py
• tests/validation/test_slice_runner.py

📝 Walkthrough

Walkthrough

Validation execution is modularized into shared runner utilities, runtime-secret handling, reliability policy helpers, and a dedicated pytest-backed slice runner. Existing validation lanes delegate to these modules, with tests covering nonzero pytest exits and corrupted reliability artifacts.

Changes

Validation runner modularization

Layer / File(s)	Summary
Shared validation foundations src/easycat/validation/_environment.py, src/easycat/validation/_lane_harness.py, src/easycat/validation/_reliability_policy.py, src/easycat/validation/_runner_support.py	Provider secrets, subprocess execution, pytest configuration, path resolution, exit normalization, and reliability failures are centralized.
Validation slice orchestration src/easycat/validation/_slice_runner.py	Adds slice specifications, isolated artifacts, pytest execution, redaction, reliability evaluation, artifact aggregation, and lane result finalization.
Existing lane integration src/easycat/validation/runner.py	Existing validation entrypoints import shared helpers and delegate slice, command, secret, and reliability handling.
Slice result validation tests/validation/test_slice_runner.py	Tests normalize nonzero pytest exits and report corrupted reliability samples as structured failures.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Validator
  participant SliceRunner
  participant Pytest
  participant ReliabilityPolicy
  participant LaneHarness
  Validator->>SliceRunner: run validation slice
  SliceRunner->>Pytest: execute configured pytest command
  Pytest-->>SliceRunner: return exit code and output
  SliceRunner->>ReliabilityPolicy: load and evaluate reliability samples
  ReliabilityPolicy-->>SliceRunner: return policy result
  SliceRunner->>LaneHarness: finalize validation result and artifacts

Loading

Poem

I’m a rabbit with secrets tucked safely away,
While pytest hops through each validation day.
Slices bloom, artifacts neatly align,
Reliability watches each boundary line.
With redacted reports and failures made clear,
The runner grows sturdier, year after year.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 14.29% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the main change: extracting the deterministic slice runner from validation/runner.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch agent/validation-slice-runner

---

_{Comment @coderabbitai help to get the list of available commands.}

[charliecreates]

Reviewed the deterministic slice extraction and its integration with the latency, live, and release validation paths. No blocking correctness or compatibility issues found.

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/easycat/validation/_runner_support.py`:
- Around line 13-22: Replace the loose CommandRunner Callable alias with a typed
Protocol defining __call__ using the expected command positional argument and
keyword-only env and cwd parameters, returning CommandResult. Preserve the
existing CommandResult contract and update any required typing imports so
injected runners are checked against this explicit signature.

In `@src/easycat/validation/_slice_runner.py`:
- Around line 259-297: Update the reliability-loading flow around
load_reliability_failure and _load_reliability so a successful validation
returns and reuses the already parsed samples instead of rereading and reparsing
reliability_path. Propagate the parsed samples through the caller, while
preserving the existing failure behavior and artifact construction via
build_reliability_artifact and reliability_budget_failure.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: c26c6fc3-8376-4854-962b-4339cfb70a12

📥 Commits

Reviewing files that changed from the base of the PR and between 1fbe83c and b944fc1.

📒 Files selected for processing (7)

• src/easycat/validation/_environment.py
• src/easycat/validation/_lane_harness.py
• src/easycat/validation/_reliability_policy.py
• src/easycat/validation/_runner_support.py
• src/easycat/validation/_slice_runner.py
• src/easycat/validation/runner.py
• tests/validation/test_slice_runner.py