Skip to content

ci(eng): blocking unit + 11-cell smoke test suite (warns on external errors) - #28

Closed
changliu2 wants to merge 1 commit into
mainfrom
ci/eng-test-suite
Closed

ci(eng): blocking unit + 11-cell smoke test suite (warns on external errors)#28
changliu2 wants to merge 1 commit into
mainfrom
ci/eng-test-suite

Conversation

@changliu2

@changliu2 changliu2 commented May 7, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Summary

P0 engineering CI suite. Two-job workflow on every PR touching p2m/**, examples/**, prompts/**, pyproject.toml, scripts/{smoke_*,regression_*}.py:

  • tier1-unit — existing pytest suite (now also covers the new smoke classifier).
  • smoke (matrix × 11 cells) — light-budget end-to-end runs across a representative slice of examples/, summarized to the PR.

Replaces .github/workflows/regression.yml (the old Tier 4 regression stub is deferred to a separate engineer-owned follow-up PR — see "Deferred" below).

Contract

Failure mode Cell verdict PR effect
Clean exit + scores produced PASS ✅
RateLimitError / 429 / TPM EXTERNAL_RATE_LIMIT ⚠️ warn, non-blocking
5xx / ServiceResponseError EXTERNAL_5XX ⚠️ warn, non-blocking
ConnectionError / ReadTimeout / DNS / TLS EXTERNAL_NETWORK ⚠️ warn, non-blocking
Azure ContentFilterError / ResponsibleAIPolicyViolation EXTERNAL_CONTENT_FILTER ⚠️ warn, non-blocking
p2m/ frame in traceback P2M_BUG ❌ block
Exit 0 but no scores produced P2M_BUG ❌ block
Exit nonzero, no recognized pattern UNKNOWN ❌ block (conservative)

External patterns take precedence over p2m frames — transients bubbling through p2m/core/model_client.py retry wrappers stay non-blocking.

Matrix — 11 cells

Label Config
langgraph examples/travel_planner_langgraph/eval_config.yaml
neurosan examples/travel_planner_neurosan/eval_config.yaml
phoenix-multinode examples/phoenix_auto_trace/eval_config.yaml
phoenix-{openai,litellm,langchain,dspy,crewai} examples/phoenix_auto_trace/eval_*.yaml
pipes-{health,simulated,generated} examples/pipes/health_assistant*.yaml

Light budget per cell (scripts/smoke_cell.py:LIGHT_BUDGET): 3 prompts × 3 scenarios × max_turns 4. Each cell isolated by suite=smoke-\-\. Wall-clock ceiling ~12 min.

Files

File Status
.github/workflows/eng.yml new
.github/workflows/regression.yml deleted (tier1 moves into eng.yml)
scripts/smoke_cell.py new — wraps p2m run with isolated suite/run ids and light-budget overrides
scripts/smoke_classify.py new — log + result classifier; --summarize mode for PR summary table
tests/test_smoke_classify.py new — 9 cases covering every classification path
.gitignore +3 lines (*.local.md, **/copilot-instructions.local.md, **/AGENTS.local.md)

Verification

uv run pytest -q → 537 passed, 14 skipped (was 528 + 9 new classifier cases).

Merge prerequisites (this PR is DRAFT until both land)

  1. Concurrency configurability and fail-fast on fatal errors #25 — concurrency + fail-fast (judge concurrency knob required to fit smoke in budget).
  2. Fix A+B PR — rollout.py target_error abort fix. Without it, a single transient seed failure aborts the whole pipeline and the classifier sees P2M_BUG for what is actually a partial run, producing high false-positive blocks.

Phased rollout

  • Phase 1 (~1 week post-merge): add continue-on-error: true to the smoke matrix job so the engineer can observe noise and tune classifier patterns without blocking other PRs.
  • Phase 2: flip smoke to required (remove continue-on-error).

Deferred to a follow-up PR (P1, engineer-owned)

  • .github/workflows/science.yml — 6 canonical efficacy metrics + paired t-test gate, continue-on-error: true.
  • Replace scripts/regression_test.py placeholder with the real implementation.
  • Spec lives in the local copilot-instructions; scaffold for science.yml saved offline (not in this PR).

Engineer handoff checklist

  • Wait for Concurrency configurability and fail-fast on fatal errors #25 to merge.
  • Wait for Fix A+B PR to merge.
  • Rebase this branch onto main; convert from draft to ready.
  • Run a couple PRs with Phase 1 (continue-on-error: true) to gauge false-positive rate.
  • Flip to Phase 2 (required).
  • Open the P1 science PR per spec.

@changliu2
ci(eng): blocking unit + 11-cell smoke test suite (warns on external …
43a7051 
…errors)

Replaces .github/workflows/regression.yml with a focused engineering test
that runs every PR touching p2m, examples, prompts, or pyproject.toml:

  tier1-unit  -> existing pytest suite (now also covers smoke classifier)
    └── smoke (matrix, 11 cells, 8-min timeout each)
        └── smoke-summary (renders per-cell verdict table to PR summary)

Smoke matrix exercises end-to-end pipeline runs on a representative slice
of examples/ at light budget (3 prompts + 3 scenarios + 4 max_turns):

  langgraph, neurosan, phoenix-{multinode,openai,litellm,langchain,
  dspy,crewai}, pipes-{health,simulated,generated}.

Sized from observed large_summary.json wall-clocks; full matrix completes
in ~12 min wall-clock with all cells in parallel.

Smart blocking via scripts/smoke_classify.py:

  - Real p2m bugs (p2m frame in traceback, exit 0 with no scores) -> BLOCK.
  - External failures (rate limit, 5xx, network, content filter) -> WARN.
  - External patterns take precedence over p2m frames so transients
    bubbling through model_client retry wrappers stay non-blocking.

Each cell uses isolated suite/run ids (smoke-\-\) so
parallel cells never trample each other's artifacts.

The previous regression.yml ran a placeholder Tier 4 stub; that science
gate is deferred to a separate engineer-owned follow-up PR (science.yml +
real scripts/regression_test.py implementation).

Also hardens .gitignore with *.local.md to keep internal-only agent
instructions out of the repo by convention.

Tests: 537 passed, 14 skipped (was 528 + 9 new classifier cases).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
jakepresent added a commit that referenced this pull request May 7, 2026
@jakepresent
docs(verification): reground matrix notes to P0/P1 CI lane split
7a10222 
Chang's PR #28 lands the P0 engineering CI lane (11-cell smoke matrix,
blocks on P2M bugs, warns on external errors). Chang explicitly carved
out the P1 science lane in PR description and Teams chat:

  'science quality - did our PR regress on the science efficacy
  inadvertently - trickier to measure and require large runs to gain
  statistical significance/ensure not spurious'

This matrix + verify.py is the foundation for that follow-up PR. The
scenario rows (TP-*) will eventually be grounded in the 11-cell example
matrix Chang set up (travel_planner_*, phoenix_auto_trace/*, pipes/*).
jakepresent added a commit that referenced this pull request May 7, 2026
@jakepresent
feat(verification): reground scenario rows in Chang's P0 scope
8abd16c 
Replaces 12 placeholder TP-* rows (scraped from
tests/regression/risks/travel_planner_*.md) with 14 grounded
scenario rows from Chang's May 7 P0 scoping in Teams chat:

  'our P0 scenario will just have 2 agent scenarios - health
  assistant and travel planner - but to probe our system breadth
  (frameworks/endpoints) and scalability (100, 1k, 10k seeds, etc.)'

New scenario rows:
- 2 behavior rows (P2M-AGENT-HEALTH, P2M-AGENT-TRAVEL)
- 9 framework breadth rows (langgraph, neurosan,
  phoenix-multinode + 5 phoenix-* endpoint variants, simulated)
  - all derived from PR #28's example matrix
- 3 scalability rows (P2M-SCALE-100/1K/10K)

Each row has a 'source' field pointing to the canonical config
file or the chat clarification. Total rows 26 -> 28 (frameworks
unchanged at 14, scenarios go 12 -> 14).

Run state vs travel-planner-langgraph-v1/baseline:
  13 PASS / 1 FAIL / 14 NOT_IMPLEMENTED out of 28 rows.
This was referenced May 7, 2026
Closed
Closed
changliu2 added a commit that referenced this pull request May 18, 2026
@changliu2
ci(science): regression-gate workflow + 6 efficacy metrics + Holm-Bon…
0aca0d2 
…ferroni gate

Replaces the placeholder regression_test.py and the old regression.yml
workflow with the real implementation deferred from PR #28 (eng smoke
classifier) "P1, engineer-owned" follow-up.

What ships
----------
- scripts/regression_metrics.py: 6 canonical + 4 auxiliary efficacy
  metrics (signal_rate, policy_violation_rate, overrefusal_rate,
  judge_failure_rate, construct_coverage, separation_strength,
  discrimination_power, failure_variety, failure_mode_count,
  item_saturation), each returning per-seed arrays for paired tests.
- scripts/regression_decision.py: McNemar's exact one-sided test for
  per-seed binary metrics, paired-bootstrap helpers (placeholder p-value
  for v1 suite-level metrics), Holm-Bonferroni step-down over the 6
  canonical metrics, and a decide() orchestrator returning a JSON-safe
  report.
- scripts/regression_test.py: orchestrator that drives p2m run at
  baseline + treatment commits, computes metrics on both, applies the
  Holm-Bonferroni gate, and writes regression_report.{json,md}.
- .github/workflows/science.yml: Phase-1 advisory gate
  (continue-on-error: true). Triggers on p2m/, prompts/,
  tests/regression/, and scripts/regression_*.py changes. Label-driven
  seed budget (seeds:50/100/200/500). Baseline runs cached by composite
  key including base SHA, config hash, judge model, seed count, and
  script hashes.
- tests/test_regression_metrics.py + tests/test_regression_decision.py:
  31 unit tests covering all metric values, McNemar against known
  binomials, Holm step-down ordering, and the gate decision matrix
  (PASS / WARN / BLOCK across direction x effect x n_pairs).
- scripts/__init__.py: makes scripts a proper package so both
  python scripts/foo.py and python -m scripts.foo work.

Statistical design (post-rubber-duck)
-------------------------------------
- One-sided McNemar in the direction of the OBSERVED effect (fixes a
  bug where testing the improvement hypothesis yielded a high p-value
  precisely when there was a regression to detect).
- alpha = 0.01 per test; Holm-Bonferroni step-down across the 6
  canonical metrics. Auxiliary metrics are reported but advisory only.
- Per-seed binary metrics can BLOCK; suite-level metrics WARN-only in
  v1 (suite bootstrap is a placeholder; honest TODO documented).
- Per-metric direction map (higher_is_better / lower_is_better / None);
  policy_violation_rate direction is None by default since it depends
  on whether the target is benign-quality or red-team — caller can
  override via directions_override.
- MIN_N_FOR_GATE = 10 → fewer pairs returns TooFewSamples, gate WARNs.

Phase 1 rollout
---------------
- continue-on-error: true so the gate is advisory while we observe
  noise and tune the per-metric MDEs.
- Exit criterion documented in workflow comment: flip to required after
  10 PRs / 2 weeks at <5% noisy WARN/BLOCK rate.
- MDE thresholds in DEFAULT_MDE are v1 placeholders documented as
  "recalibrate after first 5 baseline runs from observed variance."

Known v1 limitations (intentional)
----------------------------------
- Suite-level p-value in compare_suite_level is a placeholder (returns
  0.5 if abs(mean_diff) < mde else 0.05) — the function is wired
  through honestly but suite metrics are advisory in v1, so the
  approximation is acceptable. Real paired-bootstrap on jointly
  resampled seed ids is a follow-up.
- Ground-truth assertion mode (Abby/Riccardo benchmark eval set) is NOT
  in this PR. Rubber-duck recommended waiting until the dataset lands;
  decide() is shaped to add a ground-truth comparator without
  restructuring.
- Default N=100 (5 mins per spec at concurrency=10). Power analysis is
  not yet pre-derived — script reports observed variance per metric
  and flags WARN if underpowered.

Tests
-----
- 31/31 new regression module tests pass in <1s.
- Full suite: 644 passed, 14 skipped, 4 pre-existing Windows-only
  failures (file lock + HTTP test) unrelated to this change. CI runs
  on Ubuntu so they don't apply.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@changliu2 changliu2 closed this May 22, 2026
AaronAspinwall123 added a commit that referenced this pull request Jul 29, 2026
@AaronAspinwall123 @changliu2
@tangym
Science regression gate (#263)
2cd5ae7 
* ci(science): regression-gate workflow + 6 efficacy metrics + Holm-Bonferroni gate

Replaces the placeholder regression_test.py and the old regression.yml
workflow with the real implementation deferred from PR #28 (eng smoke
classifier) "P1, engineer-owned" follow-up.

What ships
----------
- scripts/regression_metrics.py: 6 canonical + 4 auxiliary efficacy
  metrics (signal_rate, policy_violation_rate, overrefusal_rate,
  judge_failure_rate, construct_coverage, separation_strength,
  discrimination_power, failure_variety, failure_mode_count,
  item_saturation), each returning per-seed arrays for paired tests.
- scripts/regression_decision.py: McNemar's exact one-sided test for
  per-seed binary metrics, paired-bootstrap helpers (placeholder p-value
  for v1 suite-level metrics), Holm-Bonferroni step-down over the 6
  canonical metrics, and a decide() orchestrator returning a JSON-safe
  report.
- scripts/regression_test.py: orchestrator that drives p2m run at
  baseline + treatment commits, computes metrics on both, applies the
  Holm-Bonferroni gate, and writes regression_report.{json,md}.
- .github/workflows/science.yml: Phase-1 advisory gate
  (continue-on-error: true). Triggers on p2m/, prompts/,
  tests/regression/, and scripts/regression_*.py changes. Label-driven
  seed budget (seeds:50/100/200/500). Baseline runs cached by composite
  key including base SHA, config hash, judge model, seed count, and
  script hashes.
- tests/test_regression_metrics.py + tests/test_regression_decision.py:
  31 unit tests covering all metric values, McNemar against known
  binomials, Holm step-down ordering, and the gate decision matrix
  (PASS / WARN / BLOCK across direction x effect x n_pairs).
- scripts/__init__.py: makes scripts a proper package so both
  python scripts/foo.py and python -m scripts.foo work.

Statistical design (post-rubber-duck)
-------------------------------------
- One-sided McNemar in the direction of the OBSERVED effect (fixes a
  bug where testing the improvement hypothesis yielded a high p-value
  precisely when there was a regression to detect).
- alpha = 0.01 per test; Holm-Bonferroni step-down across the 6
  canonical metrics. Auxiliary metrics are reported but advisory only.
- Per-seed binary metrics can BLOCK; suite-level metrics WARN-only in
  v1 (suite bootstrap is a placeholder; honest TODO documented).
- Per-metric direction map (higher_is_better / lower_is_better / None);
  policy_violation_rate direction is None by default since it depends
  on whether the target is benign-quality or red-team — caller can
  override via directions_override.
- MIN_N_FOR_GATE = 10 → fewer pairs returns TooFewSamples, gate WARNs.

Phase 1 rollout
---------------
- continue-on-error: true so the gate is advisory while we observe
  noise and tune the per-metric MDEs.
- Exit criterion documented in workflow comment: flip to required after
  10 PRs / 2 weeks at <5% noisy WARN/BLOCK rate.
- MDE thresholds in DEFAULT_MDE are v1 placeholders documented as
  "recalibrate after first 5 baseline runs from observed variance."

Known v1 limitations (intentional)
----------------------------------
- Suite-level p-value in compare_suite_level is a placeholder (returns
  0.5 if abs(mean_diff) < mde else 0.05) — the function is wired
  through honestly but suite metrics are advisory in v1, so the
  approximation is acceptable. Real paired-bootstrap on jointly
  resampled seed ids is a follow-up.
- Ground-truth assertion mode (Abby/Riccardo benchmark eval set) is NOT
  in this PR. Rubber-duck recommended waiting until the dataset lands;
  decide() is shaped to add a ground-truth comparator without
  restructuring.
- Default N=100 (5 mins per spec at concurrency=10). Power analysis is
  not yet pre-derived — script reports observed variance per metric
  and flags WARN if underpowered.

Tests
-----
- 31/31 new regression module tests pass in <1s.
- Full suite: 644 passed, 14 skipped, 4 pre-existing Windows-only
  failures (file lock + HTTP test) unrelated to this change. CI runs
  on Ubuntu so they don't apply.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(science): isolate baseline checkout via git worktree + PYTHONPATH

The orchestrator was running both baseline and treatment from REPO_ROOT,
so `commit_sha` only labeled the output dir — both runs imported the
same source code, making the comparison a no-op.

This adds:

* `ensure_worktree(commit_sha)` / `remove_worktree` — git worktree
  per commit under .regression-worktrees/.
* `run_pipeline` now uses `cwd=worktree` AND prepends the worktree
  to `PYTHONPATH` so `import p2m` resolves to the worktree's source
  (not the editable install pointing at REPO_ROOT). Without this,
  `BASE_DIR = Path(__file__).resolve().parents[2]` would still point
  at the main checkout and load prompts/* from the treatment.
* `main()` cleans up worktrees in a `try/finally` so failed runs
  don't leak state.
* Re-running with the same (config, commit, n_seeds, judge_model) tuple
  short-circuits when scores already exist — keeps the workflow's
  baseline cache layer meaningful.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* ci(science): re-trigger gate when PR labels change

Default `pull_request` trigger only fires on opened/synchronize/
reopened. Without `labeled`, adding a `seeds:N` label after the
last push has no effect — the gate uses the seed count from whatever
labels were on the PR at push time.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(science): drive seed/judge overrides via rendered YAML, copy results out

The `p2m run` CLI accepts only `--config` (no `--suite`,
`--save-dir`, `--run`, `--set` exist). Sample sizes, judge model,
and output suite/run come from the YAML body.

* New `_render_config()` materialises a per-run YAML inside the
  worktree with `suite`, `run`, `pipeline.seeds.*.sample_size`,
  `pipeline.judge.model.name` overridden.
* `run_pipeline()` runs `p2m.cli run --config <rendered>` from the
  worktree, then copies `<worktree>/artifacts/results/<suite>/<run>/`
  out to `REPO_ROOT/artifacts/regression-runs/` so the result
  survives worktree teardown and the workflow cache picks it up.
* Keep PYTHONPATH/cwd/worktree wiring from the previous commit — that
  part was right; only the CLI invocation was wrong.

Found by the live PR #41 run: `Error: No such option: --suite Did you
mean --quiet?`

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(science): emit rendered YAML next to source so sibling concept md resolves

Concept-backed configs reference `concept: { name: <stem> }` and the
loader looks for `<config_dir>/<stem>.md` next to the YAML. Writing
the temp YAML to the worktree root broke that lookup.

Found by the live PR #41 run: `concept markdown is required ...
expected concept.md or travel_planner_safety.md next to ...`.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(science): drive policy/seeds/auditor through gpt-5.4 to avoid silent payload drops

The default `gpt-5.4-mini` in the regression configs is cheap enough
for human iteration, but adversarial scenario seed schemas trip its
content-filter / structured-output handling, and the model returns
null/empty parsed payloads. The pipeline catches that as
`invalid seeds payload` and aborts, so neither baseline nor
treatment ever produce scores.

Now: `--upstream-model` (default `azure/gpt-5.4`) overrides the
model on the policy, prompt seed, scenario seed, and auditor stages.
Judge already runs gpt-5.4 unchanged.

Found by the live PR #41 200-seed run (all 100 scenarios failed,
~38s into seed gen, before any rollout/judge work).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* debug(science): inject seeds-payload diagnostic into both worktrees

Baseline runs the BASE SHA's source which doesn't have the diagnostic
print. Patch the worktree's seeds.py after checkout so both branches
surface finish_reason/text on the invalid-payload path.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(science): bump seed-generation max_tokens to 16000

Diagnostic from run 25757697217 surfaced finish_reason=length with
completion_tokens=3000 — the project default
DEFAULT_GENERATION_MAX_TOKENS=3000 truncates scenario seed batches
(20-40 seeds × ~250 tokens each at sample_size=200 + behavior_count=5),
leaving incomplete JSON that fails to parse.

Override max_tokens=16000 for both prompt and scenario seed models in
the rendered regression configs. The qualevalexpeus endpoint has content
safety filtering disabled per project policy, so empty/null parses are
truncation, not refusals.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(science): install langgraph + examples extras in CI

The rollout target examples.travel_planner_langgraph.auto_trace:chat_sync
imports langchain/langgraph at module load. The previous '[dev]' install
only pulled pytest — every rollout job fast-failed with a ValueError at
runtime.open() because langgraph wasn't on sys.path.

Switch to '[all,dev]' (= otel + langgraph + analysis + examples +
regression + dev tooling).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(science): drop Phoenix OTLP wrapper, bump concurrency=10, timeout=180min

Three coordinated fixes for the 200-seed regression run:

1. Switch regression configs from auto_trace:chat_sync to agent:chat_sync.
   auto_trace registers phoenix.otel with auto_instrument=True, which sets
   up a BatchSpanExporter pointing at localhost:4317. CI has no Phoenix
   collector running, so every span emit triggers a gRPC retry storm and
   the BatchSpanExporter background asyncio task survives session shutdown,
   producing 'Event loop is closed' RuntimeErrors. agent:chat_sync skips
   the Phoenix wrapper entirely. Judge dimensions only score conversation
   messages and tool calls, not LangGraph node spans, so we lose nothing
   evaluation-relevant.

2. Bump rollout concurrency from 2 to max(existing, 10) in orchestrator.
   At seeds=200, 30s/conversation, conc=2 = 200 min/spec/commit -> 800 min
   total. Conc=10 -> 80 min total. qualevalexpeus has plenty of Azure
   quota for these deployments.

3. Bump workflow timeout from 60 to 180 minutes for safety margin while
   we calibrate real per-conversation latency.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(science): keep auto_trace path, run real Phoenix collector in CI

User feedback: bypassing auto_trace:chat_sync (the happy path) makes the
regression gate unfaithful to production. The judge sees a richer trace
via OTelTracedSession (per-turn span tree, per-tool args/results,
per-LLM-call params); without auto_trace it sees only messages. Different
trace fidelity -> different judge scores -> false positives/negatives on
PRs that touch span enrichment, OTel handling, or anything trace-adjacent.

Fix:
1. Revert configs to auto_trace:chat_sync + trace.backend=phoenix.
   (Restored from the prior commit that swapped to agent:chat_sync.)
2. Launch 'phoenix serve' as a background process in the workflow before
   the regression gate runs. Phoenix binds OTLP gRPC :4317 + UI :6006.
   With a real collector accepting spans, BatchSpanExporter flushes
   cleanly and there's no asyncio teardown noise.
3. Wait for both ports with nc -z; fail fast (60s budget) if Phoenix
   doesn't start. Upload phoenix.log as a CI artifact for triage.

Phoenix is already installed via the .[all] extra (arize-phoenix). No
new deps. Local sanity check: 'phoenix serve' binds both ports inside
the existing venv (Windows; CI is Ubuntu where bind is even cleaner).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(science): align regression metrics with test cases

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: label dataset-level test as 'mde_threshold' not 'bootstrap'

compare_dataset_level() uses a heuristic p-value based on whether the
absolute delta exceeds the MDE — no bootstrap resampling is performed.
Labeling the test as 'bootstrap' is misleading and confuses reviewers
reading the regression report JSON.

* chore: remove dead --alpha-canonical-only CLI flag

The flag is parsed (action='store_true', default=True) but never
referenced — decide() already hardcodes Holm correction over canonical
metrics only. Keeping the flag creates a false impression it's wired up.

* chore: consolidate duplicate _REPO_ROOT / REPO_ROOT into single REPO_ROOT

Both resolved to the same path. _REPO_ROOT was used only for the
sys.path hack; REPO_ROOT was used everywhere else. Merge into one.

* chore: rename UPSTREAM_STAGE_GLOBS → UPSTREAM_STAGE_FILES

The tuple contains exact relative paths, not glob patterns. The check
uses 'f in UPSTREAM_STAGE_FILES' (exact membership), not fnmatch.

* test: add coverage for compare_dataset_level()

Tests the MDE-threshold logic: within-MDE returns p=0.5 (inconclusive),
exceeding MDE returns p=0.05 (degraded/improved depending on direction),
no-direction yields Info, too-few-samples yields TooFewSamples, and
output shape (granularity, detail keys).

* revert: restore paired_bootstrap_ci and bootstrap_delta_pvalue

These functions are part of the PR's stated scope (bootstrap-based
regression testing). Restore them for future use as an alternative
statistical test alongside the current MDE-threshold heuristic.

* chore(science): port regression gate to current main

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 467f2cda-f4b4-40bc-b736-dbed856e355f

* ci(science): limit PR trigger to regression files

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 467f2cda-f4b4-40bc-b736-dbed856e355f

* ci(science): remove duplicate tier 4 gate

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 467f2cda-f4b4-40bc-b736-dbed856e355f

* fix(science): make regression gate truly paired

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 467f2cda-f4b4-40bc-b736-dbed856e355f

* fix(science): bound PR regression runtime

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 467f2cda-f4b4-40bc-b736-dbed856e355f

* test(viewer): align inference label expectation

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4c77c0fe-56d8-4aba-907a-4da7c31f3ee9

* fix(ci): gate on permissibility-split violations

Replace the inherited prototype efficacy metrics with policy violation rates conditioned on relevant permissible and non-permissible behavior judgments. Apply one-sided McNemar tests for degradation and Holm-Bonferroni across the two regression p-values.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 88322bba-fc22-4759-b3e5-55b1d1f57d37

* fix(ci): increase regression gate power

Evaluate each one-sided McNemar degradation test independently at p < 0.10. Remove Holm-Bonferroni correction so the default 20-case PR run can detect smaller regressions.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 88322bba-fc22-4759-b3e5-55b1d1f57d37

---------

Co-authored-by: Chang Liu <changliu2@microsoft.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Yeming Tang <tangym@users.noreply.github.com>
Copilot-Session: 467f2cda-f4b4-40bc-b736-dbed856e355f
Copilot-Session: 4c77c0fe-56d8-4aba-907a-4da7c31f3ee9
Copilot-Session: 88322bba-fc22-4759-b3e5-55b1d1f57d37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@tangym tangym Awaiting requested review from tangym tangym will be requested when the pull request is marked ready for review tangym is a code owner
@jakepresent jakepresent Awaiting requested review from jakepresent jakepresent will be requested when the pull request is marked ready for review jakepresent is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@changliu2