Skip to content

feat: dataset adapters + stage runners + scrutiny + prompt registry (ready to run on real data) - #66

Merged
sebasmos merged 1 commit into
mainfrom
feat/adapters-runners
Jul 20, 2026
Merged

feat: dataset adapters + stage runners + scrutiny + prompt registry (ready to run on real data)#66
sebasmos merged 1 commit into
mainfrom
feat/adapters-runners

Conversation

@sebasmos

Copy link
Copy Markdown
Member

What

Implements the remaining pipeline logic so the repo is ready to run once real data is staged. Everything here is fixture/mock tested offline (no API keys, no real data).

  • Dataset adapters (real formats, tested against synthetic fixtures matching the raw layout): MedQA (JSONL), MIMIC-CXR (metadata + chexpert-label + report join, one Case per image), CheXpert (with the Support Devices flag), NIH ChestX-ray14, and an EHR resource-context loader for stage 5. Each maps into the shared Case schema.
  • prompts.py: versioned PromptTemplate + PromptRegistry with the core committee/orchestrator/referee prompts; the run manifest pins the (name, version) used.
  • experiments.py: the stage runners that wire existing functions, run_pilot (stage 0), run_solo_baselines + run_holes_test (stage 1), run_cascade (stage 2), all injectable and offline-testable.
  • scrutiny.py: the stage-5 benchmark-scrutiny panel (planted defects, flaw-detection rate, coverage-gap estimate, pluralistic vs homogeneous comparison).
  • interventions.py: the real-time referee driver (uses the pre-emptive hook) + CheXpert natural-cue validation (rank correlation of injected vs natural-device susceptibility).
  • docs/PIPELINE.md: the end-to-end data-contract doc.
  • Fixes the two stale test_datasets.py tests (adapters now fail loudly on missing data instead of raising NotImplementedError).

What still needs real data (not code)

Only the data downloads and a live Gemini key. The dataset adapters parse the documented formats and are fixture-tested, so an owner points raw_root at the credentialed MIMIC/CheXpert or the open NIH/MedQA and runs. Those verification steps are tracked in the open dataset issues (#45/#46/#47/#49).

Suite: 292 passed, 5 skipped, ruff clean.

…utiny panel, live-referee/natural-cue drivers, pipeline doc

Implements everything whose logic can be written and fixture-tested now, leaving only the
real data download for contributors:
- datasets: real parsers for MedQA (JSONL), MIMIC-CXR (metadata+chexpert+reports join),
  CheXpert (Support Devices), NIH ChestX-ray14; ehr resource-context loader. Each mapped to
  the shared Case schema and tested against a synthetic fixture matching the real raw format.
- prompts.py: versioned PromptTemplate + PromptRegistry with the core agent prompts.
- experiments.py: stage runners wiring existing functions (run_pilot, run_solo_baselines,
  run_holes_test, run_cascade), testable offline with mock backends.
- scrutiny.py: stage-5 benchmark scrutiny panel (planted defects, flaw-detection rate,
  coverage-gap estimate, pluralistic vs homogeneous comparison).
- interventions.py: real-time referee driver (pre-emptive hook) + CheXpert natural-cue
  validation (rank correlation).
- docs/PIPELINE.md: the end-to-end data-contract document.
- fix the two stale test_datasets.py tests (adapters now fail loudly on missing data rather
  than raise NotImplementedError).

Suite: 292 passed, 5 skipped (optional-dep paths), ruff clean. On real data + a Gemini key
these run unchanged.
@sebasmos
sebasmos requested a review from felipeocampoos July 18, 2026 20:30

@felipeocampoos felipeocampoos left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the code and tested a real merge against current main (this branch is 11 commits behind, so I didn't trust GitHub's "mergeable" flag alone — that only checks for textual conflicts, not behavioral consistency with the code that landed since).

What I checked:

  • Read through datasets/medqa.py, scrutiny.py, interventions.py in full, skimmed the other adapters and experiments.py/prompts.py. Consistent with the rest of the codebase's style: careful edge-case guards (flaw_detection_rate returns nan on nothing-planted rather than a false-perfect score, natural_cue_validation explicitly rejects a nan correlation via corr == corr, the scrutiny moderator can only veto/intersect with what seats actually flagged, never invent a detection).
  • ehr.py intentionally raises NotImplementedError for the stage-5 loader — documented as such, not a stray stub.
  • No secrets, no hardcoded paths, no bare except:/TODOs left in the new modules.

Tested a real merge, not just the branch in isolation:

  • git merge main --no-ff from this branch: merges cleanly (no textual conflicts).
  • On the merged tree: pytest → 476 passed, 6 skipped, 1 failed — the 1 failure is test_validate.py::test_checksum_is_stable_and_matches_hashlib, confirmed pre-existing on main itself (Windows CRLF vs a literal-string hash fixture), unrelated to this PR.
  • ruff check benchmaxxing tests on the merged tree — clean.
  • On the branch as-is (pre-merge, before validate.py/#100's fixes existed), 293 passed / 4 skipped, matching the PR description's "292 passed" closely enough (env-dependent skip count).

One non-blocking note: the branch is quite stale (11 commits behind main) — worth an actual rebase/merge-commit before landing so the merge that GitHub performs is the same one I tested, not just "no conflicts."

Code quality and test coverage are solid — approving.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants