feat: dataset adapters + stage runners + scrutiny + prompt registry (ready to run on real data) - #66
Merged
Merged
Conversation
…utiny panel, live-referee/natural-cue drivers, pipeline doc Implements everything whose logic can be written and fixture-tested now, leaving only the real data download for contributors: - datasets: real parsers for MedQA (JSONL), MIMIC-CXR (metadata+chexpert+reports join), CheXpert (Support Devices), NIH ChestX-ray14; ehr resource-context loader. Each mapped to the shared Case schema and tested against a synthetic fixture matching the real raw format. - prompts.py: versioned PromptTemplate + PromptRegistry with the core agent prompts. - experiments.py: stage runners wiring existing functions (run_pilot, run_solo_baselines, run_holes_test, run_cascade), testable offline with mock backends. - scrutiny.py: stage-5 benchmark scrutiny panel (planted defects, flaw-detection rate, coverage-gap estimate, pluralistic vs homogeneous comparison). - interventions.py: real-time referee driver (pre-emptive hook) + CheXpert natural-cue validation (rank correlation). - docs/PIPELINE.md: the end-to-end data-contract document. - fix the two stale test_datasets.py tests (adapters now fail loudly on missing data rather than raise NotImplementedError). Suite: 292 passed, 5 skipped (optional-dep paths), ruff clean. On real data + a Gemini key these run unchanged.
This was referenced Jul 18, 2026
sebasmos
requested review from
Agastya191,
PeterLi-jpg,
aaaaaashz,
aadityapanchal7,
amarzullo24,
armaanvgrewal,
dr-shrey,
duckyquang and
maximinl
July 18, 2026 20:30
This was referenced Jul 19, 2026
This was referenced Jul 20, 2026
sebasmos
requested review from
MohShahin,
arshnoor08,
azzy-chemE,
dr-shrey and
renato-umeton
and removed request for
Agastya191,
MohShahin,
PeterLi-jpg,
aaaaaashz,
aadityapanchal7,
amarzullo24,
armaanvgrewal,
arshnoor08,
azzy-chemE,
dr-shrey,
duckyquang,
felipeocampoos and
maximinl
July 20, 2026 09:14
felipeocampoos
approved these changes
Jul 20, 2026
felipeocampoos
left a comment
Collaborator
There was a problem hiding this comment.
Reviewed the code and tested a real merge against current main (this branch is 11 commits behind, so I didn't trust GitHub's "mergeable" flag alone — that only checks for textual conflicts, not behavioral consistency with the code that landed since).
What I checked:
- Read through
datasets/medqa.py,scrutiny.py,interventions.pyin full, skimmed the other adapters andexperiments.py/prompts.py. Consistent with the rest of the codebase's style: careful edge-case guards (flaw_detection_ratereturnsnanon nothing-planted rather than a false-perfect score,natural_cue_validationexplicitly rejects anancorrelation viacorr == corr, the scrutiny moderator can only veto/intersect with what seats actually flagged, never invent a detection). ehr.pyintentionally raisesNotImplementedErrorfor the stage-5 loader — documented as such, not a stray stub.- No secrets, no hardcoded paths, no bare
except:/TODOs left in the new modules.
Tested a real merge, not just the branch in isolation:
git merge main --no-fffrom this branch: merges cleanly (no textual conflicts).- On the merged tree:
pytest→ 476 passed, 6 skipped, 1 failed — the 1 failure istest_validate.py::test_checksum_is_stable_and_matches_hashlib, confirmed pre-existing onmainitself (Windows CRLF vs a literal-string hash fixture), unrelated to this PR. ruff check benchmaxxing testson the merged tree — clean.- On the branch as-is (pre-merge, before
validate.py/#100's fixes existed), 293 passed / 4 skipped, matching the PR description's "292 passed" closely enough (env-dependent skip count).
One non-blocking note: the branch is quite stale (11 commits behind main) — worth an actual rebase/merge-commit before landing so the merge that GitHub performs is the same one I tested, not just "no conflicts."
Code quality and test coverage are solid — approving.
This was referenced Jul 21, 2026
sebasmos
added a commit
that referenced
this pull request
Aug 4, 2026
feat: dataset adapters + stage runners + scrutiny + prompt registry (ready to run on real data)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Implements the remaining pipeline logic so the repo is ready to run once real data is staged. Everything here is fixture/mock tested offline (no API keys, no real data).
Caseschema.PromptTemplate+PromptRegistrywith the core committee/orchestrator/referee prompts; the run manifest pins the (name, version) used.run_pilot(stage 0),run_solo_baselines+run_holes_test(stage 1),run_cascade(stage 2), all injectable and offline-testable.test_datasets.pytests (adapters now fail loudly on missing data instead of raisingNotImplementedError).What still needs real data (not code)
Only the data downloads and a live Gemini key. The dataset adapters parse the documented formats and are fixture-tested, so an owner points
raw_rootat the credentialed MIMIC/CheXpert or the open NIH/MedQA and runs. Those verification steps are tracked in the open dataset issues (#45/#46/#47/#49).Suite: 292 passed, 5 skipped, ruff clean.