results: MedQA experiments (reproduction script + saved results); closes #93, #96 - #135
Conversation
…rst real-model runs Closes #93 (stage-0 MedQA pilot) and #96 (cascade experiment on real data). Adds experiments/medqa/reproduce.py (sanitized: reads GEMINI_API_KEY from env, all paths as args, no secrets, uses only modules on main) and a saved copy of the results under results/: solo susceptibility (flash 0.79 / flash-lite 0.89 flip, noise floor 0.000), same-lineage overlap, and the cascade (first-distractor seed was invalid because it coincided with the committee baseline in 16/20 cases; the baseline-relative seed gives near-zero contagion). Per-twin records, per-case cascade rows, config and provenance included; the call cache and transcripts are regenerable and not committed. Adds a .env gitignore guard. The cascade null is a setup problem, not a thesis problem; the redesign (voiceable cue-rationale seed, hard-case selection, majority pressure, natural-cue imaging cascade) is tracked in #115/#116/#117/#120/#121.
…offline reproduction The call cache (model+prompt -> response) lets `reproduce.py --out experiments/medqa/results` reproduce every number with zero API calls; the transcripts replay each cascade run (v1 and the baseline-relative v2) offline. Both scrubbed clean of local paths.
maximinl
left a comment
maximinl
left a comment
There was a problem hiding this comment.
Thanks for publishing the full result bundle — having the raw cache and transcripts available is valuable, and the write-up is refreshingly candid about the null cascade result.
There might be a reproducibility mismatch worth resolving before merge. RESULTS.md says the saved numbers were produced by medqa_full.py and cascade_v2.py, while the PR commits a different consolidated reproduce.py. That script also writes different result filenames, and its v2 cascade transcripts use {case_id}_shared.jsonl / {case_id}_isolated.jsonl, which appear likely to overwrite the saved v1 transcripts rather than reproduce the committed _v2_... files. It would be helpful either to commit the exact generating scripts or make reproduce.py regenerate the committed bundle byte-for-byte (apart from documented metadata).
One smaller point: _require_key() is called before the cache is read, so a fully cached run still cannot execute without an API key. The claim that the bundle reproduces offline/with zero API calls may be clearer if cached runs genuinely work without credentials, or if the documentation states that a key is still required even when no request is made.
Would you mind aligning the reproduction script, filenames, and documentation so reviewers can verify that the committed results came from the committed code?
…+ honest attribution Per @maximinl: - The key check ran before the cache read, so a fully cached run still needed a key. The key is now lazy: CachedBackend builds the live backend only on a cache miss, and a keyless run reproduces the committed solo/cascade numbers from the cache with zero calls (verified: flip 0.787/0.893, no new cache lines). The uncached noise-floor control is skipped without a key. - reproduce.py wrote {case}_shared.jsonl / _isolated.jsonl, which would overwrite the committed v1 transcripts; it now writes {case}_repro_*.jsonl so it can never clobber the committed run. - RESULTS.md/README no longer reference the uncommitted medqa_full.py/cascade_v2.py; they point at reproduce.py and state exactly what reproduces keyless vs what needs a key.
|
Thanks @maximinl, both correct. Fixed in
|
…le vs reproduce.py verifier outputs; _repro_ transcripts never overwrite; verified keyless zero-call reproduction 2026-07-21)
|
Thanks @maximinl, provenance addressed in
The earlier per-stage runners that produced the original bundle are superseded by Verified 2026-07-21: a keyless |
… a stray leading 'A', mis-scoring ~85% as option A; solo accuracy was really 0.89/0.78, flip 0.06/0.12). See the 2026-07-21 re-grade
|
Parser-bug correction (2026-07-21). The answer parser (
|
…cy 0.89/0.78, flip 0.06/0.12; corrected)
Activates benchmaxxing.blind_arms.misaligned_proxy_run (built and unit- tested with mocks only, invoked by no data runner) on the committed longest_option cue rows of solo_records.jsonl (n=200). Proxy = length of the seeded/contaminated answer text; truth = contaminated_correct; decision = whether the case flipped toward the seeded option. Honest reading of the result: uptake_delta is positive (0.498) but this is driven mostly by the strong negative truth correlation (flipping toward the seeded option is usually wrong, by design), not a strong positive proxy correlation (length only weakly predicts which cases flip, r=0.093). Does not overclaim that answer length specifically drives adoption. Stacked on this branch (not #135/results/medqa-experiments, which is stale - 39 commits behind main) since this branch already carries an equivalent, current copy of solo_records.jsonl. Imaging-lane half is on results/imaging-reanalysis (PR #226).
…(part of #174) Activates benchmaxxing.onset.cascade_onset (the plan's flagship reusable artifact, exported and unit-tested but never reported on committed data) on the baseline-relative arm's transcripts. Uses _repro_shared.jsonl (post-parser-fix) rather than the plain medqa-*_shared.jsonl the issue names, since those predate the fix and were never regenerated. Result: onset detected on all 20/20 cases, always at the same turn (2) - the trivial detection of the seeded agent's one-off wrong turn immediately reverting, not a genuine sustained tipping point. Consistent with the already-established robust null on this arm. Also reports cascade_v2_per_case.jsonl's existing onset field (anchored arm), explicitly flagged as confirmed tainted by the pre-fix parser bug and not treated as trustworthy - included because the issue asks for it. Also fixed #135's own staleness (39 commits behind main) by merging main in place rather than superseding the PR, since it already carries felipeocampoos's approval and Max's two remaining review concerns (reproducibility mismatch, _require_key ordering) both appear already resolved in the current committed code.
|
Two updates to this branch, no changes to the reviewed code:
On the two open review points:
Both look resolved in the current committed code from this side; flagging in case a fresh look |
…117) Does a wrong 2-of-3 majority move the holdout more than a single wrong peer? Seeds k of the committee's peers with the same case-anchored wrong answer + rationale and measures the non-seeded flash-lite holdout's adoption at k=0/1/2. Genuinely exercises run_committee's multi-spec seed_turn (a sequence of specs plants several slots) on real committee data for the first time; the capability itself is covered by tests/test_blackboard.py's multi-seed tests. 45 new API calls at n=40; verified keyless afterward (0 new calls). Real finding: a wrong 2-of-3 majority does NOT move the flash-lite holdout meaningfully more than a single wrong peer (adoption 0.125 -> 0.15, McNemar 1-vs-2 p=1.0, only 1 net discordant case of 40), and stays low throughout (<=0.15) even with a persuasive rationale on every seeded peer. The Asch prediction fails for a safety-trained same-lineage committee. Mirrors the imaging majority-pressure result (#172) from the opposite direction: there adoption saturated at one peer near-total (0.97), here it saturates at one peer at a low floor (~0.13); both lanes agree the 1-to-2-peer increment is null. Two bugs caught and fixed before this final run: (1) an early draft seeded with a bare answer and no rationale (rediscovering the known "bare seed is weak" null at the wrong k); (2) an early draft measured the FLASH holdout (the resistant tier) instead of flash-lite (the susceptible one, matching referee_deployable/scale_c) - both fixed before drawing any conclusion. Uses the real MedQA manifest available locally on this machine.
|
Added the text-lane majority-pressure (Asch) variant here ( It genuinely exercises |
|
Hi @maximinl, both points from your review are resolved in committed code. The reproduce/provenance mismatch is documented in |
…ents # Conflicts: # experiments/medqa/results/solo_records.jsonl
Activates benchmaxxing.blind_arms.misaligned_proxy_run (built and unit- tested with mocks only, invoked by no data runner) on the committed longest_option cue rows of solo_records.jsonl (n=200). Proxy = length of the seeded/contaminated answer text; truth = contaminated_correct; decision = whether the case flipped toward the seeded option. Honest reading of the result: uptake_delta is positive (0.498) but this is driven mostly by the strong negative truth correlation (flipping toward the seeded option is usually wrong, by design), not a strong positive proxy correlation (length only weakly predicts which cases flip, r=0.093). Does not overclaim that answer length specifically drives adoption. Stacked on this branch (not #135/results/medqa-experiments, which is stale - 39 commits behind main) since this branch already carries an equivalent, current copy of solo_records.jsonl. Imaging-lane half is on results/imaging-reanalysis (PR #226).
…+ honest attribution Per @maximinl: - The key check ran before the cache read, so a fully cached run still needed a key. The key is now lazy: CachedBackend builds the live backend only on a cache miss, and a keyless run reproduces the committed solo/cascade numbers from the cache with zero calls (verified: flip 0.787/0.893, no new cache lines). The uncached noise-floor control is skipped without a key. - reproduce.py wrote {case}_shared.jsonl / _isolated.jsonl, which would overwrite the committed v1 transcripts; it now writes {case}_repro_*.jsonl so it can never clobber the committed run. - RESULTS.md/README no longer reference the uncommitted medqa_full.py/cascade_v2.py; they point at reproduce.py and state exactly what reproduces keyless vs what needs a key.
…le vs reproduce.py verifier outputs; _repro_ transcripts never overwrite; verified keyless zero-call reproduction 2026-07-21)
…cy 0.89/0.78, flip 0.06/0.12; corrected)
…(part of #174) Activates benchmaxxing.onset.cascade_onset (the plan's flagship reusable artifact, exported and unit-tested but never reported on committed data) on the baseline-relative arm's transcripts. Uses _repro_shared.jsonl (post-parser-fix) rather than the plain medqa-*_shared.jsonl the issue names, since those predate the fix and were never regenerated. Result: onset detected on all 20/20 cases, always at the same turn (2) - the trivial detection of the seeded agent's one-off wrong turn immediately reverting, not a genuine sustained tipping point. Consistent with the already-established robust null on this arm. Also reports cascade_v2_per_case.jsonl's existing onset field (anchored arm), explicitly flagged as confirmed tainted by the pre-fix parser bug and not treated as trustworthy - included because the issue asks for it. Also fixed #135's own staleness (39 commits behind main) by merging main in place rather than superseding the PR, since it already carries felipeocampoos's approval and Max's two remaining review concerns (reproducibility mismatch, _require_key ordering) both appear already resolved in the current committed code.
…iant results Follow-up to #135. Adds break_it.py (sanitized: GEMINI_API_KEY from env, paths as args) and the saved summaries for the full cascade sweep: baseline-relative seed, majority pressure, framing diagnostic, and the A/C/D mechanism sweep. Finding: the text-lane cascade is null under conformity (bare seed, baseline-relative, 2-of-3 majority 0/30, framing 0.07/0.00/0.00), but the mechanism sweep found the lever: C (case-anchored plausible seed) raises holdout conformity 0.25 -> 0.45. A is confounded (flag == baseline), B and D null. Relates to #136/#137/#138/#139.
4 participants
What
Saves the first real-model MedQA experiments into the repo: a sanitized reproduction script plus the full results (including raw call cache and transcripts). Closes #93 (stage-0 MedQA pilot) and closes #96 (cascade experiment on real data + onset/contagion).
No secrets:
reproduce.pyreadsGEMINI_API_KEYfrom the environment, all paths are arguments, a.envgitignore guard is added, and every committed file was scrubbed of local paths.Results
Solo susceptibility (Story 2): strong. 100 MedQA cases, same-lineage Gemini tiers, answer-preserving MCQ cues. Noise floor = self-inconsistency when the same clean case is run twice with no cue (uncached control), so flip-above-noise is the honest susceptibility.
Per model x cue flip rate:
Same-lineage overlap: within-lineage phi 0.56 (cross term undefined with one lineage; needs the open-weights arm).
Cascade (Story 1): null, but the setup was the problem, not the thesis. Two seed designs:
With a proper counterfactual seed, same-lineage Gemini committees hold their independent answer even against a confident wrong peer on the shared board. The redesign (voiceable cue-rationale seed, hard-case selection, majority pressure, and the natural-cue imaging cascade) is tracked in #115 / #116 / #117 / #120 / #121.
Contents
experiments/medqa/reproduce.pymainexperiments/medqa/results/RESULTS.mdexperiments/medqa/results/solo_records.jsonlexperiments/medqa/results/cascade_v2_*experiments/medqa/results/call_cache.jsonlexperiments/medqa/results/transcripts/experiments/medqa/results/config.json,versions.jsonNote
Building the MedQA manifest needs the adapter from #66; once that merges the whole flow runs from
main.