results: model-dependence of the plausibility cascade (C), rebased + corrected + cross-tier referee (closes #183) - #223
Merged
Conversation
…corrected (closes #131) Rebased fresh off current main (the original branch predated #142/#143/#146/#148/#154/#157/ #161/#219 and would have deleted all of that merged work if landed as-is). Addresses @Agastya191's review on #159: - flash_lite_reference was hardcoded to scale_c's PRE-parser-fix numbers (0.33/0.51 at n=150, p<1e-4). Now reads dynamically from --scale-c-summary (scale_c's committed summary, PR #141) so it cannot drift out of sync with a future fix there. Regenerated: 0.729/0.847 at n=85, p=0.041. - Also caught in the same pass: the docstring and README's own flash numbers were ALSO stale (60 hard cases, generic 0.10/anchored 0.083) versus the actually-committed, already-corrected summary (28 hard cases, generic 0.714/anchored 0.679). Rewrote both to match the real data and restated the finding precisely: both tiers conform substantially to a bare peer (~0.71-0.73), but only flash-lite's conformity climbs further under a case-anchored rationale (+0.12, p=0.041); flash's does not move. The anchoring lever is model-dependent, not conformity itself. Verified end-to-end: keyless reproduction confirmed with the key unset (new_api_calls_this_run: 0, n_hard_cases: 28, matches exactly), ruff clean, 625 tests pass on the rebased base, no hardcoded personal paths, 0 em dashes.
sebasmos
changed the title
Probes referee_independence_note: does a referee from a DIFFERENT model tier than the holdout catch adoption as well as a same-tier referee (the in-scope Gemini proxy for the blocked cross-lineage arm)? Every committed referee so far re-queried the same tier as the holdout. Both directions turn out zero-cost: reconstructed the identical bare-prompt hash per case (using the local MedQA manifest for ground truth only) and confirmed both tiers' answers are already cached for both case sets (40/40 flash hits for referee_deployable's 40 cases, 28/28 flash-lite hits for cascade_C_flash's 28 hard cases) - no new API calls, verified keyless with no key set at all. Result: same-tier is trivially perfect by construction in both directions; cross-tier catches the large majority of adoptions with perfect precision but real recall degradation - misses 1/15 (flash auditing flash-lite) and ~4/19 (flash-lite auditing flash, a bigger drop, consistent with a weaker referee tier being less able to independently out-reason a stronger committee's confidently-wrong answer). Caught and fixed a real bug before shipping: an early draft compared cascade_C_flash.jsonl's already-boolean "anchored" field against the wrong answer TEXT again (bool == str, always False), zeroing out the ground truth and making both referees score 0.0/0.0. Fixed to use the field directly as the boolean it already is.
sebasmos
changed the title
…-dependence # Conflicts: # experiments/referee/README.md
sebasmos
added a commit
that referenced
this pull request
Aug 4, 2026
…dence results: model-dependence of the plausibility cascade (C), rebased + corrected + cross-tier referee (closes #183)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Supersedes #159 (rebased fresh off current main). Extends the model-dependence question raised by clean-A and scale_c (#141); no tracked issue is closed by this PR alone.
Model-dependence of the plausibility cascade (C): does the anchored-vs-generic lever that moves flash-lite (0.73 -> 0.85, p=0.041) also move the stronger flash?
What changed vs #159
@Agastya191 flagged on #159 that
flash_lite_referencewas hardcoded to scale_c's pre-parser-fix numbers (0.33/0.51 at n=150, p<1e-4). Fixed: it now reads dynamically from--scale-c-summary(scale_c's committed summary, PR #141), so it cannot drift out of sync again.While fixing that I found the docstring and README's own flash numbers were also stale (60 hard cases, generic 0.10/anchored 0.083) versus the already-corrected committed summary (28 hard cases, generic 0.714/anchored 0.679). Rewrote both to match.
Result (28 flash hard cases, corrected parser)
Both tiers conform substantially to a bare peer assertion (~0.71-0.73). The difference is what happens when the rationale becomes case-anchored: only flash-lite's conformity climbs further; flash's does not move. The anchoring lever is model-dependent, not conformity to a peer in general.
Why a new PR instead of pushing to #159's branch
#159's branch (
results/cascade-model-dependence) predates #142/#143/#146/#148/#154/#157/#161/#219 and would have deleted all of that merged work if landed as-is (the same stale-base issue already caught on #143/#150/#141). This branch is rebased fresh off currentmain.Verified: keyless reproduction with the key unset (
new_api_calls_this_run: 0, exact numeric match), ruff clean, 625 tests pass, no hardcoded personal paths, 0 em dashes.Also added: cross-tier referee independence (closes #183)
Probes
referee_independence_note: does a referee from a DIFFERENT model tier than the holdoutcatch adoption as well as a same-tier referee (the in-scope Gemini proxy for the blocked
cross-lineage arm)? Zero-cost: reconstructed the identical bare-prompt hash per case and confirmed
both tiers' answers are already cached for both
referee_deployable.jsonl(40 cases) and thisfolder's own
cascade_C_flash.jsonl(28 hard cases) - no new API calls, verified keyless.Independence does not require cross-lineage in either direction: a different Gemini tier catches
the large majority of adoptions with perfect precision, but recall degrades - more so when the
referee tier is weaker than the committee it audits. Caught and fixed a real bug before shipping:
an early draft compared
cascade_C_flash.jsonl's already-booleananchoredfield against thewrong-answer text again (bool == str, always False), zeroing out the ground truth.