feat(#121): contaminated shared-context experiment - #146
Conversation
Inject a spurious artifact into the workspace itself (not a peer's turn) and measure whether the committee anchors on it, isolating the shared-context channel from peer conformity. - blackboard: add board_seed to run_committee (ambient context artifact placed before the member loop, not attributed to any agent, kept out of committed, consuming no member slot); reject member-attributed context turns; make context turns visible in isolated mode - schema: add Turn.context flag for ambient workspace artifacts - contaminated_context: 2x2 present/absent x shared/isolated runner reporting the anchoring effect and shared_amplification, holding the condition constant across arms; compose the referee and blind-metric probe
a53d49c to
bacfaf1
Compare
|
Reviewed the design (not merging, this one is yours @Agastya191). This is a clean and valuable addition: Two notes:
Nice work. |
|
confirmed context = True is excluded so it never can be counted as a member turn. Coordinated With Mohammad as well. Merging to main now. |
|
Thanks @Agastya191, that resolves my note (the |
…ntext feat(#121): contaminated shared-context experiment
…corrected (closes #131) Rebased fresh off current main (the original branch predated #142/#143/#146/#148/#154/#157/ #161/#219 and would have deleted all of that merged work if landed as-is). Addresses @Agastya191's review on #159: - flash_lite_reference was hardcoded to scale_c's PRE-parser-fix numbers (0.33/0.51 at n=150, p<1e-4). Now reads dynamically from --scale-c-summary (scale_c's committed summary, PR #141) so it cannot drift out of sync with a future fix there. Regenerated: 0.729/0.847 at n=85, p=0.041. - Also caught in the same pass: the docstring and README's own flash numbers were ALSO stale (60 hard cases, generic 0.10/anchored 0.083) versus the actually-committed, already-corrected summary (28 hard cases, generic 0.714/anchored 0.679). Rewrote both to match the real data and restated the finding precisely: both tiers conform substantially to a bare peer (~0.71-0.73), but only flash-lite's conformity climbs further under a case-anchored rationale (+0.12, p=0.041); flash's does not move. The anchoring lever is model-dependent, not conformity itself. Verified end-to-end: keyless reproduction confirmed with the key unset (new_api_calls_this_run: 0, n_hard_cases: 28, matches exactly), ruff clean, 625 tests pass on the rebased base, no hardcoded personal paths, 0 em dashes.
…referee) (#162/#163/#164/#165) Provenance manifest via the registered nih_cxr14 adapter with sha256 checksums per used image (#162); solo cue-susceptibility across cable/corner-tag/watermark/laterality, watermark +0.11 above the temperature-resampled noise floor (#163); the watermark cascade, shared adopt 0.97 vs isolated 0.34, contagion +0.63 (#164); a deployable referee with no privileged knowledge (transcript + one private re-read) catching peer-driven adoptions at P/R 0.86, FPR 0.23 versus a naive conformity gate's FPR 0.92 (#165). Branched fresh off main (not off the stale results/cascade-at-scale, which predates #146/#148/ #154/#157/#161 and would delete that work if merged) and reuses the already-committed, already-reviewed solo/cascade logic and numbers from that branch. All four runners are fully keyless-reproducible from the committed img_cache.jsonl (new_api_calls_this_run: 0, verified).
…loses #108) Rebased fresh off current main (the original branch predated #142/#143/#146/#148/#154/#157/ #161/#219 and would have deleted all of that merged work if landed as-is, same stale-base issue as #143/#150/#141/#159). Addresses @Agastya191's review on #158: - README/PR body had stale pre-fix numbers (accuracy 0.29/0.14, flash flip 0.76 vs 1.00 at p=9.8e-5) versus the already-corrected committed summary (accuracy 0.89/0.78, flash flip 0.079 vs 0.455 at p=3.3e-3, flash-lite 0.090 vs 0.636 at p=4.4e-7). Rewrote to match. - ever_flipped collapsed the three cues into a per-case boolean that doesn't match the stated headline flip rate (0.063/0.117). Now reports both per_record_flip_rate (the headline quantity) and ever_flipped_rate, clearly labeled and distinguished. - Grading-standard mismatch: clean_correct was imported from the prior solo pipeline rather than re-verified. Now re-graded by exact match against ground truth, the same standard as options_only_correct. Re-graded values are identical to the imported ones (0.89/0.78), so no actual discrepancy, but the standard is now transparent and consistent. - Cache race: _Cache.complete checked the store under lock but called the API outside it, so two threads could both miss the same key and both append a duplicate entry (confirmed: the committed cache had exactly one). Lock now spans the full miss-check-fetch-store sequence. Cache also pruned from 6054 shared-cache entries down to the 400 this script actually requests (100 cases x 2 models x 2 probes). Verified end-to-end: keyless reproduction confirmed with the key unset (new_api_calls_this_run: 0, all numbers match exactly), ruff clean, 625 tests pass, no hardcoded personal paths, 0 em dashes.
Rebased fresh off current main (the original branch predated #142/#143/#146/#148/#154/#157/ #161/#219 and would have deleted all of that merged work if landed as-is, same stale-base issue as #143/#150/#141/#159/#158). Addresses @Agastya191's review on #155: - README/PR body carried pre-fix numbers (shared 0.175/0.15/0.175/0.15/0.15, p=1.0) versus the already-corrected committed summary (0.275/0.3/0.275/0.325/0.325, gained 3 lost 1 p=0.625). Rewrote to match and restated the finding as no significant compounding at p=0.625. - Real bug: the isolated holdout's prompt was byte-identical to the bare prompt every round (board text was built only from peer votes, which are never visible in isolated mode), so at temperature 0 it deterministically reproduced round 1's answer every round. Isolated adoption was 0 by construction, not by measurement. Fixed by reminding the isolated holdout of its own previous-round answer (visible to it in isolated mode; only peer turns are hidden), so the prompt genuinely differs round to round. The shared arm's board construction is UNCHANGED, so the shared numbers are identical to what was already reviewed; only the isolated arm is new. Result: isolated adoption is now a genuine (if still near-floor) measurement: 0.0/0.025/0.0/ 0.025/0.0, at most 1 of 40 cases per round, rather than a trivial constant 0.0. Verified end-to-end: ran for real with the isolated fix (206 new calls), then re-verified keyless reproduction with the key unset (new_api_calls_this_run: 0, exact match on both curves), ruff clean, 629 tests pass, no hardcoded personal paths, 0 em dashes.
What
Adds the contaminated shared-context experiment (#121), which injects a spurious signal into the shared workspace itself instead of through a peer's turn, isolating the shared-context channel from peer conformity. It adds:
board_seedparam onrun_committee(blackboard.py) that drops an artifact -- a pre-screen flag, a "prior similar cases were all X" note, a leaked proxy metric -- onto the board before the member loop, not attributed to any agent, kept out of the committed map, and consuming no member slot (soseed_turnindexing is untouched). A board seed attributed to a committee member now raises, since that would leak a peer.contextflag onschema.Turnmarking that artifact as ambient workspace context, so_visible_turnskeeps it visible even in isolated mode -- isolation is meant to hide peers, not the shared workspace, which is the whole point of the isolated control.contaminated_context.py, which runs the 2x2 (present/absent x shared/isolated) and reports the anchoring: present minus absent is the artifact's pull, and shared minus isolated (shared_amplification) is the extra pull only the shared channel adds. The twin condition is held constant across all four arms on purpose, so the only thing differing between present and absent is the artifact, not theAgentView.conditionlabel.score_shortcutreliance) and the blind-metric probe (blind_metric_uptake, with the artifact as the hidden proxy), rather than reinventing either.Plain
run_committeebehavior is unchanged whenboard_seedisn't passed -- the newcontextfield defaults to False and every existing path stays byte-identical, same as before.Note: this touches the shared
Turnschema (a newcontextfield) rather than overloading the existingseededflag, because the cascade'sseed_turnis a peer utterance that must stay hidden in isolated mode while a workspace artifact must not -- flagging in case you'd rather key isolated-mode visibility off an agent_id convention instead. One thing I left alone: the two visibility arms share a run_id (sharedisn't in it), same asrun_cascadealready does, andmeta["shared"]disambiguates them -- say the word if you'd rather I make run_id unique.Testing
tests/test_contaminated_context.py:
board_seed contract tests appended to tests/test_blackboard.py:
committed, never shiftsseed_turn's slot index, and neither mutates the caller's Turn nor aliases it across two runsSuite: 492 -> 519 passing locally, offline mock path only (couldn't run
ruff checkin this environment, so it still needs a local lint pass before merge).