Skip to content

feat(#121): contaminated shared-context experiment - #146

Merged
Agastya191 merged 1 commit into
mainfrom
feat/contaminated-shared-context
Jul 21, 2026
Merged

feat(#121): contaminated shared-context experiment#146
Agastya191 merged 1 commit into
mainfrom
feat/contaminated-shared-context

Conversation

@Agastya191

Copy link
Copy Markdown
Collaborator

What

Adds the contaminated shared-context experiment (#121), which injects a spurious signal into the shared workspace itself instead of through a peer's turn, isolating the shared-context channel from peer conformity. It adds:

  • a board_seed param on run_committee (blackboard.py) that drops an artifact -- a pre-screen flag, a "prior similar cases were all X" note, a leaked proxy metric -- onto the board before the member loop, not attributed to any agent, kept out of the committed map, and consuming no member slot (so seed_turn indexing is untouched). A board seed attributed to a committee member now raises, since that would leak a peer.
  • a context flag on schema.Turn marking that artifact as ambient workspace context, so _visible_turns keeps it visible even in isolated mode -- isolation is meant to hide peers, not the shared workspace, which is the whole point of the isolated control.
  • contaminated_context.py, which runs the 2x2 (present/absent x shared/isolated) and reports the anchoring: present minus absent is the artifact's pull, and shared minus isolated (shared_amplification) is the extra pull only the shared channel adds. The twin condition is held constant across all four arms on purpose, so the only thing differing between present and absent is the artifact, not the AgentView.condition label.
  • composition with the existing referee (score_shortcut reliance) and the blind-metric probe (blind_metric_uptake, with the artifact as the hidden proxy), rather than reinventing either.

Plain run_committee behavior is unchanged when board_seed isn't passed -- the new context field defaults to False and every existing path stays byte-identical, same as before.

Note: this touches the shared Turn schema (a new context field) rather than overloading the existing seeded flag, because the cascade's seed_turn is a peer utterance that must stay hidden in isolated mode while a workspace artifact must not -- flagging in case you'd rather key isolated-mode visibility off an agent_id convention instead. One thing I left alone: the two visibility arms share a run_id (shared isn't in it), same as run_cascade already does, and meta["shared"] disambiguates them -- say the word if you'd rather I make run_id unique.

Testing

tests/test_contaminated_context.py:

  • the 2x2 runner produces the anchoring ordering (shared > isolated > absent) on a small set
  • referee flags reliance on the artifact, and blind-metric composition gives a positive uptake delta when the artifact leaks a proxy
  • a condition-keying backend produces zero anchoring, proving the condition label can't confound the contrast
  • ground truth stays untouched (the absent arms recover the correct answer), the run is deterministic, and the guards fire (empty cases/committee, agent-id collision, duplicate case_id, None implied answer, implied == correct)

board_seed contract tests appended to tests/test_blackboard.py:

  • the artifact lands before the first agent and is visible in isolated mode too
  • it never leaks into committed, never shifts seed_turn's slot index, and neither mutates the caller's Turn nor aliases it across two runs
  • a board seed or pre_hook context turn attributed to a member is rejected

Suite: 492 -> 519 passing locally, offline mock path only (couldn't run ruff check in this environment, so it still needs a local lint pass before merge).

Inject a spurious artifact into the workspace itself (not a peer's turn) and
measure whether the committee anchors on it, isolating the shared-context
channel from peer conformity.

- blackboard: add board_seed to run_committee (ambient context artifact placed
  before the member loop, not attributed to any agent, kept out of committed,
  consuming no member slot); reject member-attributed context turns; make
  context turns visible in isolated mode
- schema: add Turn.context flag for ambient workspace artifacts
- contaminated_context: 2x2 present/absent x shared/isolated runner reporting
  the anchoring effect and shared_amplification, holding the condition constant
  across arms; compose the referee and blind-metric probe
@Agastya191
Agastya191 force-pushed the feat/contaminated-shared-context branch from a53d49c to bacfaf1 Compare July 21, 2026 01:39
@sebasmos

Copy link
Copy Markdown
Member

Reviewed the design (not merging, this one is yours @Agastya191). This is a clean and valuable addition: board_seed as ambient workspace context, flagged context=True and visible even in isolated mode, is exactly the right way to separate the shared-context channel from peer conformity. It also makes a first-class harness feature of what our clean-A result already saw empirically, a wrong "system pre-screen" flag is adopted by the weaker tier ~0.33 of the time (control 0.00) while a bare peer is resisted (see #141).

Two notes:

  1. This and feat: reasoned rationale for cascade seed turns (addresses #115) #149 both extend run_committee's signature, so whichever lands second will need a trivial rebase, worth coordinating with @MohShahin.
  2. Please confirm the context=True turn is excluded from the per-agent committed map and from the conformity / cascade-onset counters, so it can never be counted as a member turn.

Nice work.

@Agastya191

Copy link
Copy Markdown
Collaborator Author

confirmed context = True is excluded so it never can be counted as a member turn. Coordinated With Mohammad as well. Merging to main now.

@Agastya191
Agastya191 deleted the feat/contaminated-shared-context branch July 21, 2026 15:20
@sebasmos

Copy link
Copy Markdown
Member

Thanks @Agastya191, that resolves my note (the context=True turn excluded from the committed map and the conformity / cascade-onset counters). Good to have coordinated with @MohShahin on the run_committee signature. Go ahead and merge.

sebasmos pushed a commit that referenced this pull request Aug 4, 2026
…ntext

feat(#121): contaminated shared-context experiment
sebasmos added a commit that referenced this pull request Aug 4, 2026
…corrected (closes #131)

Rebased fresh off current main (the original branch predated #142/#143/#146/#148/#154/#157/
#161/#219 and would have deleted all of that merged work if landed as-is).

Addresses @Agastya191's review on #159:
- flash_lite_reference was hardcoded to scale_c's PRE-parser-fix numbers (0.33/0.51 at n=150,
  p<1e-4). Now reads dynamically from --scale-c-summary (scale_c's committed summary, PR #141)
  so it cannot drift out of sync with a future fix there. Regenerated: 0.729/0.847 at n=85,
  p=0.041.
- Also caught in the same pass: the docstring and README's own flash numbers were ALSO stale
  (60 hard cases, generic 0.10/anchored 0.083) versus the actually-committed, already-corrected
  summary (28 hard cases, generic 0.714/anchored 0.679). Rewrote both to match the real data
  and restated the finding precisely: both tiers conform substantially to a bare peer (~0.71-0.73),
  but only flash-lite's conformity climbs further under a case-anchored rationale (+0.12, p=0.041);
  flash's does not move. The anchoring lever is model-dependent, not conformity itself.

Verified end-to-end: keyless reproduction confirmed with the key unset (new_api_calls_this_run: 0,
n_hard_cases: 28, matches exactly), ruff clean, 625 tests pass on the rebased base, no hardcoded
personal paths, 0 em dashes.
sebasmos added a commit that referenced this pull request Aug 4, 2026
…referee) (#162/#163/#164/#165)

Provenance manifest via the registered nih_cxr14 adapter with sha256 checksums per used
image (#162); solo cue-susceptibility across cable/corner-tag/watermark/laterality, watermark
+0.11 above the temperature-resampled noise floor (#163); the watermark cascade, shared adopt
0.97 vs isolated 0.34, contagion +0.63 (#164); a deployable referee with no privileged knowledge
(transcript + one private re-read) catching peer-driven adoptions at P/R 0.86, FPR 0.23 versus
a naive conformity gate's FPR 0.92 (#165).

Branched fresh off main (not off the stale results/cascade-at-scale, which predates #146/#148/
#154/#157/#161 and would delete that work if merged) and reuses the already-committed,
already-reviewed solo/cascade logic and numbers from that branch. All four runners are fully
keyless-reproducible from the committed img_cache.jsonl (new_api_calls_this_run: 0, verified).
sebasmos added a commit that referenced this pull request Aug 4, 2026
…loses #108)

Rebased fresh off current main (the original branch predated #142/#143/#146/#148/#154/#157/
#161/#219 and would have deleted all of that merged work if landed as-is, same stale-base
issue as #143/#150/#141/#159).

Addresses @Agastya191's review on #158:
- README/PR body had stale pre-fix numbers (accuracy 0.29/0.14, flash flip 0.76 vs 1.00 at
  p=9.8e-5) versus the already-corrected committed summary (accuracy 0.89/0.78, flash flip
  0.079 vs 0.455 at p=3.3e-3, flash-lite 0.090 vs 0.636 at p=4.4e-7). Rewrote to match.
- ever_flipped collapsed the three cues into a per-case boolean that doesn't match the stated
  headline flip rate (0.063/0.117). Now reports both per_record_flip_rate (the headline
  quantity) and ever_flipped_rate, clearly labeled and distinguished.
- Grading-standard mismatch: clean_correct was imported from the prior solo pipeline rather
  than re-verified. Now re-graded by exact match against ground truth, the same standard as
  options_only_correct. Re-graded values are identical to the imported ones (0.89/0.78), so no
  actual discrepancy, but the standard is now transparent and consistent.
- Cache race: _Cache.complete checked the store under lock but called the API outside it, so
  two threads could both miss the same key and both append a duplicate entry (confirmed: the
  committed cache had exactly one). Lock now spans the full miss-check-fetch-store sequence.
  Cache also pruned from 6054 shared-cache entries down to the 400 this script actually
  requests (100 cases x 2 models x 2 probes).

Verified end-to-end: keyless reproduction confirmed with the key unset (new_api_calls_this_run:
0, all numbers match exactly), ruff clean, 625 tests pass, no hardcoded personal paths, 0 em
dashes.
sebasmos added a commit that referenced this pull request Aug 4, 2026
Rebased fresh off current main (the original branch predated #142/#143/#146/#148/#154/#157/
#161/#219 and would have deleted all of that merged work if landed as-is, same stale-base
issue as #143/#150/#141/#159/#158).

Addresses @Agastya191's review on #155:
- README/PR body carried pre-fix numbers (shared 0.175/0.15/0.175/0.15/0.15, p=1.0) versus the
  already-corrected committed summary (0.275/0.3/0.275/0.325/0.325, gained 3 lost 1 p=0.625).
  Rewrote to match and restated the finding as no significant compounding at p=0.625.
- Real bug: the isolated holdout's prompt was byte-identical to the bare prompt every round
  (board text was built only from peer votes, which are never visible in isolated mode), so at
  temperature 0 it deterministically reproduced round 1's answer every round. Isolated adoption
  was 0 by construction, not by measurement. Fixed by reminding the isolated holdout of its own
  previous-round answer (visible to it in isolated mode; only peer turns are hidden), so the
  prompt genuinely differs round to round. The shared arm's board construction is UNCHANGED, so
  the shared numbers are identical to what was already reviewed; only the isolated arm is new.

Result: isolated adoption is now a genuine (if still near-floor) measurement: 0.0/0.025/0.0/
0.025/0.0, at most 1 of 40 cases per round, rather than a trivial constant 0.0.

Verified end-to-end: ran for real with the isolated fix (206 new calls), then re-verified keyless
reproduction with the key unset (new_api_calls_this_run: 0, exact match on both curves), ruff
clean, 629 tests pass, no hardcoded personal paths, 0 em dashes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants