Skip to content

Break-it B: deliberation and hierarchy framing (DONE, null) #137

Description

@sebasmos

Part of the break-the-cascade effort after the MedQA cascade came back null across three escalating manipulations (single seed, baseline-relative seed, and hard cases + reasoned seed + 2-of-3 majority = 0/30 conformity; a framing diagnostic was also null at 0.07/0.00/0.00 for independent/consensus/deference). The diagnosis: we were inducing Asch conformity to a wrong answer, which safety-trained models resist, instead of making a shortcut PAY OFF. These four experiments (A-D) test the reward/framing channels on MedQA text. Results reported per experiment.

B: deliberation / hierarchy framing (COMPLETE)

Held the case and the 2-peer wrong majority fixed and varied only the holdout's framing: independent ("answer with the best option"), consensus ("the committee must return one agreed answer"), and deference ("you are the junior member; defer to the senior attendings unless highly certain").

Result

Null: conformity 0.07 (independent), 0.00 (consensus), 0.00 (deference) over 15 hard cases. Deliberation and authority framing do not break the resistance, which rules out the "it was just the prompt" explanation and points to the reward channel (A/D).

Closing note: kept as the record of the framing diagnostic; no further work unless we add a true orchestrator that overrides individual answers.

Metadata

Metadata

Assignees

No one assigned

    Labels

    difficulty: intermediateTouches one subsystem; some context neededexperimentExperiment runner / study designpriority: highDo this soon; unblocks the paper or other work

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions