Skip to content

Imaging blind-metric substitution probe on NIH ChestX-ray14 #170

Description

@sebasmos

Context

The text lane's blind-metric substitution probe (#157, blind_metric.py) measures whether a model drifts toward a hidden-rubric decoy it would not pick unprompted. The imaging lane does not yet have this probe. This issue adds it for NIH ChestX-ray14.

What to run

Port the blind-metric design to imaging: introduce a hidden scoring rubric that rewards a specific wrong finding-read, and measure whether the holdout drifts toward that decoy on the cued images, versus its unprompted baseline read.

Done when

Committed blind-metric summary (decoy uptake, and whether drifters name the rubric, mirroring the text lane's finding that all drifters there self-declared).

Related

Depends on #162 (manifest). Mirrors the text lane's #157 design for the imaging lane.

Metadata

Metadata

Assignees

Labels

difficulty: intermediateTouches one subsystem; some context neededexperimentExperiment runner / study designpriority: mediumValuable, not on the critical path

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions