Skip to content

Test scoring gates across independent item-selection schedules #411

Description

@jamiepratt

The v2 tuning, held-out, and stress phases each generate one deterministic 160-item balanced schedule, then reuse it across every cell and replicate in that phase. This gives paired, reproducible model comparisons, but the reported gate metrics are conditional on one schedule per phase. The expanded v2 article now discloses this limitation.\n\nBefore promoting a future continuous-frequency scorer:\n\n- run supported and stress gates across multiple independently seeded balanced schedules;\n- separate within-schedule replicate variation from between-schedule variation;\n- report how aggregate and worst-cell coverage, MAE ratios, and quiz length vary by schedule;\n- version the schedule seeds/count and numerical shortcut in result artifacts;\n- preserve an untouched held-out phase and do not retune from its results;\n- update the executable article and parity/runner tests with the resulting decision.\n\nThis is a validation follow-up, not a reason to reopen or retune continuous-pair-frequency-logistic-v2.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions