Measure whether random eviction preserves answer quality under controlled cache budgets. This is required before a performance-focused launch.
Acceptance criteria:
- Add a CLI that evaluates dense and RandKV generation over the same immutable prompt set.
- Start with one openly licensed, exact-match reasoning dataset and document its name, split, version/revision, and scoring normalization.
- Record exact model revision, generation parameters, device, seed, budget, buffer, per-example outputs, and aggregate accuracy in machine-readable JSON.
- Force identical prompt formatting and generation limits across modes.
- Include a small local fixture so evaluator/scorer tests run offline in CI; the real dataset must not be downloaded in CI.
- Keep evaluation dependencies optional and outside the core runtime install.
- Report failures and invalid generations; never silently drop examples.
Please propose the initial dataset and output schema in a comment before implementation.
Measure whether random eviction preserves answer quality under controlled cache budgets. This is required before a performance-focused launch.
Acceptance criteria:
Please propose the initial dataset and output schema in a comment before implementation.