[Planning] Plan of initial replay scheduler

[stevenshinechen]

Create on a block diagram level design of the initial replay scheduler and agree on the main building blocks and core ideas that we'll be using

(We can add more approaches after the initial replay scheduler is up)

Ideally the initial replay scheduler will be as simple as possible and only contain core ideas.

[tokenbender]

objective

• Deliver a minimal replay scheduler for GRPO that lets us prioritize prompts/domains using a single control signal (e.g., pass-rate EMA) without touching the core trainer.
• Produce a block-diagram view of the scheduler so we agree on the core components before layering more elaborate strategies or metrics.

shared ideas on discord so far

• Keep the scheduler strategy-agnostic: each training run should be able to swap between domain-based replay, "maximum surprise" sampling, random baselines, etc., by plugging in different scorer functions.
• Represent every prompt in the JSONL with an include flag; dataset loader ingests all prompts and defers to the scheduler to decide which ones stay enabled for the next batch.
• House each scheduling heuristic in a small module/function that consumes recent rollout signals (pass/fail, TD error, reward gaps) and returns priority scores, so experimentation is fast.
• Start simple with a priority queue over prompts, potentially seeding with random priorities or fixed order, then evolve toward signal-driven sampling (e.g., high TD error, gap between max vs mean reward).
• Persist lightweight state (e.g., per-domain/prompt EMAs) so we can resume and compare prioritized vs uniform baselines during toy runs without losing history.
• Aim first for an end-to-end prototype that runs fast enough to iterate; we can optimize throughput once the plumbing is validated.