Feat: Lightweight downstream Zero-Shot Evaluation (Tiny-Eval)

[aniruddhaadak80]

Problem

The success metric for the agent is currently pure val_bpb (validation bits-per-byte). While a great proxy, isolated cross-entropy loss can sometimes disguise models that have collapsed in specific linguistic capabilities or are cheating via strange formatting.

Proposal

Integrate a very lightweight zero-shot evaluation loop at the end of the TIME_BUDGET. This could test against miniature versions of MMLU, Hellaswag, or a simple QA format. Emitting a real "accuracy" percentage in results.tsv alongside val_bpb will give the AI a secondary, qualitative anchor to ensure its architectural hacks are improving actual model reasoning.

[MustakimShaikh01]

This makes sense. Relying only on val_bpb could hide regressions in reasoning or language understanding.

Adding a small zero-shot evaluation (even with tiny subsets of MMLU or QA datasets) would provide a more interpretable metric like accuracy. That would make it easier to verify whether architectural changes are improving real model capabilities rather than just lowering cross-entropy.

[svlandeg]

On nanochat, val_bpb proved to be more robust than the CORE score, which actually takes quite a while to compute. With just a 5 minute compute window, any eval you'd want to set up will be even more brittle. I don't think the metric needs to be more "interpretable", it needs to be as stable/smooth as possible, which val_bpb should be, as I understand it.

[MustakimShaikh01]

That’s a fair point about stability, especially with such a short compute window. If val_bpb is already giving a smooth signal for optimization, keeping the evaluation lightweight makes sense.

One possible compromise could be running the zero-shot checks only periodically (for example every N runs or only on the final checkpoint) rather than within the strict 5-minute loop. That way val_bpb remains the primary optimization metric, while a small downstream sanity check could still help detect cases where improvements in compression don’t translate to actual language understanding.

But I agree that if the additional eval becomes noisy or expensive, the benefit might not justify the overhead.