Skip to content

Reject stale completed one-head nanoGPT runs - #41

Merged
charlesmartin14 merged 14 commits into
mainfrom
agent/fix-nanogpt-stale-completion
Aug 9, 2026
Merged

Reject stale completed one-head nanoGPT runs#41
charlesmartin14 merged 14 commits into
mainfrom
agent/fix-nanogpt-stale-completion

Conversation

@charlesmartin14

Copy link
Copy Markdown
Member

Root cause

The one-head nanoGPT runner treated the existence of run_complete.json as sufficient proof that a result could be reused. That shortcut ran before the current prepared-data identity and protocol fingerprint were calculated. run_is_complete() likewise checked only completed: true.

A stale directory from an older protocol, different prepared corpus, changed optimizer profile, changed model/training configuration, or missing terminal artifact could therefore be silently accepted.

Changes

  • add a strict completed-run validator for the one-head nanoGPT suite
  • load and verify prepared-data metadata before deciding whether a completed run is reusable
  • calculate the current optimizer profile, training horizon, and protocol fingerprint before the completion shortcut
  • validate completion and manifest fingerprint, optimizer, seed, and total-step identity
  • validate final and validation-selected test-result steps
  • validate metrics and epoch-metrics coverage, monitoring-only test policy, spectral coverage, six-matrix inventory, and the exact epoch-checkpoint paths recorded by the run
  • validate latest, final, and validation-best checkpoint fingerprints, optimizer/seed identity, steps, and best-validation state before training skips a completed run
  • make run_is_complete() use artifact validation instead of trusting a Boolean marker
  • add regression tests for valid reuse, protocol drift, missing artifacts, and stale checkpoint fingerprints

Behavior

A compatible and complete result is reused without initializing the training device or model. A stale, incomplete, corrupt, or ambiguous result fails explicitly and instructs the caller to use a new result directory or rerun with explicit overwrite. Existing artifacts are never silently replaced.

Validation

  • git diff --check
  • python -m compileall -q baseline/nanogpt_one_head/src baseline/nanogpt_one_head/tests
  • pytest -q baseline/nanogpt_one_head/tests17 passed
  • pip check — no broken requirements

No optimizer mathematics, hyperparameters, model architecture, notebook source, committed output notebook, or experiment result was changed. This PR is independent of PR #40.

@charlesmartin14
charlesmartin14 marked this pull request as ready for review August 9, 2026 21:36
@charlesmartin14
charlesmartin14 merged commit 4e713b7 into main Aug 9, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant