fix(storage#792): reject invalid checkpointing --num-processes early; clarify §4.3.5 message

[FileSystemGuy]

Closes #792.

The reporter's scenario

Ran mlpstorage closed checkpointing run file --num-processes 408 for
llama3-405b (51 hosts × 8 ranks). Rules.md §4.6.1 requires exactly 8
(subset per §4.3.5) or 512 (full) processes for 405b in CLOSED. 408 is
neither. The tool accepted the run, DLIO wrote and read 10 checkpoints,
and mlpstorage validate then reported the confusing error:

[ERROR] [4.3.5 checkpointSubsetRunValidation] ...: subset run
requires exactly 8 accelerators, got 408

Two root causes

1. No fail-fast gate. CheckpointingRunRulesChecker
(mlpstorage_py/rules/run_checkers/checkpointing.py) only checked
benchmark_type and model. num_processes was never validated, so a
misconfigured CLOSED run consumed hours of I/O time before eventual
rejection at validate.

2. Misleading post-hoc message.
CheckpointingBenchmark.add_checkpoint_params in
mlpstorage_py/benchmarks/dlio.py:1104-1111 auto-labels any downscaled
run (num_processes < ClosedGPUs) with checkpoint.mode="subset" so
DLIO applies the model.parallelism.data override. §4.3.5 then saw the
subset label and reported the accelerator count against the subset
rule — implying the user had explicitly opted into subset mode.

Fix

• mlpstorage_py/rules/run_checkers/checkpointing.py — new
  check_num_processes. Returns None for the two valid CLOSED forms
  (LLM_SUBSET_PROCS = 8 or ClosedGPUs per model), Issue(OPEN) when
  the count is a positive multiple of TP*PP (§4.6.4 — prompts the
  user to re-run under the open positional), and Issue(INVALID)
  otherwise. The INVALID message names both valid CLOSED forms so the
  user sees which knob to turn. Silent-pass for non-LLM models (that
  surface is owned by check_model).

• mlpstorage_py/submission_checker/checks/checkpointing_checks.py —
  §4.3.5 subset-run message now names the specific model's two valid
  CLOSED forms (e.g. "requires exactly 8 (subset) or 512 (full)") and
  cites the auto-labeling behavior. Users with existing invalid results
  get a clearer message. Genuine 8-accelerator subset runs and
  non-LLM/unknown-model paths keep the original message.

Tests

mlpstorage_py/tests/test_issue792_checkpoint_num_processes.py — 13 new
tests:

• Pre-run gate — full CLOSED for all four models (8, 64, 512, 1024);
  subset (8) for 70b/405b/1t; the exact 408×405b reporter case →
  INVALID with both 8 and 512 in the message; 256×405b → OPEN with
  re-run instruction; non-LLM model → silent-pass; method-discovery
  guard.
• Clarified §4.3.5 message — asserts both 8 and 512 appear for
  the 408×405b case; genuine 8-accelerator subset run still passes
  (locks the change to the misconfig path only).

Test plan

• uv run pytest mlpstorage_py/tests/test_issue792_checkpoint_num_processes.py — 13/13 pass
• uv run pytest mlpstorage_py/tests tests/unit — 3611 passed, 1 skipped (unchanged from baseline)
• Manual: attempt mlpstorage closed checkpointing run file --num-processes 408 --model llama3-405b … and confirm the run aborts pre-DLIO with the new error message naming both valid forms.
• Manual: re-validate an existing invalid results dir (like the reporter's) and confirm the new §4.3.5 message references both valid forms and storage#792.

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅