Validation §4.7.1 `checkpointCacheFlushValidation` unfairly charges the write phase's final multi-node collection to the failover-callout budget

[wolfgang-desalvador]

Description

The §4.7.1 failover-callout gap check measures the handoff between the
write-phase and read-phase invocations of a split-mode checkpointing run:

gap = read.metadata.invocation_start_time − write.summary.end_time

and fails the submission when gap > 30s.

The read-side origin already excludes per-invocation framework startup
(#696 / #714). The write-side origin does not. write.summary.end_time is
the end of the write benchmark's timed section, but the write invocation then
performs a final, multi-node data collection (the end cluster snapshot)
after the timed section closes. On large topologies that collection takes
tens of seconds, keeps the write nodes busy, and delays the read invocation
from launching — yet all of it is charged to the 30-second failover-callout
budget.

The result is a false failure on a legitimate split-mode submission: the
real failover handoff is within budget, but the reported gap includes teardown
work that has nothing to do with the failover callout.

Steps to reproduce

1. Run a split-mode (two-invocation) closed checkpointing submission on a large
   node count, so the write phase's final cluster collection takes >30s.
2. Run mlpstorage validate <submission_dir>.
3. Observe the §4.7.1 failure.

Actual behavior

[ERROR] [4.7.1 checkpointCacheFlushValidation]
  .../results/<STORAGE-CONFIG>/checkpointing/llama3-1t:
  failover-callout gap 64.4 seconds exceeds 30-second limit
  (write end=2026-07-13T06:36:25.548721, read invocation_start=2026-07-13T06:37:29.960616)

The write-phase logs show the write nodes are still busy with the final cluster
collection well past write.summary.end_time. The genuine inter-phase handoff —
from the completion of that collection to the read invocation start — is
comfortably within 30s.

Expected behavior

The failover-callout budget should be measured from the point at which the write
phase actually releases the nodes (i.e. after its post-benchmark collection),
not from the end of the timed section. Legitimate split-mode submissions whose
real handoff is <30s should pass.

Root cause

The gap origin on the write side is the benchmark summary end, which precedes
the post-benchmark collection window. That collection is genuine, unavoidable
write-side teardown — directly analogous to the read-side framework startup
already excluded in #696 / #714 — but it is currently charged to the submitter.

Proposed fix

Prefer the earliest post-benchmark collection_timestamp recorded in the
write-phase metadata.json as the gap origin, falling back to
write.summary.end_time only when no post-benchmark collection timestamp is
present.

• _iter_collection_timestamps(metadata) — recursively yields every
  collection_timestamp in the nested write-phase metadata.
• _oldest_final_collection_timestamp(metadata, after_iso) — returns the
  earliest collection_timestamp at or after write.summary.end_time,
  normalising the trailing Z (UTC designator) so it compares naively against
  the summary timestamps (both are wall-clock instants on the write host).
  Start-of-run collection timestamps (before the summary end) are ignored so
  the origin is never moved earlier than the summary end.

Choosing the oldest (earliest) qualifying timestamp is deliberately
conservative: it credits the write phase for its final-collection window while
still charging as much of the inter-phase gap to the write side as the evidence
allows.

The check now also emits an INFO line for every write/read pair with the
computed gap and the chosen origin, so the measurement is transparent whether
the pair passes, warns, or fails:

[INFO] [4.7.1 checkpointCacheFlushValidation] failover-callout gap 27.0s
  (write final-collection timestamp=2026-07-13T06:37:03,
   read invocation_start=2026-07-13T06:37:29.960616,
   write_ts=20260713_061954, read_ts=20260713_063729)

Worked example

Quantity	Value
write.summary.end_time (timed section ends)	2026-07-13T06:36:25.5
earliest post-benchmark collection_timestamp (nodes released)	2026-07-13T06:37:03
read.metadata.invocation_start_time	2026-07-13T06:37:29.9
gap charged before fix (read_start − summary_end)	64.4s → FAIL
gap charged after fix (read_start − collection_end)	27.0s → PASS

The ~37s of write-side final collection is no longer counted against the 30s
failover-callout budget.

Backward compatibility

• Results dirs whose write-phase metadata carries no post-benchmark
  collection_timestamp fall back to write.summary.end_time (previous
  behaviour, unchanged).
• The existing read-side fallback (read.summary.start_time when
  invocation_start_time is absent, downgraded to a warning) is unchanged.

Related

• §4.7.1: mandated write/read split can't meet its own ≤30s gap (read-invocation startup ~46s); submission_checker accepts combined mode — please reconcile #696 / fix(#696): §4.7.1 gap origin = read invocation start, not summary start #714 — read-side framework-startup exclusion (same class of unfair
  charge, applied to the read origin).

[wolfgang-desalvador]

@FileSystemGuy

[wolfgang-desalvador]

Thank you for the fix @FileSystemGuy .

Can I ask you if we can include #783 PR also?

It is based on your changes and allows to have a fallback for runs previous this fix. This would allow us to avoid re-run of time expensive and resource consuming cases. It uses the collection_timestamp (the oldest one) as reference

[FileSystemGuy]

@wolfgang-desalvador I haven't looked closely at this one, do you still want it or was it superseded by PR#787 that we just merged?

[wolfgang-desalvador]

This is fixed