v0.2.0 — Cross-process evidence and no-progress validation

v0.2.0 — Cross-process evidence and no-progress validation

v0.2.0 turns the lab's evidence method into a bounded workflow that another
engineer can run without a GPU. It adds collect, offline verify, and an
installed replay command while preserving undetermined as a valid outcome
when the evidence cannot support a stronger claim.

Five-minute replay

python -m pip install .  # installs distribution vllm-runtime-dfx-lab
vllm-dfx replay results/vllm-zmq-backpressure-stage1-r3-20260916

The replay verifies the frozen #53859 four-cell evidence, file identities,
health-green no-progress claim, blocking-stack claim, fix-arm progress, and the
measured four-batch event-loss trade-off. It does not rerun the GPU experiment.

Delivered in v0.2.0

• bounded collect bundles with operator-supplied PID identity, endpoint
  privacy, and always-closed observation windows;
• independent server-counter and opt-in client-request progress producers;
• explicit demand evidence and producer-conflict reporting;
• five closed verdicts with deterministic precedence: process_missing,
  health_lost, progress_observed, alive_health_ok_no_progress, and
  undetermined;
• offline, fail-closed verdict recomputation from closed-shape JSON;
• typed optional py-spy availability and producer identity while raw stacks
  remain private and non-decisional;
• a machine-checked registry that classifies every public leaf field as
  decisional or explicitly non-decisional;
• a compatibility projection that reports evidence missing from the legacy
  #53859 result instead of inventing native v0.2 observations; and
• vllm-dfx replay as the installed no-GPU entry point.

Evidence-backed results

• Lab-originated PyTorch #196968
  established that a missing Flight Recorder dump did not mean a missing rank;
  PR #197232 proposes the C++
  fix. Both were open on 2026-09-21.
• The lab independently validated reported
  vLLM #53859 and proposed
  PR #53883: /health
  remained 2xx while EngineCore token progress stopped under deterministic
  event-queue backpressure. This was not a lab-originated bug.
• Lab-originated PyTorch #196996
  reduced an apparent distributed hang to a single-GPU FSDP2 mixed-gradient
  dtype correctness failure.

Publication boundary

The complete #196968 case study is not part of this release. Its publication
gate requires an explicit upstream outcome for #197232: merge, explicit design
acceptance with another landing path, or explicit rejection/supersession with a
documented reason. An open PR, passing CI, bot labels, or silence is not an
upstream outcome.

#49869 is an independent upstream contribution and is not a lab discovery.
#52178 is a separately found lifecycle bug for which the lab supplied
process-level validation.

Known boundaries

This remains an alpha research tool, not a production monitor, automatic
process/rank discovery system, general cross-rank or cross-host joiner,
native-state classifier, remediation controller, or automatic root-cause
classifier. Stack availability and producer implementation cannot change a
verdict. Endpoint-only flat progress remains undetermined because repeated
2xx responses do not establish process liveness.

Immutable release references

• v0.2.0 source tree
• v0.2 collect/verify contract
• Field-role audit
• Published #53859 replay evidence

Release artifact SHA-256 values and clean-install transcripts are published
alongside the release assets built from the exact tagged commit.

v0.1.0-alpha.4

v0.1.0-alpha.4

This release adds the first lab case that was not designed by the project: a
four-GPU reconstruction of the known FSDP2 conditional-parameter hang tracked
by pytorch/pytorch#158719 and pytorch/torchtitan#2747.

Delivered evidence

• One host with 4 x RTX 4090, PP=2 and DP=2.
• Three PyTorch 2.11 DebugLevel.DETAIL oracle trials reported a stable
  _REDUCE_SCATTER_BASE input-shape mismatch ([1024] versus [960]).
• Three normal hung runs with automatic ProcessGroupNCCL Flight Recorder
  capture reconstructed the same semantic mismatch in 3/3 trials.
• The public verifier checks the expected stage-0 DP group [0, 2], uniform
  dtype family, completed collectives before the hang, and every retained
  lifecycle cleanup record.
• Raw stderr, prepared source and private audit material are excluded from the
  public result directory.

Decision boundary

This is a known-answer reconstruction, not discovery of an unknown root cause,
and the campaign remains short of GO. A same-version no-divergence control is
still required. The attempted PyTorch 2.13 opt-in control hit a separate
mixed-gradient-dtype assertion in all three trials and was recorded as blocked,
not counted as evidence of success. Protocol revision 2026-09-12.6 was
untracked at execution time, so its freeze timing remains author-declared.

The result validates an external c10d-visible evidence workflow. It does not
validate the proposed vLLM EngineCore incident Snapshot or establish its
diagnostic utility.

Review entry

• Review entry
• GPU result
• Derived-only public evidence

Wheel SHA-256:

67fb31e293901b3e9f6159ff0d5edaa9b9f916bad3d7862697b01cb7f5fed0e4

Derived-only evidence archive SHA-256:

a7d0e4d3c90324f7e5d9c35f9f3e25ef43d7da1a56c65b4bb514f35172e820f8

v0.1.0-alpha.3

v0.1.0-alpha.3

This release makes the public validation path reproducible from a clean clone
and restores machine-checked requirement-to-test traceability.

Corrections

• The README now installs the dev extra before running the test suite, so the
  documented clean-clone path includes jsonschema and Ruff.
• The installed schema path now uses
  share/vllm-runtime-reliability-lab/schema.
• Functional and safety requirements have stable FR-* and SR-* identifiers.
• tests/cases.json maps every requirement to one or more concrete tests.
• test_testplan.py checks bidirectional requirement coverage, non-empty
  mappings, and the existence of every referenced test method.

Validation

The documented clean-clone path passes 33 tests on the release source, with the
POSIX permission test skipped on Windows. CI also runs lint, formatting,
compilation, CLI smoke tests, and the suite on Python 3.10, 3.12, and 3.13.

The runtime contract and schema are unchanged from alpha.2. The RTX 4090 fatal
matrix remains documented in
results/gpu-20260909-alpha2/VALIDATION_SUMMARY.md.

Wheel SHA-256:

15b812029a382b8feeb5699daff186e389e9483bc1fbd861268d16f89590d5c4

v0.1.0-alpha.2

v0.1.0-alpha.2

This corrective alpha preserves the external observer's evidence boundary: it
does not infer target package versions from the recorder's Python environment.

Correction

• vllm_version and torch_version now remain null by default.
• Operators may supply exact observed-server values with
  --target-vllm-version and --target-torch-version.
• Two regression tests cover both unknown and explicitly supplied versions.

All alpha.1 functionality remains: closed schema, bounded cadenced history,
privacy canaries, four-file rotation, fail-open persistence, fake-service tests,
and Markdown summaries.

Fresh RTX 4090 validation covers intentional SIGTERM, three EngineCore
SIGKILL trials, three controlled execute_model CUDA OOM trials, an
unavailable writer directory, and a recorder-disabled control. See
results/gpu-20260909-alpha2/VALIDATION_SUMMARY.md
for the exact environment, exclusions, and remaining gates.

Wheel SHA-256:

fb5ac984e19347fbb304c41ae3da8695e0a8a1c257b4df1b8f691982b3e60ce6

v0.1.0-alpha.1

v0.1.0-alpha.1

This first public alpha turns the existing vLLM reliability experiments into a
reproducible, installable external recorder with a deliberately narrow trust
boundary.

Included

• a closed external-runtime-observation-v1 JSON contract;
• bounded, independently cadenced health, metrics, process, and GPU sampling;
• process-exit, health-loss, KV-pressure, and preemption triggers;
• 256 KiB artifact cap, four-file rotation, atomic replacement, and POSIX
  0600 permissions;
• privacy canaries and per-process HMAC incident identifiers;
• fail-open collection and persistence behavior;
• a fake vLLM HTTP service and 30 CPU tests;
• reviewed summaries of earlier RTX 4090 and TP=2 experiments.

Validation completed for this tag

• clean editable install on Windows;
• Ruff lint and format checks;
• 30 unit/local-service tests passed, with the POSIX permission test skipped on
  Windows and scheduled for Linux CI;
• end-to-end fake-service health-loss capture and Markdown summarization;
• published example validated against the Draft 2020-12 JSON Schema.

Not established by this alpha

Fresh GPU validation, production diagnostic utility, DP/NCCL behavior,
long-duration stability, and automatic remediation remain out of scope. See
TEST_PLAN.md for the next evidence gates.