Skip to content

New Skill: hrr-replay-analysis - #117

Open
andraghetti wants to merge 2 commits into
amd:mainfrom
andraghetti:hrr-replay-analysis
Open

New Skill: hrr-replay-analysis#117
andraghetti wants to merge 2 commits into
amd:mainfrom
andraghetti:hrr-replay-analysis

Conversation

@andraghetti

@andraghetti andraghetti commented Aug 5, 2026

Copy link
Copy Markdown

Summary

Adds hrr-replay-analysis, the skill the catalog has advertised as _planned_ since #94. It replays a HIP Record and Replay (HRR) archive on the host GPU and reports what went wrong as a structured finding: fault class, fault address, failing event index, and the implicated kernel.

An HRR archive is a deterministic record of a real HIP workload, so a customer failure can be reproduced from the recording alone, without their application, source or data. The skill stops at the finding and hands off; kernel patching, rocgdb and fix verification are deliberately out of scope.

Layout

skills/hrr-replay-analysis/
├── SKILL.md
├── reference.md
├── examples.md
├── skill-card.md
├── scripts/run_hrr_replay.sh
├── scripts/analyze_replay_finding.py
└── evals/
    ├── evals.py
    └── fixtures/

Testing

Structural gates:

  • ./.github/scripts/check.sh: 7 skills, 0 errors, Cursor and Codex manifests in sync
  • Offline link check over the skill's markdown, 0 errors
  • SkillSpector scan: no HIGH or CRITICAL findings (2 MEDIUM for subprocess, both advisory)

End to end on a gfx950 host, against three real archives:

  • Clean workload. An offline vLLM inference capture (185,469 events, 13,233 kernels, 33 code objects, 2 threads) replays clean in 39 s with all 32 device-to-host checks exact, and is reported as a pass.
  • Crashing workload. A capture whose workload faults reproduces the memory access fault on replay, and the finding carries the fault class, the fault address and the faulting kernel.
  • Unreadable archive. An archive whose format version the playback build cannot read is reported as unreadable, not as a workload failure.

That validation changed the skill. The clean workload was reporting a hipBLASLt GEMM as its kernel, matched out of the archive's own kernel listing, which next to a pass reads as a culprit; a clean replay now implicates nothing. The crashing workload was reporting no kernel at all, because a memory fault tears the process down before the failing dispatch is attributed, so the archive's kernel table is now used as an explicitly-marked fallback when it holds exactly one kernel. Both cases are covered by fixtures. The full list of parser and device-masking fixes is in the commit message.

Behavioral evals come in two tiers. Four tests run against replay logs checked into evals/fixtures/, so they grade the agent's reasoning (right fault class, right address, right kernel) on a runner with no GPU, and they include an unreadable-archive guard and a false-activation screen. A fifth test replays a real archive end to end and skips unless /dev/kfd, an hrr-playback binary and HRR_EVAL_ARCHIVE are all present, so it stays opt-in rather than failing where no GPU is available.

@andraghetti
andraghetti force-pushed the hrr-replay-analysis branch 2 times, most recently from ed9f08e to 017bc10 Compare August 5, 2026 09:53
Ships the HRR replay skill the catalog has advertised as planned, under the
published catalog name so the skill and the catalog row agree, and publishes
it in the Claude, Cursor and Codex manifests.

The skill replays a HIP Record and Replay archive and reports what went wrong
as a structured finding: fault class, fault address, failing event, and the
implicated kernel. It stops at the finding; kernel patching, rocgdb and fix
verification are out of scope and handed off.

Behavioral evals come in two tiers. Four tests run against replay logs checked
into evals/fixtures, so the reasoning is graded on a runner with no GPU, and
they include an unreadable-archive guard and a false-activation screen. A fifth
replays a real archive and skips unless /dev/kfd, an hrr-playback binary and
HRR_EVAL_ARCHIVE are all present.

Validated on a gfx950 host against a real offline vLLM inference capture
(185,469 events, 13,233 kernels), a crash capture, and an archive whose format
the playback build cannot read. Defects fixed along the way:

- A clean replay still reported a hipBLASLt GEMM as the kernel, matched out of
  the archive's own kernel listing; next to a PASS that reads as a culprit.
  The kernel fields are now cleared for replay_pass.
- A memory fault tears the process down before HRR attributes the failing
  dispatch, so the finding named no kernel even for a single-kernel archive.
  The --info kernel table is now parsed and used as a fallback when the archive
  holds exactly one kernel, marked as inferred. Names truncated by the table's
  column width are ignored, since a truncated symbol cannot be looked up.
- Classification ran per input and the archive --info pass overwrote the replay
  verdict with UNKNOWN; it now runs once over every input.
- A GPU memory fault also trips the generic abort line, which was tested first,
  so a read-only page fault reported replay_fatal_api.
- The Complete: matcher accepted only YES/NO while --info prints
  "yes (clean shutdown)", so a clean archive never matched.
- Format mismatches were unclassified and read as a workload result; they now
  report archive_version_mismatch.
- The GPU picker's awk emitted free VRAM before reading the used-memory line,
  so "most free VRAM" always selected the largest card, and GPU selection ran
  before the --info early exit, which must work on a host with no GPU.
- The replay masked the device with ROCR_VISIBLE_DEVICES, which re-indexes
  devices underneath the HIP mask and can land the replay on a different card.
@andraghetti
andraghetti force-pushed the hrr-replay-analysis branch from 017bc10 to 7382263 Compare August 5, 2026 09:55
@andraghetti
andraghetti marked this pull request as ready for review August 5, 2026 09:57
Review against the HRR runbooks found three gaps. Two are applied as proposed;
the third is applied as a caveat rather than a new fault class.

- RE_MEM_FAULT_ERR anchors on the two fields actually consumed. The bracket's
  leading fields vary by ROCm build, and on builds that omit `host:` the kernel
  name was dropped entirely, which is the one field the report exists to give.
- `--info` kernel rows tolerate a missing id column, and the launch total is
  recovered from the API call-count block when no `Kernels:` line is printed.
- run_hrr_replay.sh adds --sync-after-launch by default, with --no-sync to opt
  out. Without it the GPU is serialized once at the end, so a fault is reported
  but never attributed: on a real gfx950 crash log this fills failing_call_index,
  which was empty before. --timing opts out on its own, since serializing every
  launch is exactly what a timing run must not do.
- Per-event progress lines are parsed, so the last launch to start before a
  fault stands in when the runtime's fault line named no kernel.

A fault on a <<<>>>-launched ATen kernel stays illegal_memory_access rather
than becoming a separate class. Capture records pointers embedded in by-value
struct args and replay translates them with a defensive rescan, so such a fault
can be genuine, and reclassifying it would suppress a real finding. The
translation is a value-based heuristic and archives predating it carry no
recorded offsets, so the finding carries a note and the skill asks for the
original failure signature rather than resolving the ambiguity either way.
@danielholanda danielholanda changed the title Add hrr-replay-analysis skill for HRR capture replay and triage New Skill: hrr-replay-analysis Aug 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant