Add hrr-replay-findings skill for read-only HRR replay triage - #91
Closed
akshayv wants to merge 6 commits into
Closed
Add hrr-replay-findings skill for read-only HRR replay triage#91akshayv wants to merge 6 commits into
akshayv wants to merge 6 commits into
Conversation
Structured parser classifies replay faults (read-only/OOB, illegal access, NaN/Inf divergence, hang, OOM) and extracts fault address, event index, kernel, and kernarg from logs and archives across diverse GPU workloads.
akshayv
force-pushed
the
hrr-replay-findings
branch
from
July 15, 2026 15:34
54aedd4 to
11435ff
Compare
added 5 commits
July 15, 2026 15:38
Add run_hrr_replay.sh (native or Docker) with optional --analyze step; update SKILL workflow to archive-first replay then structured finding.
Remove HRR_REPO_ROOT and internal lab script deps; auto-pick GPU by free VRAM; default docker replay uses stock container ROCm without HIP_SO paths.
Replay runs hrr-playback on the host with auto GPU selection; no container image or docker dependency in the skill or run_hrr_replay.sh.
Auto-find hrr-playback on PATH and /opt/rocm/bin; auto LD_LIBRARY_PATH; ask user once only for archive or playback if discovery fails. No Docker, no hardcoded lab paths, no run-this-script user instructions.
6 tasks
andraghetti
added a commit
to andraghetti/skills
that referenced
this pull request
Aug 5, 2026
Ships the HRR replay skill the catalog has advertised as planned, adapted from the hrr-replay-findings draft (PR amd#91) under the published catalog name so the skill and the catalog row agree. Carried-over defects fixed while verifying the log parsing against the hrr-playback sources in ROCm/rocm-systems: - analyze_replay_finding.py classified per input and recomputed the verdict on each one, so the archive --info pass that --analyze always performs overwrote the replay log's verdict with UNKNOWN. Classification now runs once over every input. - A GPU memory fault also emits an HRR abort line, and the abort branch was tested first, so a read-only page fault reported replay_fatal_api. Memory faults are now classified ahead of the generic abort and hang branches. - The Complete: matcher only accepted YES/NO, but --info prints "yes (clean shutdown)" or "NO (...)", so a clean archive never matched. - Archive/reader format mismatches were unclassified and read as a workload result; they now report archive_version_mismatch. - --info reports events and kernels as labelled lines, which were not parsed at all, so an --info-only run had no archive totals. - run_hrr_replay.sh picked a GPU with awk that emitted the free-memory figure before reading the used-memory line, so "most free VRAM" always selected the largest card. Totals are now collected per device and subtracted once. GPU selection also moved after the --info early exit, which must work on a host with no GPU. SKILL.md additionally gains the multi-process process-table step, the version-mismatch stop condition, capture instructions for a user who has no archive yet, and evidence guardrails. Co-authored-by: AMD HRR Team <hrr-team@amd.com>
andraghetti
added a commit
to andraghetti/skills
that referenced
this pull request
Aug 5, 2026
Ships the HRR replay skill the catalog has advertised as planned, under the published catalog name so the skill and the catalog row agree, and publishes it in the Claude, Cursor and Codex manifests. Supersedes the hrr-replay-findings draft in amd#91. The skill replays a HIP Record and Replay archive and reports what went wrong as a structured finding: fault class, fault address, failing event, and the implicated kernel. It stops at the finding; kernel patching, rocgdb and fix verification are out of scope and handed off. Behavioral evals come in two tiers. Four tests run against replay logs checked into evals/fixtures, so the reasoning is graded on a runner with no GPU, and they include an unreadable-archive guard and a false-activation screen. A fifth replays a real archive and skips unless /dev/kfd, an hrr-playback binary and HRR_EVAL_ARCHIVE are all present. Validated on a gfx950 host against a real offline vLLM inference capture (185,469 events, 13,233 kernels), a crash capture, and an archive whose format the playback build cannot read. Defects fixed along the way: - A clean replay still reported a hipBLASLt GEMM as the kernel, matched out of the archive's own kernel listing; next to a PASS that reads as a culprit. The kernel fields are now cleared for replay_pass. - A memory fault tears the process down before HRR attributes the failing dispatch, so the finding named no kernel even for a single-kernel archive. The --info kernel table is now parsed and used as a fallback when the archive holds exactly one kernel, marked as inferred. Names truncated by the table's column width are ignored, since a truncated symbol cannot be looked up. - Classification ran per input and the archive --info pass overwrote the replay verdict with UNKNOWN; it now runs once over every input. - A GPU memory fault also trips the generic abort line, which was tested first, so a read-only page fault reported replay_fatal_api. - The Complete: matcher accepted only YES/NO while --info prints "yes (clean shutdown)", so a clean archive never matched. - Format mismatches were unclassified and read as a workload result; they now report archive_version_mismatch. - The GPU picker's awk emitted free VRAM before reading the used-memory line, so "most free VRAM" always selected the largest card, and GPU selection ran before the --info early exit, which must work on a host with no GPU. - The replay masked the device with ROCR_VISIBLE_DEVICES, which re-indexes devices underneath the HIP mask and can land the replay on a different card.
Collaborator
|
Closing this PR as #117 replaces this one |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
hrr-replay-findings— a read-only Agent Skill for triaging HIP Runtime Replay (HRR) captures and replay logs across diverse GPU workloads (vLLM, PyTorch finetune, hipBLASLt StreamK).reference.md) and worked examples from MAF repro sessionsscripts/analyze_replay_finding.py(log + optional archive--info+ multi-replay sweep TSV → JSON/Markdown)Designed as the first, low-risk triage step before deeper crash analysis / patching workflows.
Skill layout
Test plan
Cijk_*_SK3_*,d2h_fail=0)npx skills add amd/skills --skill hrr-replay-findingsafter mergereplay_passclassificationUsage
Made with Cursor