DO NOT MERGE: diagnose a GPU memory fault on the retry that follows it by sbryngelson · Pull Request #1800 · MFlowCode/MFC

sbryngelson · 2026-09-02T01:35:52Z

Draft, and the top commit is a deliberate bug. 3ef4c282 injects an out-of-bounds device write so CI produces a real GPU memory fault and the diagnostics below can be seen working in an actual job log. Revert it before this is considered for merge.

The problem

A GPU memory fault reaches CI as an address and nothing else:

Memory access fault by GPU node-9 (Agent handle: 0x32463c0) on address 0x15443991c000

That is not actionable. Twelve jobs failed this way in a two-week sample — all Frontier CCE, the largest non-infrastructure failure class on that machine — and none produced anything anyone could act on.

What the runtimes will say if asked

Measured on a Frontier compute node with MFC's own module set (cpe/25.03, rocm/6.3.1, craype-accel-amd-gfx90a), using a deliberate out-of-bounds write:

ACC: Execute kernel fault_$ck_L8_1 async(auto) from fault.f90:8
Memory access fault by GPU node-4 ...

CRAY_ACC_DEBUG=1 names the kernel and the source line of the launch that faulted. CRAY_ACC_DEBUG=2 adds allocate 'arr(?:?)' (800 bytes) / present 'arr(:)' (800 bytes), which separates an out-of-bounds access from an unmapped one. Level 3 is unusable noise.

On the AFAR toolchain frontier_amd uses, verified on the identical compiler drop (therock-afar-23.2.1-gfx90a), OFFLOAD_TRACK_ALLOCATION_TRACES=true gives:

Device pointer 0x7f158f7e2000 does not point into any (current or prior) host-issued allocation

Why it goes on the retry

Neither can be on for a whole run: CRAY_ACC_DEBUG=1 emits 142,777 lines for one 800-cell 1D case, one per kernel launch and per transfer.

But MFC already retries a failed case up to three times, and those retries rescue almost nothing — 0 of 235 in bench, with every recorded failed test showing the full attempt count. That last fact is what makes this work: when a case fails it fails all its attempts, so the retry is a reproduction of the fault that has already been paid for and currently produces nothing. On a GPU memory fault, the next attempt now re-runs with the diagnostics on and keeps the tail.

Both variables are set rather than detecting the cluster: each runtime provably ignores the other's, checked on both toolchains.

Everything here was measured, not assumed

question	answer	how
Does the env reach the binary through `mfc.sh run` → mako → `srun`?	yes	142,777 `ACC:` lines on Frontier
Volume	142,777 lines for one 1D case → capped to the last 80	same run
Slowdown	1.27x (13.7s → 17.3s)	timed on Frontier
Timeout risk	none — a case would need ~2800s unaided to be pushed past `TEST_TIMEOUT_SECONDS`; the slowest seen in CI is ~1000s	derived
Does the AMD variable do anything under CCE?	no, ignored	tested on Frontier
Blast radius	one case, one retry; `case_env` is local to `handle_case`, so a suite with no GPU faults is bit-for-bit unaffected	code

Deliberately not done

Nothing in the .mako templates. Those generate job scripts for every ./mfc.sh run on all 18 supported clusters, so anything set there would follow users into production runs. The environment is built per subprocess in the test harness instead — also why it is a fresh dict rather than os.environ, since cases run in worker threads and a mutated global would leak per-kernel logging into every concurrent case.
No GPU core dumps. They do land on Frontier when the working directory is node-local, but one faulting run wrote 1.1 GB of core plus fifteen gpucore files of ~157 MB each. The CI message GPU core dump failed / Failed to allocate file: Bad file descriptor is the runner workspace being on Lustre, and it is accidentally protective.

Unverified, and it cannot be forced

That these faults reproduce on the retry. It is inferred from every recorded failed test showing Attempts: 3, but there has been no occurrence in the last twelve Test Suite runs, so history cannot confirm it. If they do not reproduce, this costs nothing — it only fires after a failure — but it buys nothing either.

The injected bug in 3ef4c282 is what turns that from an assumption into an observation.

https://claude.ai/code/session_013573Qr8zEMdYLkP4XyVfiy

A GPU memory fault reaches CI as an address and nothing else: Memory access fault by GPU node-9 (Agent handle: 0x...) on address 0x... Measured on a Frontier compute node with MFC's own module set, the offload runtimes will say considerably more than that. Under CCE, CRAY_ACC_DEBUG=1 names the kernel and the source line of the launch that faulted: ACC: Execute kernel fault_$ck_L8_1 async(auto) from fault.f90:8 Memory access fault by GPU node-4 ... Under the AFAR toolchain frontier_amd uses, OFFLOAD_TRACK_ALLOCATION_TRACES states whether the address ever belonged to a host-issued allocation, which separates an out-of-bounds write from an unmapped one. Neither can be on for a whole run: CRAY_ACC_DEBUG prints per kernel launch and per transfer, and MFC launches thousands per timestep. So spend a retry on it. MFC already retries a failed case up to three times, and those retries rescue almost nothing -- 0 of 235 in bench, with every recorded failed test showing the full attempt count. That last fact is what makes this work: when a case fails it fails all its attempts, so the retry is a reproduction of the fault that has already been paid for and currently produces nothing. On a GPU memory fault the next attempt now re-runs with both variables set. Both, rather than detecting the cluster: each runtime ignores the other's, verified on both toolchains. Nothing changes for any other failure, and nothing changes on the happy path. Not placed in the .mako templates. Those generate job scripts for every ./mfc.sh run on all 18 supported clusters, so anything set there would follow users into production runs. The environment is built per subprocess in the test harness instead -- also the reason it is a fresh dict rather than os.environ, since cases run in worker threads and a mutated global would leak per-kernel logging into every concurrent case. Measured while establishing the above, on the same AFAR drop Frontier uses: allocation tracking costs 10.5x on a loop that maps and unmaps every iteration, and nothing measurable on MFC's shape (map once, then kernels and target updates: 2.606s -> 2.620s over 2000 iterations). Also learned and deliberately not acted on: GPU core dumps do land on Frontier when the working directory is node-local, but a single faulting run wrote 1.1 GB of core plus 15 gpucore files of ~157 MB each. The CI failure "GPU core dump failed / Failed to allocate file: Bad file descriptor" is the runner workspace being on Lustre, and it is accidentally protective. 510 tests pass.

Measured on Frontier: CRAY_ACC_DEBUG=1 emits 142,777 "ACC:" lines for a single 800-cell 1D case, one per kernel launch and per transfer. The previous commit echoed a failing attempt's output whole, so a diagnostic retry would have buried the failure it exists to explain under six figures of runtime chatter. Only the tail is worth keeping. The fault comes last, and the launch immediately before it is what names the kernel and source line: ACC: Execute kernel syscheck_$ck_L89_1 from .../syscheck.fpp:89 Memory access fault by GPU node-4 ... Ordinary failures still print in full -- they are short and the whole thing is useful. Only the diagnostic retry is capped, and the complete capture remains in out_pre_sim.txt for anyone who wants it. The same session settled the two things this design rested on: chain the variable does reach the binary through ./mfc.sh run -> frontier.mako -> srun -> binary, so the change is live, not inert cost 13.7s -> 17.3s (1.27x) on the case that produced those 142,777 lines. Against the 1 hour TEST_TIMEOUT_SECONDS a case would need to take ~2800s unaided before a diagnostic retry could push it over, and the slowest case seen in CI is around 1000s. So the retry cannot convert a fault into a timeout, which would have hidden the very thing it is meant to surface. The logging is per case and per retry -- case_env is local to handle_case and only set once that case has faulted -- so a suite with no GPU faults is bit-for-bit unaffected, and one with a fault pays 1.27x on exactly one case. 511 tests pass.

A deliberate out-of-bounds device write in the RK update, so CI produces a real GPU memory fault and the retry diagnostics from the preceding two commits can be seen end to end in a job log rather than argued about. Scoped to the igr branch, which a handful of tests exercise, rather than firing for every GPU case and burning the whole matrix. Not in syscheck: a faulting syscheck would trip the preflight, which would then start excluding perfectly healthy nodes. Expected in the log of a Frontier or Phoenix GPU leg: <case> failed, Memory access fault by GPU node-N ... GPU memory fault: retrying <case> with offload diagnostics enabled ACC: Execute kernel <name> from src/simulation/m_time_steppers.fpp:<line> Memory access fault by GPU node-N ... Revert this commit before the PR is considered for merge. Claude-Session: https://claude.ai/code/session_013573Qr8zEMdYLkP4XyVfiy

github-actions · 2026-09-02T01:37:40Z

Claude Code Review

Head SHA: 3ef4c28

Files changed:

4
src/simulation/m_time_steppers.fpp
toolchain/mfc/test/case.py
toolchain/mfc/test/test.py
toolchain/mfc/test/test_gpu_fault_diagnostics.py

Findings:

src/simulation/m_time_steppers.fpp (around the new lines after if (igr) then): deliberate out-of-bounds GPU write left in the diff, explicitly labeled "DO NOT MERGE". The added line q_cons_ts(1)%vf(i)%sf(j + 100000000, k, l) = 1._wp is an intentional out-of-bounds device write, per the surrounding comment, meant only to trigger a GPU fault for testing CI diagnostics. As written this is memory-corrupting code inside the simulation time-stepper's hot path and must not be merged into master — it will crash or silently corrupt every IGR run once built. This needs to be removed before merge regardless of the CI-diagnostics feature's value.
toolchain/mfc/test/test.py: the retry loop's GPU-fault detection can never fire, so the new diagnostic-retry feature is dead code. In _handle_case (line ~648), a detected fault is re-raised as MFCException(f"Test {case}: Failed to execute MFC. [gpu-memory-fault]") — a synthetic message that does not contain the raw fault text. In handle_case's except block (line ~884), the retry logic then calls is_gpu_memory_fault(str(exc)), which matches only against GPU_FAULT_SIGNATURES ("memory access fault by gpu", "offload error: memory access fault"). Since str(exc) is the synthetic "...[gpu-memory-fault]" message and never contains either signature string, is_gpu_memory_fault(str(exc)) will always evaluate False, so case_env is never populated and the diagnostic retry (CRAY_ACC_DEBUG/OFFLOAD_TRACK_ALLOCATION_TRACES) never activates on a real GPU fault. The added unit tests in test_gpu_fault_diagnostics.py don't catch this because they check that is_gpu_memory_fault and gpu-memory-fault/diagnostic_env/case_env each appear somewhere in the source independently, rather than exercising the actual exception round-trip from _handle_case into handle_case.

GPU_PARALLEL_LOOP expands to nothing on CPU builds, so the deliberate out-of-bounds write was also executing on the host in every CPU igr test. That is undefined behaviour rather than the clean device fault this is meant to produce, and it would have made the CPU legs fail for a reason unrelated to what is being demonstrated. Gated on MFC_GPU so only the GPU legs fault. Claude-Session: https://claude.ai/code/session_013573Qr8zEMdYLkP4XyVfiy

Copilot

🟡 Changes recommended

The diagnostic retry currently won’t trigger because the retry loop checks the wrong text for the GPU-fault signature, and the PR also contains an intentional out-of-bounds device write that must be removed before merge.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This draft PR aims to make GPU memory faults actionable in CI by using the test harness’ existing retry mechanism: when a run fails with a GPU memory fault, the next attempt re-runs with offload-runtime diagnostics enabled and prints only the bounded tail of the output.

Changes:

Thread a per-subprocess env through the test harness so retries can enable offload diagnostics without mutating global os.environ.
Detect GPU memory fault signatures and (intended to) trigger a diagnostic retry that prints only the last N lines to avoid log flooding.
Add unit tests for the GPU-fault detection/diagnostic-env logic, and (intentionally, per PR description) inject an out-of-bounds device write to force a real GPU fault.

File summaries

File	Description
toolchain/mfc/test/test.py	Adds GPU-fault detection + diagnostic retry env plumbing and bounds CI log output on diagnostic retries.
toolchain/mfc/test/case.py	Allows passing a per-subprocess environment to `mfc.sh run` via `common.system(..., env=...)`.
toolchain/mfc/test/test_gpu_fault_diagnostics.py	Adds unit tests covering GPU fault signature matching and diagnostic env composition/usage.
src/simulation/m_time_steppers.fpp	Injects a deliberate out-of-bounds device write to force a GPU memory fault (must be removed before merge).

Review details

Files reviewed: 4/4 changed files
Comments generated: 3
Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+                                ! ############ DO NOT MERGE ############
+                                ! Deliberate out-of-bounds device write, to make
+                                ! CI produce a real GPU memory fault so the retry
+                                ! diagnostics can be seen end to end in a job log.
+                                ! Scoped to igr so only a handful of tests fault
+                                ! instead of the whole GPU matrix.
+                                ! Revert with: git revert <this commit>
+                                q_cons_ts(1)%vf(i)%sf(j + 100000000, k, l) = 1._wp
+                                ! ######################################


+            if is_gpu_memory_fault(str(exc)) and case_env is None:
+                case_env = diagnostic_env(dict(os.environ))
+                cons.print(f"    [yellow]GPU memory fault[/yellow]: retrying {case.trace} with offload diagnostics enabled")


+def test_diagnostics_are_not_enabled_for_ordinary_failures():
+    import inspect
+
+    from mfc.test import test as t
+
+    src = inspect.getsource(t.handle_case)
+    # the enabling is guarded by the fault check, not unconditional
+    assert "if is_gpu_memory_fault(" in src


The first attempt used j + 1e8 (762 MB past the array base) and produced no fault at all: the Frontier benchmark leg ran the igr case five times and passed. Reproduced on an MI210 with the same AFAR toolchain. The offset has to clear MFC's *whole* device footprint, not one array: 32 MB allocated, offset 1e8 -> faults (why the first local test misled me) 4 GB allocated, offset 1e8 -> NO fault (lands in the next allocation) 4 GB allocated, offset 2e9 -> memory access fault 4 GB allocated, offset 1e11 -> HSA_STATUS_ERROR_MEMORY_APERTURE_VIOLATION So there is a window. Too small and it silently corrupts a neighbouring array; too large and the runtime reports an aperture violation, which is a different message that is_gpu_memory_fault does not match and which would not exercise the diagnostic path either. 2e9 elements is 16 GB, clear of MFC's allocations and still an ordinary memory access fault. It also stays inside a default 4-byte integer. Claude-Session: https://claude.ai/code/session_013573Qr8zEMdYLkP4XyVfiy

The failure site raised "[gpu-memory-fault]" while the retry searched for "memory access fault by gpu", so the two never matched and the diagnostic could not fire. CI proved it: a Frontier gpu-omp shard hit 216 memory access faults and enabled diagnostics zero times. The marker is now one of the signatures verbatim, and parenthesised rather than bracketed -- Rich parses "[...]" as a style tag and deletes it, which is why that shard logged a bare "Failed to execute MFC. " with the marker missing. The three tests this replaces asserted only that the source text contained certain identifiers, which cannot detect a mismatch between the string one side writes and the string the other side reads. The two new tests exercise the hand-off and the Rich rendering; both are verified red against the respective bugs. Claude-Session: https://claude.ai/code/session_013573Qr8zEMdYLkP4XyVfiy

The diagnostic fired correctly on Frontier CCE (39 retries, 4413 CRAY_ACC_DEBUG lines) and then pointed at the wrong kernel. With a known out-of-bounds write injected into m_time_steppers, the last kernel logged before each fault was s_write_run_time_information in 111 of 140 faults, s_igr_riemann_solver in 23, and m_time_steppers in none. Dispatches are asynchronous, so the fault is reported long after the launch that caused it and the trace's tail is whatever ran next. A trace that confidently accuses the wrong kernel is worse than no trace, so the retry now sets AMD_SERIALIZE_KERNEL/COPY=3. Also corrects the docstring claim that CRAY_ACC_DEBUG=1 names the launch that faulted; this run falsified it. Whether CCE's offload runtime honours the HIP serialization vars is what the next run measures. Claude-Session: https://claude.ai/code/session_013573Qr8zEMdYLkP4XyVfiy

Measured on the CCE gpu-acc shard at 45ec609: with AMD_SERIALIZE_KERNEL/COPY=3 set, the same injected fault still blamed s_write_run_time_information in 168 of 213 faults and m_time_steppers in none -- the distribution is unchanged from before serialization. They are HIP runtime variables; CCE's offload runtime is not HIP. This also removes a claim I had no measurement for: the previous docstring said serialization was "verified to be honoured on the AFAR/HIP path". It was inferred from the variables being HIP's, not measured. The AMD lanes are still pending. Claude-Session: https://claude.ai/code/session_013573Qr8zEMdYLkP4XyVfiy

The fault-injection experiment measured what each variable was worth, and most were worth nothing. AFAR/OpenMP already names the faulting kernel unaided -- "Kernel 0: omp target in _QMm_time_steppersPs_tvd_rk @ 486", correct in 90 of 90 faults and printed on the FIRST attempt, before any diagnostic is enabled. That falsifies the premise this retry was built on, that a fault reports only an address. CCE/OpenACC cannot name it at all: across three runs CRAY_ACC_DEBUG blamed s_write_run_time_information 386 times and the true culprit 0 of 473, because dispatch is asynchronous and its log's tail is whatever ran next. AMD_SERIALIZE_* changed neither lane. Both are removed -- a confidently wrong suspect is worse than no diagnostic. OFFLOAD_TRACK_ALLOCATION_TRACES stays: it reports whether the faulting address was ever a real host allocation (60 retried faults, 0 unretried), which the runtime does not volunteer. Claude-Session: https://claude.ai/code/session_013573Qr8zEMdYLkP4XyVfiy

codecov · 2026-09-02T10:51:07Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 61.69%. Comparing base (ac08ffe) to head (d544695).
⚠️ Report is 2 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #1800   +/-   ##
=======================================
  Coverage   61.69%   61.69%           
=======================================
  Files          84       84           
  Lines       21620    21620           
  Branches     3196     3196           
=======================================
  Hits        13338    13338           
  Misses       6090     6090           
  Partials     2192     2192

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

The diagnostics only emit when the runtime is already aborting on a memory fault, so they are inert in a healthy run and there is nothing to save by withholding them. Setting them on every run makes attempt 1 carry the evidence, which is what the retry existed to obtain. Adds OFFLOAD_TRACK_NUM_KERNEL_LAUNCH_TRACES=8 -- host stack traces for recent launches, which the runtime advertises in its own fault message -- alongside the allocation verdict. Adds NVHPC's wording to the fault signatures. It says "Accelerator Fatal Error / CUDA_ERROR_ILLEGAL_ADDRESS", nothing like AMD's "memory access fault by GPU", so 189 faults on a Phoenix gpu-acc shard were never recognised as GPU faults. Still not setting CRAY_ACC_DEBUG: it streams a line per launch for the whole run and, because CCE dispatches async by default (acc_model=auto_async_kernel), its tail names whatever ran next -- the wrong kernel in 81 of 102 traced faults. The flag that would fix that, -h acc_model=auto_async_none, is a compile flag no retry can set. Claude-Session: https://claude.ai/code/session_013573Qr8zEMdYLkP4XyVfiy

…tion CCE defaults to acc_model=auto_async_kernel, so a memory fault surfaces at an unrelated sync point and its trace names the wrong kernel (81 of 102 traced faults blamed s_write_run_time_information, 0 named the culprit). auto_async_none executes kernels synchronously, which should make the abort land on the faulting kernel. Scoped to Cray + OpenACC: acc_model is an OpenACC flag, so the OpenMP offload builds are unaffected by construction. This is a measurement, not a proposal. Even if it works it should probably not ship in CI builds: it would stop the test suite exercising the asynchronous dispatch that production runs use. Claude-Session: https://claude.ai/code/session_013573Qr8zEMdYLkP4XyVfiy

… attribution" This reverts commit 7e21c29.

Removes the three DO NOT MERGE commits' effect on m_time_steppers.fpp (3ef4c28, cc4b52e, c2d0579). The file is now byte-identical to master. The injection did its job: it is the only reason the diagnostics could be checked against a known ground truth, which is how the original design was found to be measuring the wrong thing on every lane. Claude-Session: https://claude.ai/code/session_013573Qr8zEMdYLkP4XyVfiy

1. The detection path had no effect. is_gpu_memory_fault tagged the exception and nothing read the tag: classify_error bucketed anything containing "failed to execute" as a generic execution failure, so a GPU memory fault -- the one execution failure a retry provably cannot fix -- was indistinguishable from a transient launcher problem. It now gets its own bucket, which is what the detection was kept for. 2. "accelerator fatal error" was too broad. NVHPC uses that prefix for unrelated failures, including "call to cuMemAlloc returned error 2: Out of memory"; classifying an OOM as a memory fault would send the reader hunting a bad index that does not exist. cuda_error_illegal_address already matches the real thing. 3. Restart cases bypassed the diagnostics entirely -- run_restart never took an env, so a fault there produced none of the output this exists to provide. 4/5. Comments still described the retry that was removed, and one clause did not parse. Findings 1 and 4 were both residue from deleting the retry: the mechanism went, its vocabulary stayed. The tests missed it because they asserted the marker round-trips, not that anything consumes it. Claude-Session: https://claude.ai/code/session_013573Qr8zEMdYLkP4XyVfiy

Measured on Frontier (CCE 19.0.0, ROCm 6.3.1): HSA_TOOLS_LIB=librocm-debug-agent.so.2 prints "Disassembly for function s_tvd_rk$m_time_steppers_$ck_L486_6" -- subroutine, module and source line of the injected fault -- plus the faulting instruction and per-wave registers, straight to the job log. The earlier conclusion looked only at CCE's own trace and generalised from it to the machine. The information was available one layer down, at ROCr. Env-only: no recompile, no execution-model change. Not enabled yet: its cost on a healthy run is being measured, and that decides always-on versus a documented recipe. Claude-Session: https://claude.ai/code/session_013573Qr8zEMdYLkP4XyVfiy

HSA_TOOLS_LIB=librocm-debug-agent.so.2 is the only thing that names a faulting kernel on CCE. Measured on Frontier (CCE 19.0.0, ROCm 6.3.1): it prints "Disassembly for function s_tvd_rk$m_time_steppers_$ck_L486_6" -- the exact injected fault site -- with the faulting instruction and per-wave registers, where no CRAY_ACC_* variable names it at all. Enabled wherever the library is reachable. The gate is evaluated per call, never at import. On Frontier the library is on disk the whole time but only reaches LD_LIBRARY_PATH once mfc.sh load runs, so an import-time gate reports "absent" on the one machine this is for, indistinguishably from Phoenix where it truly is missing. It probes for the file rather than dlopen'ing it, so testing the subprocess's environment does not load a debug agent into the harness. The agent emits ~14k lines per fault, almost all of it one disassembly and register dump repeated per wave. summarize_rocm_debug_agent collapses that to ~37 lines. A fixed tail cannot substitute: on the real report the first 80 lines are one wave's registers and the last 80 another's, and the kernel name appears in neither. The stop-PC histogram is kept because the modal PC was a load while the fault is a write, so a single PC would name the wrong instruction. Cost on a healthy run: 4.5645 ns/gp/eq/rhs against an agent-free spread of 4.5301-4.5614 -- 0.07% above a range 0.69% wide. That is n=1 by decision, not by measurement, and the comment says so. Claude-Session: https://claude.ai/code/session_013573Qr8zEMdYLkP4XyVfiy

Every command in the investigation was --gpu acc. The OpenMP-offload lane was never built or run, so listing it as working was an inference sitting in a table of measurements. The agent hooks ROCr, below both OpenACC and OpenMP offload, so it should fire either way -- but the claim is attribution, not firing. s_tvd_rk$m_time_steppers_$ck_L486_6 is CCE's OpenACC symbol mangling, and whether module, subroutine and line survive in the OpenMP-offload form is unverified. The summarizer is unaffected: its regex takes whatever the symbol is. Also upgrades the fixture to the real report's format -- the "(Agent handle: ...)" clause, "End of disassembly." as terminator, and the blank line plus "scalar registers:" header before a wave's dump -- and adds a test pinning the field set the summarizer produced from the genuine 14,635-line log. Structure only: pinning the wave counts or PC histogram would encode one fault instead of testing the code. Claude-Session: https://claude.ai/code/session_013573Qr8zEMdYLkP4XyVfiy

The regexes were written against ROCm 6.3.1 and silently produced '' for 65,210 lines of real 7.2.0 output -- on the AFAR lane, the very one they were meant to serve, with no error to explain it. Two format changes: wave line: 7.2.0 inserts kernel_code_entry= and kernargs= BETWEEN the pc and "(stopped, reason:", which the adjacency-requiring regex rejected. fault line: "OFFLOAD ERROR: memory access fault ... at virtual address ... Reasons:" instead of "Memory access fault ... on address ... Reason:", and the lookup was case-sensitive on "Memory". Fixed at three sites; the fault line now reuses is_gpu_memory_fault, which already knows every wording, instead of hardcoding one version's. Both formats are pinned by fixtures built from real reports, and neither may be fixed at the other's expense. This is the failure a single-version fixture cannot catch: it passes while the lane produces nothing. Found only because someone ran it against the real file. Claude-Session: https://claude.ai/code/session_013573Qr8zEMdYLkP4XyVfiy

CCE OpenMP closes the last cell: s_tvd_rk$m_time_steppers_$ck_L486_16, the same scheme as the OpenACC lane's _ck_L486_6 and differing only in a trailing counter. That overturns the assumption behind the previous comment. The symbol form is set by the COMPILER, not the offload model: CCE emits its own scheme for both acc and mp, while AFAR's Flang form (__omp_offloading_..._QMm_time_steppersPs_tvd_rk_l486) is different again. Reading any two lanes suggests the offload model decides; only all three show otherwise. All carry module, subroutine and line. Also records that the summarizer is now validated against three real reports (14,635/13,826/65,210 lines in, 37/35/36 out) rather than one, and that the stop-PC histogram earns its place most on CCE OpenMP, which halts at seven distinct PCs against four for CCE OpenACC and one for AFAR. Claude-Session: https://claude.ai/code/session_013573Qr8zEMdYLkP4XyVfiy

… summary is missing Two gaps closed. 1. Silent degradation is now loud. When the debug agent is reachable and the failure IS a GPU memory fault but no agent report is recognised, that is either a failed load or a format change -- and until now the only symptom was raw output where a summary should have been. That is exactly how a ROCm 6.3.1-only parser sat on the AFAR lane returning nothing for 65,210 lines. It now says so. 2. bench.py and run_case_optimization.sh had no fault handling at all. Both run GPU cases; neither set the diagnostics, and bench printed a fixed log_tail on failure, which cannot surface an agent report -- on a real one the tail is a single wave's registers and the kernel name is not in it. The diagnostics move to mfc/gpu_diagnostics.py now that three callers share them; bench.py depending on the test module to explain a crash would be the wrong way round. .github/scripts/summarize_gpu_fault.py gives the shell script the same summary, exiting 1 when there is no agent report so the caller falls back. The bench test needed padding past log_tail's 60-line window: with a 20-line fixture the tail contains the kernel name and the test passes against the old behaviour, proving nothing. Claude-Session: https://claude.ai/code/session_013573Qr8zEMdYLkP4XyVfiy

mfc.sh test and mfc.sh bench are developer commands, not only CI entry points, and the agent was enabled purely on the library being reachable -- so it switched on for local runs on any ROCm machine. mfc.sh run is untouched and unaffected. Two ways that was wrong, both silent. Setting HSA_TOOLS_LIB behind someone collecting a GPU core dump gives them "Failed to enable debug interface" and no dump, because the agent and ROCr core dumps are mutually exclusive -- the same path an attached rocgdb trips. And the OFFLOAD_TRACK_* values overwrote whatever the caller had chosen. An explicit setting is now authoritative: the agent is skipped when HSA_TOOLS_LIB or HSA_ENABLE_DEBUG is already set, and the other two are defaults rather than overrides. Same rule in the case-optimization script. Claude-Session: https://claude.ai/code/session_013573Qr8zEMdYLkP4XyVfiy

Frontier CCE --gpu mp, ROCm 6.3.1, agent 2.0.3, four interleaved pairs, all twelve runs valid. Healthy run: no effect detected. The agent's whole range sits inside the agent-free range, paired differences split 2 up / 2 down, mean -0.045%. Resolution is ~0.8% set by the agent-free spread, so this is "no effect detected at n=4", not "no effect". Healthy-run log noise is zero -- 2661-2662 bytes with and without. Faulting run: +0.387 s, 1.60x of a 0.647 s baseline, which is 0.011% of the 1-hour test timeout. A fault cannot become a timeout through the agent -- the risk worth checking, since a diagnostic that hides the fault it explains is worse than none. The cost that is real is volume: 6.7 MB / ~13,630 lines per faulting test on that lane, ~65,000 on AFAR. That makes the summarizer load-bearing rather than an optimisation. Also replaces the AFAR-interaction caveat with the measurement that settled it: the agent does not supersede libomptarget (OFFLOAD ERROR 1, Libomptarget 8, identical with and without); it is mutually exclusive with ROCr core dumps only. Timings are CCE only; the AFAR lane produces twice the waves and was not re-timed. Claude-Session: https://claude.ai/code/session_013573Qr8zEMdYLkP4XyVfiy

sbryngelson added 3 commits September 1, 2026 19:19

sbryngelson marked this pull request as ready for review September 2, 2026 01:42

Copilot AI lite review requested due to automatic review settings September 2, 2026 01:42

sbryngelson changed the title ~~DRAFT: diagnose a GPU memory fault on the retry that follows it~~ DO NOT MERGE: diagnose a GPU memory fault on the retry that follows it Sep 2, 2026

Copilot started reviewing on behalf of sbryngelson September 2, 2026 01:42 View session

Copilot AI reviewed Sep 2, 2026

View reviewed changes

sbryngelson added 5 commits September 1, 2026 21:29

sbryngelson added 4 commits September 2, 2026 09:01

Revert "DO NOT MERGE: measure whether auto_async_none fixes CCE fault…

bac1376

… attribution" This reverts commit 7e21c29.

sbryngelson mentioned this pull request Sep 2, 2026

GPU fault diagnostics: attribution on all four lanes; CCE needs the ROCm debug agent #1801

Open

sbryngelson added 9 commits September 2, 2026 09:56

Navigation Menu

Uh oh!

DO NOT MERGE: diagnose a GPU memory fault on the retry that follows it - #1800

DO NOT MERGE: diagnose a GPU memory fault on the retry that follows it#1800
sbryngelson wants to merge 22 commits into
masterfrom
ci/gpu-fault-diagnostics

sbryngelson commented Sep 2, 2026

Uh oh!

github-actions Bot commented Sep 2, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

codecov Bot commented Sep 2, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

sbryngelson commented Sep 2, 2026

The problem

What the runtimes will say if asked

Why it goes on the retry

Everything here was measured, not assumed

Deliberately not done

Unverified, and it cannot be forced

Uh oh!

github-actions Bot commented Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

codecov Bot commented Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

2 participants

github-actions Bot commented Sep 2, 2026 •

edited

Loading

codecov Bot commented Sep 2, 2026 •

edited

Loading