PGR2 Race to Menu Corruption Lessons

PGR2 Race-to-Menu Corruption — Lessons Learned

Canonical subject: Vulkan surface/guest-memory coherence across a retail state transition Issue: #106 — PGR2 race-to-menu corrupt textures Repair history: PR #119 — preserve guest writes over cached surfaces Historical status: The root cause below was identified and a focused repair validated at PR #119 head. A related corruption recurred on released main ef1a7fc4 in September 2026; #106 is open again. The new Vulkan capture again shows an already-corrupt BC3 image, but its earlier writer is not yet identified. Do not assign the historical readback sequence to the recurrence without a new writer trace.

Why this investigation matters

Issue #106 began as a menu corruption that appeared after leaving an active PGR2 race. The visible symptom was narrow green, magenta, and blue blocks in glyph and atlas regions. Several plausible explanations existed:

• shader-route or Hybrid handoff;
• the shader fast path;
• native BC texture upload;
• cached-image overwrite timing;
• missed texture dirtiness;
• descriptor or texture-key aliasing;
• final display composition or host presentation;
• a stale cached surface writing older pixels back into guest memory.

The decisive result was the last one. The affected 256 × 256 BC3 atlas was initially uploaded correctly. Later, two cached dirty surfaces downloaded older contents into the atlas's 64 KiB guest-memory range. The texture cache then received a valid dirty notification, uploaded the now-corrupt guest bytes into the same Vulkan image, and correctly considered that binding clean.

The cache was not failing to notice the final change. The failure occurred earlier, at the ownership boundary between cached GPU surfaces and newer guest-memory writes.

This distinction is broadly useful. A corrupt sampled image does not prove that its texture cache, descriptor, or upload command is the writer. The sampled image may faithfully contain bytes that another representation incorrectly made authoritative.

Root-cause summary

The relevant range was:

Atlas source: 0x037f0000 + 65536 bytes

The target-only trace identified two later surface downloads:

Surface A
  range:  0x037f8000 + 32768
  shape:  128 × 128, pitch 256, swizzled zeta
  atlas hash transition:
          1df2ca19ebbabcc7 → 2c1156e3ae28cebb

Surface B
  range:  0x037e8000 + 65536
  shape:  256 × 256, pitch 256, linear color
  overlap with atlas:
          0x037f0000 .. 0x037f8000
  atlas hash transition:
          2c1156e3ae28cebb → e765ced37ace75c8

The initial 65,536 staging bytes used by vkCmdCopyBufferToImage() decoded byte-for-byte to the correct atlas. Corruption was introduced later by the surface readbacks, not by the initial upload.

Actual failure flow

flowchart TD
    A[Guest atlas bytes are correct] --> B[Texture uploads correct BC3 atlas]
    B --> C[Cached race surfaces remain draw-dirty]
    C --> D[Guest or PGRAPH writes newer data into overlapping VRAM]
    D --> E[Shared dirty notification is cleared by another consumer or skipped under TCG]
    E --> F[Inactive cached surface still appears authoritative]
    F --> G[Surface readback downloads old GPU contents into guest VRAM]
    G --> H[Texture dirty client correctly observes changed guest bytes]
    H --> I[Texture reuploads corrupted bytes into existing Vulkan image]
    I --> J[Menu draw samples already-corrupt atlas]
    J --> K[Host presenter repeatedly displays the corrupted guest output]

Loading

The final texture upload and presentation were functioning according to their inputs. Fixing those downstream stages would have hidden or displaced the ownership error.

Correct ownership flow

The repair gives surface coherency its own dirty-memory client and applies the same authority rule under TCG and accelerated execution:

flowchart TD
    A[CPU or PGRAPH writes overlapping guest VRAM] --> B[Mark surface-coherence dirty client]
    B --> C[Later surface overlap check]
    C --> D{Guest-memory dirty for overlap?}
    D -->|No| E[Dirty cached surface may download through reference path]
    D -->|Yes| F[Guest memory is newer authority]
    F --> G[Cancel pending surface download]
    G --> H[Clear cached surface draw-dirty authority]
    H --> I[Mark surface upload pending]
    I --> J[Reload surface from guest memory before reuse]

Loading

The focused state transition is equivalent to:

if (guest_memory_dirty) {
    surface->download_pending = false;
    surface->draw_dirty = false;
    surface->upload_pending = true;
}

That helper is not the complete coherence system. It expresses the crucial authority transfer: a cached surface may not later overwrite guest bytes that were written after the surface's contents became stale.

Investigation timeline

Stage	Evidence	Conclusion
Deterministic retail transition	Fresh private HDD per role; exact race-to-menu input sequence; first and late menu captures	Defect is real and persistent, not merely one bad screenshot
Same-executable role matrix	Hybrid/shortcut On and Off, repeated A/B/C cycle	First cycle suggested a shortcut; second cycle disproved that attribution
Retained baseline	Corruption with Hybrid and shortcut Off	Defect predates PR #105 and is not owned by the shortcut
OpenGL control	Same retail checkpoint clean under OpenGL	Failure belongs to Vulkan resource handling, not common guest menu state
Native-BC disabled	Corruption remained	Native compressed upload is not the owner
Old serialization guard restored	Corruption remained	Queued cached-image overwrite serialization is not the owner
Forced source validation	Hash and rebuild every active RAM-backed texture; menu became clean	Freshness/authority path is implicated, but forced hashing is only a diagnostic
Immutable source snapshot	Hash and upload from one owned snapshot; one clean run and one corrupt run	Hash/upload TOCTOU is real technical debt but not this root cause
Direct NV2A RenderDoc capture	Corrupt 256 × 256 BC3 image already wrong before menu draw	Inspect the image writer, not the consuming shader or presenter
Upload-boundary verification	Initial staging bytes and first image contents correct	Corruption occurs after the first upload
Target-only surface-lifetime trace	Two dirty surface downloads change exact atlas-range hashes	Stale surface readback is the writer
Dedicated dirty client	Two consecutive fresh-boot runs clean, where shared-client candidates failed on repeat	Independent ordering channel plus TCG coverage is required

Lessons learned

1. Test the transition, not only both steady states

The existing retail captures ended during active gameplay. They could show that the race rendered correctly, but they could not qualify the transition from race resources back to menu resources.

A state-transition defect needs an oracle that crosses the boundary:

race-like state
    → exit/target/resource transition
    → first menu-like draw
    → repeated menu-like draw

The first post-transition observation and a delayed observation serve different purposes:

• first wrong, late correct: transient ordering or incomplete publication;
• first and late wrong: persistent bad state or bad resource contents;
• race clean, menu wrong: the transition is the unit under test, not either endpoint alone.

Regression suites should include transitions when caches, surfaces, descriptors, DMA mappings, snapshots, backends, or presentation routes survive across scene changes.

2. One clean run is not evidence that a race is fixed

Several diagnostics produced one clean run and then corrupted on an identical repeat. The immutable-snapshot diagnostic also produced one clean and one corrupt run with zero observed source changes.

For timing-sensitive defects:

one clean run = observation
repeat clean runs with a failing negative control = evidence

The final repair was accepted only after:

• the earlier shared-client candidates provided a known second-run failure;
• the dedicated client passed the same transition twice;
• first and late checkpoints were both clean;
• the focused unit passed;
• the tested executable, source, runtime tree, settings, and hashes were pinned.

Two repeats are not a universal statistical standard. They were the minimum discriminating control in this case because earlier candidates failed on their second identical run.

3. Freeze identities before changing behavior

Every role recorded or pinned:

source commit
runtime-tree identity
executable SHA-256
symbol identity when applicable
renderer and scale
Hybrid requested/effective state
shortcut requested/effective state
shader-cache state
GPU and driver
presentation transport
configuration hash
HDD seed hash
input-runner hash

Without those identities, a clean run can accidentally compare a different binary, cache, menu, disk state, route, or driver state.

The runner used one immutable HDD seed and a private copy per cell. Persistent shader caching was disabled for the initial matrix. This prevented one role from warming or mutating another role's state.

4. The exact visual checkpoint is part of the test identity

A clean capture in the car showroom was rejected because the established failure occurred at the Street Racing event-selection menu. A nearby screen is not a control for an atlas/resource-lifetime defect.

Record:

• the exact phase and menu;
• the input that admits that phase;
• the first stable checkpoint;
• the delayed checkpoint;
• screenshot dimensions and hashes;
• whether the window was covered, minimized, stale, or presenting an older image.

Do not accept a visually convenient neighboring state merely because it is easier to reach.

5. Separate guest output from host presentation

A host window can be wrong for two fundamentally different reasons:

guest/display image wrong
    → renderer, composition, resource, or guest-state problem

guest/display image correct + host window wrong
    → presentation handoff, host-copy, shared-image, or UI problem

Issue #106 eventually captured the corruption inside the NV2A Vulkan frame. Issue #90 explained why the same completed image could be presented repeatedly and why an external overlay reported thousands of frames per second. It did not create the corrupt atlas.

Always capture or hash both layers when the architecture permits it:

renderer-produced guest/display image
real host window

6. Host-present rate is not guest progress

RenderDoc reported approximately 2,003–2,097 FPS while the corrupt menu remained visible. That count represented xemu's OpenGL UI/presentation loop repeatedly presenting the completed output, not Xbox guest-frame progression or useful Vulkan rendering throughput.

Use one of these as the experiment clock:

• guest flip;
• guest frame counter;
• fixed guest instruction/progress marker;
• exact workload transition;
• in-process renderer event tied to guest work.

Do not trigger a capture or claim performance from a host-present count unless the experiment specifically studies host presentation.

7. Capture at the producer boundary, not the nearest convenient window API

An F12 frontend capture could observe the OpenGL presentation path. The useful capture was initiated through RenderDoc's in-process API at the NV2A Vulkan guest-flip boundary.

The first direct capture showed that the sampled atlas was already corrupt, but contained no writes to that image. Another settled-menu capture would only show the same readers. The next capture had to move earlier, to the image's upload or writer boundary.

General rule:

reader is wrong
    → inspect its exact bound resource
resource is already wrong
    → find last writer before the reader
writer output is correct
    → inspect later mutation or lifetime reuse

8. Transient debugger IDs are not emulator identities

RenderDoc identified the resource as ResourceId::200807. That ID was useful inside one capture but could not identify the same emulated texture in another run.

Correlate external debugger resources with stable emulator fields:

guest source address and length
format and dimensions
palette source address/length
retained content hash
texture key hash
cache-binding lifetime ID
host-image lifetime ID
upload sequence
surface lifetime ID and draw version

When a stable ID does not exist, add a monotonically increasing lifetime ID at object creation. Do not serialize host pointers and do not rely on debugger-assigned IDs across captures.

9. Trace authority transitions and writers, not only cache hits

A conventional texture trace might record:

cache hit
cache miss
dirty bit
upload result

That would show a valid dirty notification and successful reupload after the atlas had already been overwritten. It would not identify the earlier surface downloads that changed guest memory.

For overlapping representations, record:

range and overlap
current authority: guest RAM / surface / texture / vertex mirror
writer type and sequence
before/after range hash
surface lifetime and draw version
download/upload pending state
cache binding/image lifetime
staging offset and payload hash
first consumer after each write

The decisive trace followed one target range across surface downloads, guest-memory hashes, texture reupload, and the menu consumer.

10. A shared test-and-clear dirty client is not a durable ordering channel

memory_region_test_and_clear_dirty() consumes information. If independent subsystems share one client, whichever checks first can erase evidence required by another subsystem.

Treat each dirty client as a consumer-specific event stream:

producer write
    → mark every consumer that must observe the write
consumer A observes and clears client A
consumer B still retains client B

Do not interpret the global absence of a dirty bit as proof that memory has not changed. It may mean another consumer already acknowledged its own notification.

Issue #106 required a dedicated DIRTY_MEMORY_NV2A_SURFACE client because texture and general NV2A consumers legitimately clear their own clients.

11. Test the accelerator that reproduces the bug

The inherited surface check skipped dirty-bitmap reads when tcg_enabled() was true. The retail launcher used TCG, so a path that appeared valid under another accelerator did not protect the actual reproducer.

Every memory-coherency test matrix should name:

accelerator
host architecture
renderer
presentation route
snapshot/fresh-boot route

A correctness guard conditioned on the accelerator deserves explicit negative tests for every supported accelerator.

12. Synthetic leaves can be stronger than retail in the wrong way

The same-handle texture and palette DMA rewrite leaves passed. They proved their narrow operations under their own synchronization. They did not reproduce the retail combination of inactive dirty surfaces, overlapping guest writes, later readback, and texture reuse.

A focused test clears only the operation it reproduces. Ask:

• Does the leaf preserve the same ownership history?
• Is its synchronization stronger than retail?
• Does it reuse the same object lifetime?
• Does it cross the same scene/target transition?
• Can an inactive cached surface survive long enough to become stale?

After finding the first divergent operation, build a regression leaf around that operation rather than a generic approximation of the game screen.

13. Expensive reference routes are diagnostic tools, not automatic fixes

Hashing every active RAM-backed texture on every draw made the corrupt menu clean. That was valuable because it localized the problem to freshness/authority handling.

It was not a production design:

• it repeated content hashing in a draw hot path;
• it rebuilt valid bindings unnecessarily;
• it hid the missing invalidation/ownership event;
• it could damage performance in unrelated titles.

Use forced validation, full rebuilds, queue waits, cache disabling, or broad flushes to answer one diagnostic question. Then remove them and repair the owning invariant.

14. Cancel less-discriminating experiments when evidence narrows the boundary

A forced queue-retirement experiment was cancelled before build after the immutable-snapshot result made direct resource inspection more discriminating.

Diagnostic work should become narrower as evidence accumulates:

broad role matrix
    → subsystem control
    → resource class
    → exact resource
    → exact writer
    → ownership invariant
    → focused fix and test

Do not continue a preplanned broad experiment merely because it was already listed.

15. Keep private assets private while publishing reproducible identities

Retail captures and GPU captures may contain copyrighted game data. The public evidence retained:

• source and executable hashes;
• input/HDD hashes;
• settings and hardware identity;
• image hashes and visual classifications;
• resource dimensions/formats;
• event numbers and state transitions;
• code-derived addresses/hashes needed for diagnosis.

Raw game images and captures remained private. This preserves auditability without redistributing game assets.

Rejected hypotheses and retained value

Hypothesis or diagnostic	Outcome	Retained lesson
Hybrid route or shader shortcut	All Vulkan roles eventually corrupted	Complete repeat matrix before assigning ownership
PR #105 introduced the bug	Retained baseline also corrupted	A patch can expose an older defect by surviving farther
Native BC upload	Decoded-upload diagnostic still corrupted	Do not equate a compressed artifact with compressed-upload ownership
Old cached-image serialization	Restored guard still corrupted	Serialization is not a substitute for resource authority
Same-handle DMA replacement	Synthetic leaves passed	A narrow leaf does not clear a combined retail transition
Hash/upload TOCTOU	Immutable snapshot still corrupted	Real weakness, wrong root cause; preserve finding separately
Final display/presentation	Direct NV2A image already corrupt	Inspect guest resource before patching presenter
Descriptor/key alias	One binding/image remained stable	Stable IDs and lifetime tracing can rule out identity switches
Initial staging/copy corruption	Initial payload decoded correctly	Move to later writers rather than changing upload again
Missed final texture dirty notification	Texture correctly reuploaded corrupt guest bytes	Dirty detection can be correct after authority was already violated

Code ownership and repair seams

PR #119 touches the following ownership chain:

File	Responsibility
include/exec/ramlist.h	Defines independent DIRTY_MEMORY_NV2A_SURFACE client
system/memory.c	Permits the surface dirty client on Xbox memory regions
system/physmem.c	Marks, queries, and clears the client with other dirty streams
hw/xbox/nv2a/nv2a.c	Enables surface dirty logging for VRAM
hw/xbox/nv2a/pgraph/gl/blit.c	Marks PGRAPH blit destination writes for surface consumers
hw/xbox/nv2a/pgraph/vk/blit.c	Same marking for Vulkan PGRAPH blits
hw/xbox/nv2a/pgraph/vk/surface-coherence.h	Encodes guest-memory-authoritative state transition
hw/xbox/nv2a/pgraph/vk/surface.c	Consumes surface client for overlaps and active target updates, including TCG
Focused unit test	Proves no-change, stale-surface discard, and upload-required transitions

The core fix belongs at the surface/guest-memory boundary. Texture code should not need to guess that a previous surface download was stale.

Required regression shape

A durable focused regression should include:

Phase 1
  Create/render a cached surface over guest range R.

Phase 2
  Make a newer CPU/PGRAPH write to overlapping bytes in R while the
  cached surface remains inactive and draw-dirty.

Phase 3
  Request a texture or vertex consumer over R.
  The stale surface must not download over the newer guest bytes.

Phase 4
  Reuse the cached surface.
  It must upload/reload the newer guest bytes before becoming authoritative.

Required assertions:

• no overlapping guest write: dirty surface follows the normal download route;
• overlapping newer guest write: download is cancelled, draw_dirty cleared, upload required;
• TCG path performs the same authority check;
• accelerated path performs the same authority check;
• texture/vertex consumer receives the newer guest bytes;
• surface later reloads those bytes;
• adjacent non-overlapping ranges remain unchanged;
• two identical retail transitions pass after a known failing negative control.

Reusable investigation checklist

Before patching a graphics corruption:

[ ] Name the exact retail transition and checkpoint.
[ ] Pin source, executable, symbols, configuration, disk/input, GPU, and driver.
[ ] Use fresh mutable state per role.
[ ] Repeat the complete role matrix.
[ ] Capture first and delayed post-transition output.
[ ] Separate guest image from host presentation.
[ ] Use guest progress—not host presents—as the clock.
[ ] Inspect the exact bound resource at the first bad draw.
[ ] Correlate debugger resources with stable emulator identities.
[ ] If the resource is already bad, trace backward to its last writer.
[ ] Record authority, overlap, lifetime, before/after hash, and sequence.
[ ] Treat dirty clients as consumable per-consumer streams.
[ ] Test the reproducer's actual accelerator.
[ ] Keep diagnostic reference routes out of the final hot path.
[ ] Add a focused red/green leaf for the first divergent invariant.
[ ] Repeat the retail transition after the focused unit passes.
[ ] Publish identities and classifications; keep private game assets private.

Related pages

• Graphics Transition Debugging Tools
• Debugging Tools & Commands
• Debugging Lag & Hot-Path Analysis
• Memory & DMA
• Vulkan Renderer
• Pull Request Lessons
• PR #119 implementation history
• Issue #106
• PR #119