Graphics Transition Debugging Tools

Graphics Transition Debugging Tools

Purpose: Reproduce and localize timing-sensitive graphics corruption across guest scene, resource, cache, and presentation transitions. Source investigation: Issue #106 — PGR2 race-to-menu corrupt textures Companion lesson: PGR2 Race-to-Menu Corruption — Lessons Learned

Issue #106 required more than a profiler, validation layer, or one GPU capture. The defect crossed several boundaries:

guest input and scene transition
    → guest VRAM ownership
    → cached render surfaces
    → texture upload and cache state
    → Vulkan image contents
    → guest display output
    → OpenGL host presentation

The useful toolchain therefore combines deterministic workload control, identity manifests, dual-boundary captures, direct Vulkan inspection, stable emulator resource IDs, and bounded writer traces.

This page documents the reusable tools and operating rules. It does not assume the private PGR2 image, HDD, input script, or GPU capture is publicly available.

Tool-selection ladder

Use the cheapest tool that can distinguish the remaining hypotheses.

flowchart TD
    A[Visual or timing-sensitive transition defect] --> B[Deterministic retail runner]
    B --> C{Repeats at exact checkpoint?}
    C -->|No| D[Record identities and park as unreproduced]
    C -->|Yes| E[Capture host window and guest/display output]
    E --> F{Guest output wrong?}
    F -->|No| G[Presentation handoff tools]
    F -->|Yes| H[Backend control and direct GPU capture]
    H --> I{Bound resource already wrong?}
    I -->|No| J[Descriptor/view/shader/input trace]
    I -->|Yes| K[Stable resource correlation]
    K --> L[Upload-boundary or target-range writer trace]
    L --> M[Focused ownership regression]

Loading

The order matters. A broad transition trace added before reproducibility can produce gigabytes of unrelated state. A final-frame capture cannot show an earlier writer when the affected resource is read-only in that frame.

1. Deterministic retail transition runner

Purpose

Drive a real title through one exact state transition with fixed inputs and isolated mutable state. The runner is an oracle driver, not merely keyboard automation.

Issue #106 used a local PowerShell GUI pipe runner. Its exact private script was identified by SHA-256 in the evidence rather than published. Future reusable runners should expose the same contract even when implemented in Python, PowerShell, or another automation layer.

Required phases

For the race-to-menu case, the fixed logical order was:

1. boot or load the selected seed
2. wait for active-race admission marker/image
3. drive for a fixed input window
4. pause when the route requires it
5. select exit race
6. wait for the first stable target menu
7. capture MENU_FIRST
8. wait five seconds without input
9. capture MENU_LATE
10. exit cleanly

The exact key names, durations, and admission markers belong in the runner manifest. Do not substitute a nearby menu or a pause screen.

Isolated run directory

Each role needs its own mutable state:

run-root/
  manifest.json
  config/
  shader-cache/
  hdd-private-copy/
  logs/
  screenshots/
  guest-output/
  gpu-captures/

Initial isolation rules:

• one immutable HDD seed;
• one private HDD copy per role;
• clean configuration directory per role;
• shader cache disabled or separately seeded as required by the experiment;
• no process from a prior role left running;
• no screenshot from a covered, minimized, or stale window;
• fixed host power/performance policy where practical.

Example PowerShell setup

The exact paths are environment-specific. The important behavior is copy isolation and identity logging:

$Seed = "D:\xemu-seeds\pgr2-base.qcow2"
$RunRoot = "D:\xemu-runs\issue106\A1"
$PrivateHdd = Join-Path $RunRoot "pgr2-A1.qcow2"

New-Item -ItemType Directory -Force $RunRoot | Out-Null
Copy-Item -LiteralPath $Seed -Destination $PrivateHdd

Get-FileHash -Algorithm SHA256 $Seed
Get-FileHash -Algorithm SHA256 $PrivateHdd
Get-FileHash -Algorithm SHA256 "C:\xemu\xemu.exe"

After each run, verify cleanup rather than assuming the window closed:

Get-Process -Name xemu -ErrorAction SilentlyContinue

If a process remains, record the abnormal termination before stopping it. Silent cleanup can erase evidence of a hang or device loss.

Role matrix

Use same-executable roles before comparing different builds:

A1: Vulkan, Hybrid On,  shortcut On
B1: Vulkan, Hybrid On,  shortcut Off
C1: Vulkan, Hybrid Off, shortcut Off
A2: Vulkan, Hybrid On,  shortcut On
B2: Vulkan, Hybrid On,  shortcut Off
C2: Vulkan, Hybrid Off, shortcut Off

Run the alternate backend only after the target transition is stable:

D1/D2: OpenGL reference

The second cycle is not optional for a race. In issue #106, the first A/B/C cycle pointed toward the shortcut; the second cycle corrupted all Vulkan roles and disproved that attribution.

Identity manifest

Record at least:

{
  "source_commit": "full SHA",
  "runtime_tree": "full SHA or tree identity",
  "executable_sha256": "...",
  "symbols_identity": "PDB/DWARF build identity",
  "renderer": "vulkan",
  "surface_scale": 4,
  "hybrid_requested": true,
  "hybrid_effective": true,
  "shortcut_requested": true,
  "shortcut_effective": true,
  "shader_cache": "off",
  "gpu_name": "...",
  "gpu_uuid": "...",
  "driver": "...",
  "presentation_transport": "...",
  "config_sha256": "...",
  "hdd_seed_sha256": "...",
  "input_runner_sha256": "..."
}

Use the effective route, not only the requested setting.

Required artifacts

Capture separately:

race_window.png
target_first_window.png
target_late_window.png
guest_scanout.bin or guest_scanout.png
run.log
manifest.json
results.csv row

Store dimensions and SHA-256 for every binary/image artifact. Raw retail images can remain private while hashes and classifications are published.

2. Artifact identity and hash ledger

Purpose

Prevent accidental comparisons between different executables, images, disks, configurations, or captures.

Windows

Get-FileHash -Algorithm SHA256 .\xemu.exe
Get-FileHash -Algorithm SHA256 .\menu_first_window.png
Get-FileHash -Algorithm SHA256 .\capture.rdc

Linux

sha256sum xemu menu_first_window.png capture.rdc

macOS

shasum -a 256 xemu menu_first_window.png capture.rdc

Source identity

git rev-parse HEAD
git rev-parse HEAD^{tree}
git status --porcelain

A clean git status --porcelain plus commit/tree identity distinguishes the intended source from an uncommitted diagnostic build.

Results table

The issue #106 evidence used a CSV shape similar to:

build,source_commit,role,hybrid,shortcut,shader_cache,
race_sha256,menu_first_sha256,menu_late_sha256,
menu_first_class,menu_late_class,outcome

Do not put subjective prose only in screenshots or chat. Preserve a machine-readable role/result ledger.

3. Dual-boundary output capture

Purpose

Separate guest renderer failure from final host presentation failure.

flowchart LR
    A[NV2A work] --> B[Renderer-produced guest/display image]
    B --> C[Shared-image or host-copy handoff]
    C --> D[OpenGL UI/presentation]
    D --> E[Host window]

    B -. capture/hash .-> F[Guest-output artifact]
    E -. capture/hash .-> G[Window artifact]

Loading

Interpretation:

guest wrong + window wrong
    → defect is before or inside display composition

guest correct + window wrong
    → presentation/shared-image/host-copy/UI route

first target wrong + late target correct
    → transient publication or transition ordering

first target wrong + late target wrong
    → persistent bad state or bad resource contents

A window screenshot alone cannot distinguish these cases.

4. RenderDoc direct NV2A Vulkan capture

Why ordinary capture was insufficient

xemu uses Vulkan for NV2A rendering and OpenGL for frontend/UI presentation. An external hotkey can capture the frontend OpenGL loop rather than the Vulkan guest frame that produced the image.

Issue #106 used RenderDoc 1.46 through its in-process capture API at the NV2A Vulkan guest-flip boundary. This produced a capture containing the actual guest Vulkan draws.

Correct trigger

host present count     wrong clock for guest rendering
frontend F12 capture   may capture OpenGL presenter
NV2A guest flip        useful boundary for guest Vulkan frame
resource upload event  useful boundary for finding an earlier writer

The RenderDoc overlay reported roughly 2,003–2,097 FPS while the same corrupted menu remained visible. That was the host presentation loop, not Xbox FPS.

Capture API shape

Exact integration depends on the RenderDoc API version already used by the project. The required control flow is:

if (capture_requested_at_guest_flip()) {
    renderdoc_api->StartFrameCapture(vulkan_device, window_handle);
}

submit_guest_vulkan_frame();

if (capture_active_for_guest_flip()) {
    renderdoc_api->EndFrameCapture(vulkan_device, window_handle);
}

Do not begin/end capture on arbitrary host presents. Store the guest frame/flip sequence in the capture manifest.

Inspection workflow

Within RenderDoc:

1. Confirm the capture is Vulkan and contains expected guest draws.
2. Locate the first draw that introduces the visible bad region into the color target.
3. Inspect fragment input resources for that draw.
4. Export the suspected image and compare it with the visible corruption.
5. Inspect resource usage history:
   • reads in this capture;
   • writes in this capture;
   • creation/import events;
   • barriers/layout transitions.
6. Inspect an unaffected peer resource as a control.
7. Decide:

bound image already corrupt
    → trace image creation/upload/later writer

image correct, draw output corrupt
    → descriptor, view, sampler, coordinates, uniforms, shader inputs

draw output correct, later target corrupt
    → later render-target writer or display composition

Issue #106 found a 256 × 256 BC3_UNORM image that was already corrupt before events 188, 275, and 297 sampled it. The capture contained no writes to that image, so another settled-menu capture could not identify the writer.

Privacy

RenderDoc captures can contain proprietary shaders, textures, and game data. Keep raw captures private unless redistribution is authorized. Publish:

• capture SHA-256 and size;
• RenderDoc version;
• source/executable identity;
• trigger boundary;
• resource dimensions and format;
• event numbers;
• read/write classification;
• derived hashes and conclusions.

5. Stable resource correlation

Problem

ResourceId::200807 identified the suspect image inside one RenderDoc capture. It was not stable across runs and did not identify the guest texture or xemu cache binding.

Required emulator identity

Add or expose already-computed fields:

typedef struct TextureTraceIdentity {
    uint64_t binding_lifetime_id;
    uint64_t image_lifetime_id;
    uint64_t upload_sequence;
    uint64_t key_hash;
    uint64_t content_hash;
    uint64_t guest_source_addr;
    uint64_t guest_source_length;
    uint64_t palette_source_addr;
    uint64_t palette_source_length;
    uint32_t format;
    uint32_t width;
    uint32_t height;
    uint32_t levels;
} TextureTraceIdentity;

Stable identity rules:

• lifetime IDs increment when host objects are created;
• same guest address with a new host lifetime receives a new ID;
• do not serialize host pointers;
• do not recompute expensive hashes solely for the tracer when retained hashes exist;
• record zero/sentinel for absent or invalid state;
• include upload sequence so several writes to one image can be ordered.

Correlation record

For a capture request, log:

guest flip/frame
binding lifetime
image lifetime
source range
format/dimensions
retained content hash
upload sequence
VkBufferImageCopy offset/extent
RenderDoc capture path/hash

This allows the external image to be traced back to its last emulator-side upload or copy.

6. BC payload and staging verifier

Purpose

Determine whether corruption exists in source bytes, staging bytes, post-copy image contents, or only later.

Issue #106 verified that the initial 65,536 bytes at VkBufferImageCopy.bufferOffset = 16384 decoded byte-for-byte to the correct 256 × 256 BC3 atlas.

A reusable verifier should accept:

staging buffer dump
buffer offset
copy extent
format
mip/layer
row/block layout
reference image or reference decoded hash

Expected report:

source/staging byte SHA-256 or fast retained hash
decoded pixel SHA-256
dimensions and format
first differing block and coordinates
number of differing blocks/pixels

For block-compressed formats, compare both:

• exact compressed block bytes;
• decoded pixels.

Equal decoded pixels do not prove byte identity, while equal compressed bytes do.

Do not use a display screenshot as proof that staging bytes were correct. Export or dump the exact copy source range.

7. Bounded Vulkan state-transition trace

Purpose

Compare failing and passing runs at one narrow guest-frame window without allocating, formatting, or writing per-draw text.

Environment contract

The issue proposed this runtime-disabled interface:

XEMU_VK_STATE_TRACE=<output.bin>
XEMU_VK_STATE_TRACE_FRAME_BEGIN=<guest frame number>
XEMU_VK_STATE_TRACE_FRAME_END=<guest frame number>
XEMU_VK_STATE_TRACE_MAX_RECORDS=8192

Rules:

• no output path means no allocation and no recording;
• reject invalid frame ranges or capacities at startup;
• preallocate one bounded record array;
• fill records from state already computed for the draw;
• no string formatting in the draw path;
• on overflow, set dropped and stop rather than wrapping over the first divergence;
• write the binary file at close or explicit frame end;
• version the header and record size.

Record groups

The full issue proposal includes these groups:

sequence / guest frame / draw index
shader-state, binding-key, and pipeline-key hashes
selected shader and pipeline lifetime IDs
Hybrid generation/route/selection epochs
requested and selected fragment route
descriptor-set identity and dynamic uniform offsets
fallback-control offset/hash
texture binding/key/content identities per stage
texture source and palette addresses
texture dirty/retry/surface-source masks
color/zeta surface lifetime, address, and draw time
scanout address and display generation
output dimensions, scale, and flags

Header shape:

typedef struct PGRAPHVkStateTraceHeader {
    uint32_t magic;
    uint32_t version;
    uint32_t record_size;
    uint32_t record_count;
    uint32_t dropped;
    uint32_t reserved;
    uint64_t source_identity_hash;
} PGRAPHVkStateTraceHeader;

Prohibited tracer behavior

The tracer must not alter the state being diagnosed:

• do not call dirty test-and-clear helpers;
• do not resolve DMA addresses again if a binding retains them;
• do not copy a large control block for every draw when a retained hash is enough;
• do not force uploads, waits, or cache misses;
• do not serialize host pointers;
• do not synchronously write a record per draw.

Unit tests

disabled trace allocates/writes nothing
frame range includes only requested frames
capacity stops without overwrite
equal snapshots serialize identically
changing each identity changes its record field
invalid bindings use explicit sentinels
partial initialization still writes valid header/count
record version/size mismatch is rejected by reader

The general trace was not the final decisive tool for issue #106, but its contract remains useful when failing and passing roles diverge in shader, descriptor, or route selection.

8. Target-range surface lifetime trace

Purpose

Follow one guest-memory range backward and forward across all representations. This was the decisive issue #106 tool.

Filter contract

Configure:

target guest address
target length
optional expected format/dimensions
maximum event count
frame/transition window

Record only operations that overlap the target range.

Event schema

typedef enum RangeEventType {
    RANGE_CPU_WRITE,
    RANGE_PGRAPH_BLIT_WRITE,
    RANGE_SURFACE_UPLOAD,
    RANGE_SURFACE_DRAW,
    RANGE_SURFACE_DOWNLOAD,
    RANGE_TEXTURE_UPLOAD,
    RANGE_VERTEX_UPLOAD,
    RANGE_GUEST_READ,
} RangeEventType;

typedef struct RangeTraceEvent {
    uint64_t sequence;
    uint64_t guest_frame;
    uint64_t host_object_lifetime;
    uint64_t before_hash;
    uint64_t after_hash;
    uint64_t overlap_start;
    uint64_t overlap_length;
    uint32_t type;
    uint32_t flags;
} RangeTraceEvent;

Additional type-specific state:

surface address/size/shape/pitch/format
draw_dirty, download_pending, upload_pending
texture binding/image lifetime and upload sequence
staging buffer offset and payload hash
dirty client observed/cleared
accelerator
first consuming draw/event

Trace flow

flowchart TD
    A[Choose exact corrupt resource source range] --> B[Log current range hash]
    B --> C[Record every overlapping writer]
    C --> D[Record before/after hash and object lifetime]
    D --> E{Hash first becomes wrong?}
    E -->|No| C
    E -->|Yes| F[Name exact writer and authority state]
    F --> G[Follow next texture/upload consumer]
    G --> H[Build focused ownership regression]

Loading

This trace established that two surface downloads—not the texture upload—changed the atlas hash.

9. Dirty-client audit tool

Purpose

Find independent consumers that incorrectly share a consumable dirty notification.

Inventory every call to:

memory_region_set_log
memory_region_set_client_dirty
memory_region_test_and_clear_dirty
physical_memory_set_dirty_range
physical_memory_clear_dirty_range

For each dirty client, document:

Field	Question
Producers	Which CPU/device/PGRAPH writes mark it?
Consumers	Which subsystem clears it?
Meaning	What authority transition does it represent?
Accelerator coverage	Is it consumed under TCG and accelerated execution?
Range granularity	Page, byte range, object, or full memory?
Destructive read	Can one consumer erase another's evidence?
Reset/snapshot	How is the client initialized and restored?

Audit rule:

Consumers may share a dirty client only when clearing the notification for one consumer is semantically equivalent to acknowledging it for every other consumer.

That was false for surface coherency versus texture/vertex validation.

10. Focused negative controls

A useful diagnostic must have an outcome that would disprove it.

Examples from issue #106:

Diagnostic	Positive result	Negative control
Shortcut ownership	A fails, B/C pass in both cycles	B/C also fail on repeat
Native BC ownership	Decoded route clears corruption	Decoded route still corrupts
Old serialization ownership	Restored guard clears corruption	Same exact menu still corrupts
Source hash/upload TOCTOU	Immutable snapshot clears repeated defect	One clean and one corrupt with zero source changes
Presentation ownership	Guest image correct, window wrong	Direct NV2A image already corrupt
Dedicated dirty client	Repeated retail runs clean	Shared-client candidates fail second run

A first clean run is not a sufficient positive result for a nondeterministic race.

11. Tool stop conditions

Stop broad instrumentation and move inward when:

exact transition repeats
backend control isolates subsystem
direct capture identifies bad resource class
resource export proves whether content is already wrong
stable identity names exact object/upload
range trace names exact writer

Stop and park rather than patch when:

symptom cannot repeat under pinned identities
runner cannot prove it reached the same checkpoint
window capture may be stale/covered
build or disk identity is uncertain
candidate route was never reached

Do not add product behavior changes until the failure boundary is narrower than the proposed fix.

12. Recommended command and evidence bundle

A complete run directory should make these questions answerable without rerunning the title:

Which source and executable ran?
Which symbols match it?
Which configuration and effective routes were active?
Which HDD/input script created the transition?
Did the exact checkpoint occur?
What did guest output contain?
What did the host window contain?
Which GPU resource was bound?
What was its stable emulator identity?
When was it last written?
Which writer first changed the target bytes?
Was the process and private disk cleaned up?

Suggested bundle:

manifest.json
results.csv
run.log
input-timeline.json
race_window.sha256
menu_first_window.sha256
menu_late_window.sha256
guest_output.sha256
trace.bin + trace-schema-version.txt
capture.rdc.sha256
resource-export.sha256
range-events.csv
build-info.txt
cleanup-receipt.txt

Related pages

• PGR2 Race-to-Menu Corruption — Lessons Learned
• Debugging Tools & Commands
• Debugging Lag & Hot-Path Analysis
• Memory & DMA
• Vulkan Renderer
• Performance Methodology
• Pull Request Lessons
• Issue #106
• PR #119

Recent diagnostic techniques worth retaining

Prior-storage sentinel — PR #138

When a producer feeds byte comparison, hashing, reflection-sized copies, or cache keys, run it twice from identical guest state with differently canary-filled output storage. Compare exactly the byte ranges downstream consumes. A failure means the experiment is comparing memory history rather than semantic state.

Same-handle remap capsule — PR #124

Keep the guest handle constant while rewriting its RAMIN descriptor to a different physical source. This detects caches that confuse handle identity with source identity.

Verify test admission first — PR #127

If an exact guest test produces zero records or an unexpected skip, confirm profile, test ID, and xemu-only enable flags before changing emulator code. A test-environment correction is not an emulator fix.

Controlled regression for concurrency — PR #132

For a lock/progress fix, deliberately remove one required progress-service step in a local regression build and require the fixture to time out. This proves the test covers the dependency cycle.

Current-main control for questionable oracle — PR #122

When a focused leaf fails on the candidate, run the same leaf on current main. If both fail identically, the oracle may encode a different ownership/timing contract. Define the intended contract before changing production code.