Debugging Tool RenderDoc

RenderDoc

← Debugging Tools and Commands

RenderDoc is a frame-capture graphics debugger. It can inspect Vulkan and OpenGL draws, attachments, textures, buffers, shaders, pipeline state, pixel history, and resource usage.

Property	Details
Supported OS	Windows and Linux desktop capture; Android capture; replay support depends on API/platform.
Download	Official stable releases
Verify	renderdoccmd --version
xemu use	Capture the guest Vulkan/OpenGL renderer at an NV2A guest-flip boundary, then replay or script the resulting .rdc.

Install and verify

Download a stable installer or portable build from the official releases page. On Windows, locate renderdoccmd.exe and qrenderdoc.exe; on Linux, verify renderdoccmd and qrenderdoc are on PATH or record their absolute paths.

& 'C:\Tools\RenderDoc\renderdoccmd.exe' --version
& 'C:\Tools\RenderDoc\renderdoccmd.exe' help

Use the same RenderDoc version for capture and replay when possible. Record its version, package source, and SHA-256 with the capture.

What it answers

• Which draw first changed a bad pixel or render target?
• Did a fragment pass depth/stencil, get discarded, or produce an unexpected output?
• Which texture, buffer, descriptor, sampler, viewport, blend state, or shader was bound?
• Was a resource already corrupt when sampled, or did a draw corrupt the destination?
• Does the capture contain guest Vulkan work or only the OpenGL presenter?

RenderDoc does not prove guest intent. Compare captured host state with the guest NV2A command/state path before choosing a production repair.

Capture safety and privacy

Captures may contain game textures, framebuffer contents, paths, machine details, and other private data. Keep raw retail captures private. Publish hashes, bounded metadata, and source-derived conclusions unless redistribution is permitted.

RenderDoc changes timing, memory use, resource retention, and driver behavior. A clean capture run can be timing-sensitive and does not qualify performance. Reproduce important findings without capture where possible.

xemu guest capture workflow

Why ordinary capture was insufficient

xemu uses Vulkan for NV2A rendering and OpenGL for frontend/UI presentation. An external hotkey can capture the frontend OpenGL loop rather than the Vulkan guest frame that produced the image.

Issue #106 used RenderDoc 1.46 through its in-process capture API at the NV2A Vulkan guest-flip boundary. This produced a capture containing the actual guest Vulkan draws.

The same distinction mattered for the Grabbed by the Ghoulies effect investigation: an external capture contained one OpenGL presentation draw, while an in-process capture contained the guest Vulkan draws. A small .rdc file or a visible screenshot is not proof that the correct API was captured.

Build and launch the capture-capable executable

The build must explicitly enable RenderDoc. The default renderdoc=auto is not proof that CONFIG_RENDERDOC was defined in the resulting executable. Use the project's normal toolchain/build instructions with -Drenderdoc=enabled, then inspect the generated configuration before using the executable:

meson configure BUILD_DIR -Drenderdoc=enabled
meson compile -C BUILD_DIR xemu
rg 'CONFIG_RENDERDOC' BUILD_DIR/config-host.h

Adjust the configuration-header path to the selected Meson build tree. Confirm it defines CONFIG_RENDERDOC; an #undef means F10 will not request a guest capture. Record the source commit, executable SHA-256, RenderDoc version, renderer, and configuration used for the run.

On Windows, launch the diagnostic executable through RenderDoc's CLI in the interactive desktop session. This is the command shape used for the Ghoulies capture; substitute your own paths and snapshot name:

$rdoc = 'C:\Tools\RenderDoc\renderdoccmd.exe'
$exe = 'C:\Path\To\xemu.exe'
$config = 'C:\Path\To\xemu.toml'
$capturePrefix = 'C:\Path\To\PrivateCaptures\guest-target'
& $rdoc capture -d (Split-Path $exe) -c $capturePrefix $exe `
    -name 'xemu-capture,debug-threads=on' -config_path $config `
    -loadvm SNAPSHOT_NAME

For automated Windows GUI runs, invoke that command through the existing interactive PowerShell pipe. A noninteractive SSH desktop is not equivalent. Check that renderdoc.dll is loaded in xemu before sending any game input.

The current Vulkan pgraph_vk_debug_frame_terminator() also requires IsTargetControlConnected() before honoring the F10 request. renderdoccmd capture can inject the API while reporting no target-control connection. If F10 produces no capture, check this condition before changing gameplay input or extending the run. The Ghoulies investigation used a private, exact-source diagnostic build that removed only this capture-control gate, logged API/start/end markers, and ended capture on the first guest flip after five seconds. It was not a rendering fix and must not silently become a release change.

Correct trigger

host present count     wrong clock for guest rendering
frontend F12 capture   may capture OpenGL presenter
NV2A guest flip        useful boundary for guest Vulkan frame
resource upload event  useful boundary for finding an earlier writer

The RenderDoc overlay reported roughly 2,003–2,097 FPS while the same corrupted menu remained visible. That was the host presentation loop, not Xbox FPS.

With CONFIG_RENDERDOC, xemu's F10 requests one guest flip and Shift+F10 requests five guest flips; these counts are frames, not seconds. Ctrl+F10 adds NV2A tracing. xemu's own F12 takes a screenshot; an external RenderDoc F12 hotkey may capture only the OpenGL presenter. A short animation can disappear during a five-frame capture or before a delayed replay trigger.

For a time-limited, automatic diagnostic capture, arm it while the game is paused, wait for an explicit StartFrameCapture marker, then send the resume input. End it on the first guest flip after the chosen elapsed time, with a safety timeout. The validated Ghoulies sequence was controller connected → Start to reach pause → arm capture → verify start marker → B for about 250 ms → automatic end at about five seconds. No attack or movement input was needed. The marker check matters: a queued F10 request does not prove the guest Vulkan capture actually started.

Capture API shape

Exact integration depends on the RenderDoc API version already used by the project. The required control flow is:

if (capture_requested_at_guest_flip()) {
    renderdoc_api->StartFrameCapture(vulkan_device, window_handle);
}

submit_guest_vulkan_frame();

if (capture_active_for_guest_flip()) {
    renderdoc_api->EndFrameCapture(vulkan_device, window_handle);
}

Do not begin/end capture on arbitrary host presents. Store the guest frame/flip sequence in the capture manifest.

Verified command-line capture and replay (2026-09-20)

The PGR2 recurrence was captured without clicking through the RenderDoc UI. On the Windows GUI test session, use RenderDoc's CLI to launch a RenderDoc-enabled diagnostic xemu. A normal release binary may lack CONFIG_RENDERDOC and therefore cannot service xemu's F10 guest-flip request. Use a fresh private HDD and the same effective renderer/settings as the reproduction.

$rdoc = 'C:\tools\RenderDoc_1.46_64\renderdoccmd.exe'
$xemu = 'C:\builds\xemu-renderdoc-diagnostic.exe'
$config = 'C:\captures\pgr2\xemu.toml'
& $rdoc capture `
    --capture-file='C:\captures\pgr2\guest-vulkan' `
    --opt-ref-all-resources `
    $xemu -name 'xemu-capture,debug-threads=on' -config_path $config

Launch from the interactive GUI PowerShell session, not an SSH service session. Wait until the corrupted menu is visible, then request the capture with xemu's F10 shortcut. On the test rig the input was sent by:

& 'C:\xemu-lab\suite\send-xemu-key.ps1' `
    -ProcessId $xemuPid -VirtualKey 121 -HoldMilliseconds 350

The private timed diagnostic used for this run logged RENDERDOC_CAPTURE_START and RENDERDOC_CAPTURE_END elapsed_ms=5003 saved=1 at NV2A flips. Its five-second duration is a diagnostic patch, not the stock F10 behavior. Verify the saved capture by replaying it: the overlay can say “Capturing OpenGL” because xemu also has an OpenGL frontend. The decisive replay result here was GraphicsAPI.Vulkan. The 245 MB capture covered 10,979 guest draws; a capture that large should be retained privately and narrowed around the writer for the next attempt.

RenderDoc 1.46 also supports automated replay through its embedded Python interpreter. This command runs a script supplied as a file, so resource inspection does not require manual GUI navigation:

& 'C:\tools\RenderDoc_1.46_64\qrenderdoc.exe' `
    --python 'C:\captures\pgr2\inspect.py'

The script can open the .rdc with renderdoc.OpenCaptureFile(), call OpenCapture(renderdoc.ReplayOptions(), None), verify GetAPIProperties().pipelineType, enumerate GetTextures(), export a candidate with SaveTexture(), and obtain its in-capture history with GetUsage(resourceId). RenderDoc's texture export example and resource-usage example show the APIs; the CLI source documents launch/injection behavior. A GetUsage() list containing only shader reads means only that this capture has no image writer; it says nothing about image creation or uploads before capture began.

The recurrence capture had a 256 × 256 BC3_UNORM image with the same colored strips visible in the menu text. RenderDoc reported 894 fragment-shader reads and no writes for that image inside the five-second capture. A separate UI-icon image was intact. The captured ResourceId is transient; correlate it with xemu's guest source address, texture key/hash, image lifetime, and upload sequence before using it in a later run. A Windows focus request can fail even when the xemu window handle exists; check screenshots, guest progress, and the capture markers rather than treating that one failure as a graphics result.

For cross-run comparison, export the raw BC3 mip with GetTextureData() and hash the same 65,536 bytes in each capture. In this case, seven of nine one-level 256 × 256 BC3 images had byte-identical clean-run peers; the colored-text image and a mostly white image did not. That localizes the damage, but capture-local resource IDs and image ordering do not establish a guest address. The corrupt capture lacked same-run upload markers, so the comparison cannot identify the earlier writer.

Inspection workflow

First reject a presenter-only capture. renderdoccmd can convert the saved .rdc for a quick, scriptable API/event inventory:

$capture = 'C:\Path\To\PrivateCaptures\capture.rdc'
& $rdoc convert -f $capture -o "$capture.zip.xml" -c zip.xml
Get-FileHash -Algorithm SHA256 $capture

Inspect the converted XML driver/API and draw calls. A useful Vulkan capture must contain the expected guest vkCmdDraw*, render passes, submissions, and the game color/depth attachments. An OpenGL capture with only the final blit is the wrong layer even if the window image shows the bug. The Ghoulies capture contained Vulkan guest draws and a 640×480 game target; the earlier external attempt contained one OpenGL presentation draw.

renderdoccmd supports capture, conversion, replay, and thumbnail operations, but does not expose a pixel-history subcommand. Use QRenderDoc's embedded Python replay for repeatable analysis; the verified qrenderdoc.exe --python script.py launch above avoids manual navigation. When a capture is already open in QRenderDoc's Python shell, pyrenderdoc.Replay().BlockInvoke(callback) supplies a replay controller. For a known game render target and pixel:

import renderdoc as rd

target = next(t.resourceId for t in pyrenderdoc.GetTextures()
              if t.width == 640 and t.height == 480
              and str(t.resourceId) == 'ResourceId::TARGET_ID')

def inspect(replay):
    replay.SetFrameEvent(LAST_CAPTURE_EVENT, False)
    changes = replay.PixelHistory(target, X, Y, rd.Subresource(),
                                  rd.CompType.Typeless)
    for change in changes:
        print(change.eventId, change.Passed(), change.depthTestFailed,
              change.shaderDiscarded, change.preMod, change.postMod)
    replay.SetFrameEvent(EVENT_ID, False)
    state = replay.GetPipelineState()
    print(state.GetDepthTestState(), state.GetColorBlends())

pyrenderdoc.Replay().BlockInvoke(inspect)

Replace TARGET_ID, X, Y, EVENT_ID, and LAST_CAPTURE_EVENT with values from this capture. Move replay to the last event before requesting whole-capture pixel history: leaving replay on an earlier event can silently return only the earlier part of the history. Resource IDs and event IDs are not stable across captures. RenderDoc's Python replay API, pixel-history guide, and QRenderDoc command-line implementation document the relevant interfaces and options.

Then, in the replay:

1. Confirm the capture is Vulkan and contains expected guest draws.
2. Locate the first draw that introduces the visible bad region into the color target.
3. Inspect fragment input resources for that draw.
4. Export the suspected image and compare it with the visible corruption.
5. Inspect resource usage history:
   • reads in this capture;
   • writes in this capture;
   • creation/import events;
   • barriers/layout transitions.
6. Inspect an unaffected peer resource as a control.
7. Decide:

bound image already corrupt
    → trace image creation/upload/later writer

image correct, draw output corrupt
    → descriptor, view, sampler, coordinates, uniforms, shader inputs

draw output correct, later target corrupt
    → later render-target writer or display composition

Issue #106 found a 256 × 256 BC3_UNORM image that was already corrupt before events 188, 275, and 297 sampled it. The capture contained no writes to that image, so another settled-menu capture could not identify the writer.

Privacy

RenderDoc captures can contain proprietary shaders, textures, and game data. Keep raw captures private unless redistribution is authorized. Publish:

• capture SHA-256 and size;
• RenderDoc version;
• source/executable identity;
• trigger boundary;
• resource dimensions and format;
• event numbers;
• read/write classification;
• derived hashes and conclusions.

Common failure modes

Symptom	Likely cause or next check
Capture contains one OpenGL blit	Presenter was captured; use xemu's guest-flip capture path and verify Vulkan replay.
F10 produces no file	Confirm CONFIG_RENDERDOC, target-control/capture markers, active Vulkan renderer, and interactive GUI session.
Animation is missing	Arm capture before resume; use automatic bounded stop instead of delayed manual input.
Pixel history is incomplete	Move replay to the last event before asking for whole-capture history.
Resource appears read-only	The writer may precede the captured interval; extend or reposition capture and correlate stable guest identities.
Replay differs from live output	Record GPU/driver/API, inspect unsupported replay features, and treat replay as diagnostic evidence only.
Capture is unexpectedly tiny	Verify API, draw count, render targets, and event inventory before inspection.

Evidence to retain

• xemu source commit/tree, executable hash, configuration hash, and RenderDoc-enabled build proof.
• RenderDoc version and executable hash.
• Renderer, GPU UUID/name, driver, resolution scale, and effective advanced settings.
• Capture start/end markers, guest flip/frame range, input timing, and .rdc SHA-256.
• Replay API, event count, target dimensions/format, responsible event IDs, and exported resource hashes.
• A control resource/pixel and the exact script used for automated inspection.

Official references

• RenderDoc project and downloads
• Stable releases
• Inspecting pixels and pixel history
• Texture viewer
• Shader debugging
• Python examples

Related xemu guides

• Graphics Transition Debugging Tools
• PGR2 Race-to-Menu Corruption Lessons
• Vulkan Renderer
• Debugging Tools and Commands