[RFC]: Measurement framework + per-workload data for readbacks_mode, and a proposal for deferred-batching

[akitaonrails]

Motivation

readbacks_mode ships off by default because the implementation has a known perf and stability cost that has been called out across PR #2668, #3178, #3305 ("currently slow and will most likely be rewritten"), #4085, #4120, and issues #3322, #3346, #3826, #4215. Every one of those threads argues from hand-measured numbers (fps deltas, "12 fps on AMD", "significant stuttering") collected with ad-hoc methodology. There is no shared measurement framework, which means successive PRs are debated on incomparable data. PR #3404's fence-detection rewrite stalled on correctness concerns that, in hindsight, could have been narrowed if a common harness existed.

I built one to investigate a separate issue (DriveClub race-start dim, CUSA00003) and it landed me on concrete numbers that I'd like to share, and on a proposal for what would actually reduce the structural cost. I can only validate on one GPU, so I want feedback before going further.

Proposed Change

Two independent pieces, in order of risk.

1. Land a measurement harness upstream — src/common/readback_metrics.{h,cpp} + ~30 lines of instrumentation across vk_scheduler, buffer_cache, texture_cache, fault_manager, page_manager, vk_presenter. Atomic counters always live (~5 ns / op), wall-clock timing gated on SHADPS4_READBACKS_METRICS=<csv path>. Zero behavioural change. CSV dump on exit (atexit, at_quick_exit, SIGTERM chain).

Counters: Scheduler::Finish / Wait / Submit counts and cumulative ns; BufferCache::ReadMemory + DownloadBufferMemory bytes + ns; TextureCache::DownloadImageMemory; FaultManager::ProcessFaultBuffer dispatches; PageManager::Protect syscalls + ns; bounded frame-time ring for p50/p95/p99.

Rationale: fills the "everyone measures differently" gap. Lets future readback PRs be A/B'd against a common baseline. Small, self-contained, opt-in at runtime.

2. Propose GpuReadbacksMode::Batched — coalesce BufferCache::ReadMemory calls within a fault window into a single Scheduler::Finish flushed at the next PM4 sync packet (WaitRegMem / fence EventWrite). Ordering guard: a guest read targeting a range with a pending batched download force-flushes first.

Rationale: per-call measured at ~68 µs on RTX 5090, ~106 ReadMemory/frame in Bloodborne gameplay = 7.23 ms/frame of cumulative CPU stall. One Finish per batch collapses that regardless of per-Finish-wait cost, which is the axis that scales poorly on AMD (see #3322, #4120).

Explicitly not proposing: flipping the global default, removing Relaxed, or reviving PR #3404's fence-detection prepass. Those are separate calls.

Data

Single hardware: RTX 5090 + Ryzen 7950X3D + Arch Linux + Nvidia proprietary. Branch akitaonrails/shadPS4:readbacks-perf, commit 133e3d7c. Methodology + raw CSVs at docs/readbacks-perf/.

game / workload	mode	p50ms	p95ms	p99ms	Finish/f	Finish_ms/f	mprotect/f
DriveClub menu idle	disabled	16.67	16.67	16.71	0	0.00	86
DriveClub menu idle	relaxed	16.67	16.67	16.71	14.9	2.66	83
DriveClub menu idle	precise	16.67	16.67	16.68	15.4	2.84	111
DriveClub race 1m	precise	33.33	33.34	33.36	19.1	2.76	170
Bloodborne menu idle	precise	33.33	33.34	33.35	0.9	0.17	4
Bloodborne gameplay	precise	33.33	33.38	50.00	106	7.23	809
Bloodborne gameplay	disabled	33.33	33.35	50.00	0	0.00	462

Observations (this hardware only):

• Bloodborne gameplay p99 is identical under Precise and Disabled (50.00 ms both). The tail-hitch is shader-compile, not readback-related.
• Relaxed matches Precise on Finish cost in our clean-cell data. Didn't reproduce the "Precise stutters during asset loads" symptom that motivated PR Low readbacks mode #4085. May be hardware-specific, may need a longer-session repro.
• FaultManager::ProcessFaultBuffer dispatches zero times in both games. The fault-buffer compute path is unused for these titles — a potential per-game opt-out.
• Per-Finish wait ≈ 7.23 ms / 106 = 68 µs on this GPU. Hardware-dependent; the same call likely costs 150–300 µs on mid-range / AMD, which is the regression vector documented in [GAME BUG]: Driveclub works fine with Nvidia GPU but fails with AMD rdna 3 #3322, video_core: opts amd low/high readback crash fixed hopefully idk pls try it i dont really have an amd gpu except for my igpu but it kinda sucks for testing lmao #4120.

Explicit non-conclusion: the data above does not justify flipping the global default. It only says Precise is viable on one high-end Nvidia config for two games. The default-flip question needs exactly the kind of data that a landed harness would let others produce.

Feedback Period

2 weeks, or until a maintainer weighs in on whether the harness is welcome and whether the batching direction aligns with the internal roadmap (especially vs PR #3404's approach).

Questions

1. Would you merge the measurement harness as a standalone PR, behind the opt-in env var? Or do you prefer this routed through Tracy zones instead?
2. Is GpuReadbacksMode::Batched a direction you'd consider, or does it overlap too much with PR video_core: Readback optimizations #3404? If so, I'd rather rebase on video_core: Readback optimizations #3404's work than start fresh.
3. Is there an existing contributor with AMD RDNA 2 / 3 / 4 hardware who'd be willing to run the harness against their setup? That's the only way to close the "does batching actually fix the AMD regression" loop.
4. Any game-specific repro cases you'd want run through the harness before deciding? (I still have Bloodborne + DriveClub set up.)

Any Other Things

• Full research dossier that seeded this work: docs/readbacks-perf/research/readbacks-research.md. Covers PR/issue timeline, per-frame cost breakdown from the source, failure modes, comparable techniques from yuzu / Ryujinx / Dolphin / PCSX2 / PPSSPP.
• Related prior work on this fork: the same harness was how I traced DriveClub CUSA00003's race-start blackout back to readbacks_mode being off (issue: guest auto-exposure is a GPU→CPU histogram feedback loop that fails silently under Disabled). Write-up at docs/driveclub-investigation/phase-31-readbacks-mode-fix.md on the gamma-debug branch.
• All benchmarks are reproducible via scripts/benchmark_readbacks.sh <mode> <label> <seconds> plus scripts/analyze_readbacks.py <run-dir> on the readbacks-perf branch.

[red-prig]

It seems I was the first to notice your research, it's a bit funny, but for now our conclusion is that we need to somehow systematize the instructions for launching the game. The situation with the misunderstanding about the readback setting for Driveclub is quite funny.

[DionSantosGG]

Adding a data point from AMD hardware, since this thread specifically asked for RDNA2/3/4 testers.

Hardware/setup: AMD Radeon RX 6750 XT, Mesa 26.0.4, Bazzite (KDE/Wayland, amdgpu, GPU clocks managed via LACT), tested via a Distrobox (Arch) build for one experiment noted below. Game: DriveClub (CUSA00093), v01.28. Symptom: With readbacks_mode: Disabled or Relaxed, the scene goes near-black a fixed ~45s into the first (night→dawn) race, matching the auto-exposure histogram feedback issue described earlier in this thread. readbacks_mode: Precise avoids the blackout but drops FPS from a native 30 to ~17-20.

Camera-direction correlation (MangoHud): Facing forward (bright sky/sun in frame): GPU utilization ~11-16%, one CPU core pinned near 100%, presented FPS 15-22 (below the game's native 30 cap). Facing backward/away from the sun (same scene, same second): GPU utilization ~16%, one CPU core still near 100%, but presented FPS hits the native 30 cap cleanly. Low GPU occupancy + a single pinned CPU core + FPS below the engine's own cap reads like a CPU-side stall rather than added GPU rendering cost, which lines up with center-weighted exposure metering (the game has an enableCentreWeightedMeteringMode config flag) doing more readback work when the bright sky/sun is in the center of frame.

Ruled out: I built @akitaonrails' gamma-debug branch (the ReinterpretMsDepthAsColor / ResolveDepthOverlap MSAA-depth-as-color fix) against this same game/version and confirmed it does not fix this particular symptom — even on that branch, Disabled/Relaxed readbacks still produce the same gradual night→dawn blackout. That fix addresses a different, always-black symptom (not the gradual/direction-dependent one), so for DriveClub the two issues appear to be separate.

Code note for whoever picks up the Batched proposal: BufferCache::ReadMemory (buffer_cache.cpp) currently always calls the sync path of DownloadBufferMemory (scheduler.Finish() then write back). There's already an async variant of the same function (DownloadBufferMemory<true>, using scheduler.DeferOperation) used elsewhere (garbage collector cleanup) — it doesn't currently guarantee data is ready by the time a guest read observes it, which is presumably why ReadMemory doesn't use it, but it looks like a useful starting point/reference for the ordering-guard logic described in the RFC.

---

Second, separate finding — AMD power governor compounding the above.

While testing, I noticed the AMD Automatic GPU power profile (Bazzite's default, driven by amdgpu) gets stuck near its lowest P-state during these races, even while GPU utilization reads 37-46%:

Power profile	GPU clock (observed)	GPU util	Frame time	FPS
Automatic (default)	~502 MHz	37-46%	51-62 ms	16-19
Forced "high" (power_dpm_force_performance_level=high equivalent, set via LACT)	~1637 MHz	18-37%	~33.5 ms	~30 (native cap)

Same race, same track section, only the clock profile changed — roughly 3x the clock and frame time cut in half just by taking the automatic governor out of the decision. I looked around for existing reports of this specific interaction (automatic AMD power state getting stuck low against this kind of readback-stall workload) and didn't find one — closest related complaint I could find is the long-standing ask in FeralInteractive/gamemode#488 about amdgpu aggressively power-saving under workloads that read as low-GPU-load. If anyone knows of a more specific existing report, happy to be pointed at it instead of duplicating.

My read on why: amdgpu's automatic governor uses GPU utilization % as its main ramp signal, and the readback-stall pattern above (short bursts of real work interleaved with Scheduler::Finish() waits) produces exactly the low/bursty utilization signature that heuristic misreads as idle — so it keeps clocks low, which then makes each burst of real work take even longer once it runs, compounding the stall on top of the sync cost itself.

Practical takeaway: forcing a fixed clock profile (high or low, either beats Automatic) is a separate, stackable mitigation on top of readbacks_mode: Precise, not a replacement for it. Video showing the live switch and MangoHud numbers, if useful: [https://www.reddit.com/r/shadps4/comments/1vjegx8/achei_um_bug_na_real_comportamento_ruim_do_driver/]

Happy to run the harness mentioned earlier in this thread on this card, or gather more traces, if it'd help either investigation.