Translate once, cache, use every core: a plan to boot like a PC game and reach 30 FPS on UE5 titles (Sword of the Sea numbers)

[AlxcBTW]

Builds on #2242 (shader code compiled at registration, now going through VK_EXT_graphics_pipeline_library) and #1612 (resolving T#/V#/S# on the GPU). It replaces neither. It puts them into one plan with a second reference title, and adds three pieces I haven't seen proposed yet:

• compute pipelines prepared ahead of the first dispatch;
• translating a command buffer on more than one thread;
• cross-queue label waits that stay on the GPU.

The idea

Proton/DXVK, Yuzu/Ryujinx and RPCS3 all got fast the same way. Translate once, cache what was translated, spread the remaining per-frame work over every core, and keep the per-frame path thin. AnyPS5 still does most of its work per frame:

• one worker per queue decodes PM4;
• pipelines are created when a draw first needs them;
• descriptor contents are walked and compared on the CPU every draw.

A PS5 title hands us raw PM4, a level below Vulkan and D3D12, so the target API isn't the lever. What matters is how often we translate and on how many threads.

Where the time goes on Sword of the Sea (PPSA25344, UE5)

Windows 11, RTX 5070 Laptop (8 GB, NVIDIA 616.92), Ryzen AI 9 365, on AC. The build is main plus #5315 and the #290 / #4515 / #4517 import fixes, with APS5_PROFILE_DRAW=1.

Area	Measured
Cold load, title → gameplay	> 700 s. Compute pipeline creation is 390 s (56 %): 85 creations over 1 s each, 66 distinct programs, median 2.5 s, worst 42 s, SPIR-V up to 1.2 M words
Repeat creations in the same run	26 creations of a byte-identical SPIR-V and specialization hash still cost 1–3 s each (46 s), with the VkPipelineCache loaded (696 MiB)
Graphics pipelines under the GPU lock	67 s of 72 s of lock hold time; longest hold 4.7 s
Queue 0 on the CPU	~130 ms per frame, serial. Blocked per frame: WAIT_REG_MEM on the compute queue 17.8 ms, DMA_DATA hook syncs 12.5 ms, ahead-helper waits 15.6 ms
Key reuse	91 % of dispatch keys are new every frame, from V# word 0 (base addresses) in per-frame descriptor tables. Same pattern as #1612 on Demon's Souls (0 % byte-identical dispatch captures)
GPU	~90 ms per frame: compute 46 ms (recompiled shaders of 100 k–1.7 M SPIR-V words at 168–255 registers), copies + staging 22 ms, draws 14 ms, barriers 9 ms
Memory	22–24 GB private, ~70 GB commit, VRAM at budget (7.1 GB). The title gets ~12.5 GB on a PS5
Frame rate	title 7.4 FPS on main (parl0v, RTX 3060; 4.9 earlier the same day); gameplay 6.6–6.8 FPS

Both walls, CPU at 130 ms and GPU at 90 ms, need roughly a 4× cut for 30 FPS. No single change gets there, which is why this is a plan rather than one PR.

Proposal: four parts

1. Boot like a PC game (pipelines)

#2242 covers graphics: libraries built at registration (phase B). On this title, compute dominates the load, so:

1. Compute at registration where possible. A compute program whose variants don't depend on dispatch-time data can be created when it's registered, on a worker pool. The open fix(shader): drop module constants nothing references after specialization #5315 (dead constants) and perf(recompiler): read the buffer stride and presence at run time in compute modules #5353 (stride and presence read at run time) cut the data-dependent variants.
2. A pipeline manifest plus a parallel preload at boot for the variants that remain data-dependent.
   • Each created pipeline gets one record: recompiled SPIR-V hash, specialization data, layout and state key.
   • The next boot rebuilds all of them on every core before the first frame, with progress in the window title, as DXVK's state cache, Yuzu and RPCS3 do. The first run still compiles once, like a PC game's "compiling shaders" step.
3. VK_KHR_pipeline_binary next to the existing VkPipelineCache, if warm creation still costs about 1 s per pipeline after 1 and 2. The NVIDIA driver above advertises it. Binaries die with driver updates, so the manifest stays the source of truth.
4. Nothing created under the GPU lock (perf(agc): create compute pipelines before taking the GPU lock #4897 does this for compute).

Target: warm boot, New Game → gameplay under 30 s, then under 10 s.

2. Translate once, on all cores (queue CPU time)

1. Keys that describe shape, not addresses. Resolve T#/V#/S# on the GPU: a persistent page table, an address-keyed image table, and a miss ring #1612 is the design for this, and this title is a second data point for it. feat(agc): count which key component differs on resource-content cache misses #5408 and feat(agc): trace draw-key cache misses #5364 count which key component changes on each miss (resource-content and draw keys), so the first component to move out of the key gets picked from data.
2. Parallel translation of one command buffer (new). Split a submitted DCB at its synchronization points (labels, waits, end-of-pipe events). Decode and record the segments on a worker pool into separate Vulkan command buffers, one pool per thread. Vulkan allows recording in parallel. One submit thread keeps the guest order. Today the ahead-helper prepares work but then waits (15.6 ms per frame above).
3. Label waits that stay on the GPU (new). Most queue-0 ↔ compute waits pair a release that is known at submit time with a later wait. Those pairs map to timeline semaphores at submit boundaries, and the CPU stops spinning on them. Labels written by shaders, or not known at submit time, keep today's path. The share each group has needs measuring first.
4. Fewer hook syncs. A DMA_DATA whose destination the CPU never reads back becomes an ordered GPU copy instead of a CPU sync. perf(agc): take a dword fill of any inline constant as a buffer fill #5380 and perf(agc): take the memory-value dword fill kernel as a buffer fill #4603 are steps in this direction: readTarget 1,030 → 530 ms per 10 s on parl0v's box.

Target: queue 0 under 33 ms per frame, then under 16 ms.

3. GPU time close to a native build (shaders)

The PS5's GPU runs these shaders at the title's own frame rate, so most of the 90 ms is in what we emit.

1. Register pressure first. Take the ten compute shaders with the most GPU time (Nsight per-shader numbers) and work on them:
   • dead-code removal after specialization;
   • shorter live ranges;
   • values the PS5 keeps in SGPRs kept uniform instead of per thread.
2. Native f16 where the guest uses packed 16-bit math, instead of emulating it in f32.
3. Exact-IEEE fixups only where they change a result (perf(shader): take the f32 mul, fma and add result fixups only when needed #5400).
4. Copies and barriers (31 ms):
   • keep resources resident;
   • upload only pages the CPU wrote;
   • merge barriers per pass.

Target: GPU frame under 33 ms (30 FPS), then under 16 ms.

4. Memory shaped for a PC

A discrete GPU can't be zero-copy with a unified-memory console, so the aim is to copy once and keep it resident:

• residency with LRU eviction and budgets (feat(agc): host import windows, least-recent eviction under the budget #290, fix(agc): select coherent buffer heaps using available budget #4446);
• lazy commit of guest heaps (perf(libkernel): commit Windows direct backing when mapped #4281);
• host-visible imports only for data the CPU really rewrites every frame.

Target: under 16 GB private for this title, no paging on 32 GB machines, VRAM within budget.

Order and how it's measured

Every step is a normal default-on PR against main, measured with the Performance report form on the same machine and scene before and after. Milestones, in order:

1. warm load under 30 s;
2. queue 0 under 33 ms;
3. GPU under 33 ms, i.e. 30 FPS in gameplay;
4. both under 16 ms, i.e. 60 FPS.

A non-UE5 title joins as the second reference once milestone 2 holds, so nothing gets tuned to one engine.

Risks

• Reusing a recorded segment needs a cheap check that the guest didn't rewrite the command memory or the tables it reads (write watch or hash).
• Some shaders depend on exact addresses (alignment tricks, pointer math). They keep the address in their key; feat(agc): count which key component differs on resource-content cache misses #5408 shows how many there are.
• Timeline semaphores only cover waits whose release is known at submit time.
• An async "skip the draw until the pipeline is ready" fallback would hide the remaining stalls but can drop effects for a moment. If anyone wants it, it should be opt-in.

Asks

• Does this split match how the maintainers see it, and are there parts already in progress that I've missed?
• I'd start on part 1 (compute at registration plus manifest preload) and the measurement side of part 2 (feat(agc): count which key component differs on resource-content cache misses #5408 data, wait-pair census), each as separate small PRs.
• Numbers for the same table from an AMD GPU and from other UE5 titles would show what is title-specific.

[CnRJay]

+1

[fma-rc]

A non-UE data point for the table: Astro's Playroom (PPSA01325) in its first level, main dacd785, Linux, RTX 5080, APS5_PROFILE_DRAW=1 and APS5_PROFILE_GPU=1, per 10 s window.

Area	Measured
Frame rate	47 presents per 10 s
Driver thread	8,910 ms in draw packets (144,414 draws, 61.7 us each), serial
Draw key reuse	144 hits in 144,521 register-key lookups
Dispatch key reuse	605 of 880 lookups equal
Compute queue	94 WAIT_REG_MEM_64, 9,168 ms
Pipelines	cache hits 100% in the scene, 13 compute pipelines created in the whole run
GPU	1,790 ms of GPU time, 1,266 ms of it in two compute programs

So here part 2 is the whole story: one thread at about 190 ms per frame with draw keys that almost never repeat, while the GPU is under 20% busy and pipeline creation plays no role. Dispatch keys mostly repeat, unlike your title.

I have no trace digest for this run; it ended before the trace window.

[AlxcBTW]

Thanks, that's a useful second shape. Astro's Playroom is the same CPU wall from the draw side: 144 hits in 144,521 draw-key lookups, against SotS's 91 % new dispatch keys. Both titles leave the GPU waiting on one thread.

The question that decides part 2 is which component of the draw key changes. On SotS it's V# word 0, the base address, moving with per-frame rings. If you can rerun the same scene with #5364's draw-key miss trace (and #5408 for resource-content keys), we'd see whether Astro churns the same way. If it does, one address-free key design covers both an engine-agnostic title and UE5, and part 2 is mostly a key change rather than new machinery.

Overnight I'm measuring a SotS PoC on current-14 (load path, memory, and a breakdown of the big compute shaders). The numbers will go into this thread.

[fma-rc]

Ran it: main 0aa4cb1 plus the heads of #5364 (2bdb521) and #5408 (5cf0aa4), same scene, both traces on.

#5408, per 10 s: draws miss 8,688 times, 8,352 of them with a key identical to the set's last one and 336 differing in an image descriptor; every V# word count is 0. Dispatches miss 464 times, all counted as layout and not comparable. So V# word 0 does not move in Playroom's resource-content keys.

#5364 cannot answer the draw-key question yet. It printed four lines in the whole run, all with shader-set=cbf29ce484222325. That is the FNV offset basis: the fifth component is set to it and never mixed, so every draw lands in one set and the four-lines-per-set cap ends the trace. The four lines were first-seen, stable-registers,user-data-registers,device twice and stable-registers. The draw register key itself still hits 144 times in 145,780 lookups.

[AlxcBTW]

That's a better answer than I expected. Two things stand out:

• 8,352 of 8,688 draw misses have a key identical to the set's last one. So in Playroom the miss isn't the key changing. Something outside the key throws the entry away. My guess is the write watch: Resolve T#/V#/S# on the GPU: a persistent page table, an address-keyed image table, and a miss ring #1612 saw titles rewrite their tables with the same values, so the pages go dirty while the content stays put. If that's it, comparing content on a dirty entry before rebuilding would turn most of those 8,688 into hits. Address-free keys wouldn't help there.
• Dispatches showing up as all "layout / not comparable" is most likely feat(agc): count which key component differs on resource-content cache misses #5408's labeller not understanding Playroom's dispatch key layout. It refuses rather than guessing when the walk doesn't end exactly at the key's end. I'll check that against your setup and fix it in feat(agc): count which key component differs on resource-content cache misses #5408.

So the two titles split the CPU wall differently. SotS has real address churn in dispatch keys; Playroom's draw entries get invalidated while their content stays the same. Both are fixable without new machinery, and I'll fold both into part 2.

[AlxcBTW]

Found the dispatch one: a dispatch content key ends with two device-handle words after the shader part (DispatchContentKey), and the labeller insisted the shader walk end exactly at the key end, so every dispatch fell into "not comparable". Fixed in #5408 (e88b5c296, with a test for keys that carry the device words). A rerun should now split Playroom's 464 dispatch misses by component.