Vulkan Shader Binding and Compilation

Vulkan Shader Binding & Compilation

This page explains how xemu turns NV2A draw state into reusable Vulkan shader objects and eventually a graphics pipeline. The important point is that shader bindings, per-stage modules, persisted SPIR-V artifacts, and graphics pipelines are separate reuse layers.

A graphics-pipeline miss does not prove a shader was compiled, and pgraph_vk_bind_shaders() does more than compilation.

High-level flow

flowchart TD
    A[NV2A register / draw state] --> B[Build ShaderState]
    B --> C{Shader binding cache hit?}
    C -->|Yes| D[Reuse ShaderBinding]
    C -->|No| E[Resolve stage modules]
    E --> F{Vertex module cached?}
    E --> G{Geometry module cached?}
    E --> H{Fragment module cached?}
    F -->|No| I[Generate vertex GLSL]
    G -->|No| J[Generate geometry GLSL]
    H -->|No| K[Generate fragment GLSL]
    I --> L[glslang -> SPIR-V]
    J --> L
    K --> L
    L --> M[Create VkShaderModule]
    M --> N[Reflect layout / uniform data]
    D --> O[Pipeline preparation]
    N --> O
    O --> P{Graphics pipeline cached?}
    P -->|Yes| Q[Bind pipeline]
    P -->|No| R[vkCreateGraphicsPipelines]
    R --> Q
    Q --> S[Draw]

Loading

Cache layers

Layer	Reuses	Typical identity	Lifetime / role
Shader binding cache	A combination of stage selections and binding state	ShaderState-derived binding key	Avoid rebuilding a binding that can reference existing stage modules
Per-stage module cache	Compiled vertex/geometry/fragment module state	Stage-specific shader state/key	Avoid regenerating and recompiling a stage in the same renderer lifetime
Persistent SPIR-V cache	Host-independent compiled shader artifact	Exact shader/source/compiler identity	Reuse previously compiled SPIR-V across renderer/process starts when valid
Graphics pipeline cache	Complete Vulkan graphics pipeline state	Pipeline key including shader modules plus fixed/dynamic state	Avoid repeated vkCreateGraphicsPipelines() for identical pipeline state

These layers solve different problems. Adding another cache without identifying which layer is missing reuse can simply duplicate existing work.

Important files

The PR #68 diagnostic head touched the following files while classifying this path:

File	Responsibility
hw/xbox/nv2a/pgraph/vk/shaders.c	Shader state selection, binding/module cache interaction, stage resolution
hw/xbox/nv2a/pgraph/vk/glsl.c	GLSL generation/compilation integration
hw/xbox/nv2a/pgraph/vk/renderer.c	Renderer lifecycle and shutdown integration
hw/xbox/nv2a/pgraph/vk/renderer.h	Vulkan renderer state and related structures
hw/xbox/nv2a/pgraph/vk/shader-identity.c/.h	Diagnostic exact-source/key classification
hw/xbox/nv2a/debug.h	Profiling/debug integration
tests/unit/test-xbox-vk-shader-identity.c	Identity-tracker tests

What happens on a stage-module miss

The diagnostic path added around shader_module_cache_entry_init() separated the expensive work into independent phases:

module-cache miss
    -> GLSL generation
    -> exact generated-source identity
    -> glslang compilation
    -> VkShaderModule creation
    -> SPIR-V reflection / layout extraction

PR #68 recorded independent timings for generation, compilation, Vulkan module creation, and reflection. It also classified a miss as one of:

• first-seen exact source,
• different key producing the same exact source,
• saturated diagnostic tracking state.

This distinction matters because a different state key can still generate identical GLSL. Conversely, a pipeline-cache miss may reuse every stage module and require no glslang work at all.

Measured evidence from PR #68

A repeatable PGR2 slow frame originally showed approximately:

Region	Time
pgraph_vk_bind_shaders()	23.906-25.034 ms
vkCreateGraphicsPipelines()	12.208-13.538 ms
Pipeline key/hash/lookup/layout	under 1.1 ms combined

The completed identity trace classified 141 stage-module misses:

Classification	Count
First-seen stage/source identities	121
Different-key / same-source repeats	20
Aggregate glslang time	463.419 ms
glslang time in duplicate-source variants	68.200 ms

The primary reproduced hitch was more important than the aggregate total:

Primary slow frame	Value
Guest interval	59.493 ms
Pipeline preparation	43.738 ms
Stage misses	2 vertex + 2 fragment
First-seen sources	4
Duplicate sources	0
glslang	18.217 ms
Vulkan module creation	0.043 ms
Reflection	1.269 ms

That result rejects a simple same-process key/source deduplication fix for the main hitch: the four shader sources in the critical frame were genuinely new.

Why NV2A_PROF_SHADER_GEN is not a compiler count

Shader-binding creation may reuse a previously created stage module. A counter attached to shader binding/generation therefore cannot be treated as an exact number of compiler invocations. Correct diagnosis requires counting the actual module-cache miss and the actual glslang call.

Ownership and lifetime

Generated GLSL and SPIR-V are CPU-side artifacts. VkShaderModule, reflected layout state, module-cache ownership, and graphics-pipeline objects belong to the Vulkan renderer lifecycle. Any persistent cache must recreate device-owned Vulkan objects rather than attempting to persist them directly.

A useful identity boundary is therefore:

stable source/compiler identity
        -> persisted SPIR-V artifact
renderer/device lifetime
        -> VkShaderModule + reflection/cache objects

Invalid or incompatible persistent artifacts must fall back to normal compilation.

Hardware fidelity boundary

NV2A does not compile GLSL or create SPIR-V modules. xemu is translating Xbox shader/combiner semantics into an executable representation for a modern host GPU.

flowchart LR
    A[NV2A shader / combiner state] --> B[Guest semantic translation]
    B --> C[Generated GLSL]
    C --> D[glslang / SPIR-V]
    D --> E[Vulkan shader module]
    E --> F[Host GPU program]

Loading

Compilation time is therefore host translation overhead, not an emulated NV2A execution delay that must be preserved. It is legitimate to cache, precompile or move that work off the draw dependency as long as the resulting program has the same guest-visible behavior and lifetime.

By contrast, deleting a state field from shader identity because it “reduces misses” is only legal if that field cannot alter effective NV2A shader behavior.

Performance implications

There are at least four independent optimization questions:

1. Can equivalent state keys be canonicalized safely when they generate byte-identical effective shader behavior?
2. Can known SPIR-V be loaded before gameplay instead of recompiling it?
3. Can genuinely first-seen shader work be removed from the immediate draw dependency?
4. Can Vulkan graphics-pipeline creation itself be reused or made cheaper independently of shader compilation?

The first two do not solve a genuinely new first-seen shader. That gap motivated Vulkan Ubershader & Hybrid Specialization.

Related Development

PR #68 — Vulkan: classify shader-module misses in the binding hot path

PR #68 was diagnostic instrumentation, not a merge candidate. It is now closed and unmerged, but its measurements remain the basis for separating shader-module cost from graphics-pipeline cost.

Key conclusions:

• The critical reproduced hitch contained genuinely first-seen shader sources.
• There was still measurable different-key/same-source duplication elsewhere.
• Module creation itself was tiny compared with glslang in the slow frame.
• Persistent SPIR-V reuse is useful for known shaders, but cannot by itself hide never-before-seen specialization.

For the architecture designed to cover that last case, continue with Vulkan Ubershader & Hybrid Specialization.

Recent identity and hybrid-family lessons

Three merged PRs refine what belongs in shader/pipeline identity.

• #131: values delivered as runtime uniforms must not create pipeline permutations unless they also affect executable or fixed-function state.
• #130: an unsupported global selector matters only when an active stage consumes the unsupported behavior.
• #129: canonical fallback-family availability and per-full-state owner admission are independent state.

flowchart LR
    A[Guest state] --> B[Executable identity]
    A --> C[Runtime uniform values]
    A --> D[Fallback admission]
    B --> E[Specialized pipeline family]
    D --> F[Per-owner admitted/rejected state]
    E --> G[Shared canonical fallback resources]
    F --> G
    C --> H[Uniform publication only]

Loading

Do not put a value into a pipeline/module key merely because it lives in a register copied near other pipeline state. Conversely, do not remove it until source generation, fixed-function state, reflection/layout, and host capability selection are proven independent.

For asynchronous preparation, keep retry enrollment explicit. Completion/failure fan-out should update only owners that actually entered preparation.