Vulkan Submission and Resource Retirement

Vulkan Submission and Resource Retirement

Lifecycle: Performance - Unproven · Type: Submission architecture
Reviewed source: 322986f4c5502b3fa887a9605ac8cd88e9b2c579
State: The synchronous route is source-confirmed. Its frame-time contribution is unmeasured.

Vulkan throughput pages: Overview · Submission & retirement · Native presentation · Dynamic uniforms · PVIDEO lifetime · GPU-resident consumers · Buffer placement

Status

Current pgraph_vk_finish() submits recorded upload/draw work, waits for its fence, then immediately resets staging offsets, descriptor-set use, and temporary framebuffer state. Auxiliary one-time commands call vkQueueWaitIdle().

Source seams:

• draw.c
• command.c
• renderer.h

Why it matters

A finish may be required because the CPU needs a result. It may also happen only because xemu wants to reuse the same descriptor array, staging range, command buffer, query storage, or framebuffer immediately.

Those are different dependencies.

Current path

flowchart TD
    A[Recorded upload and draw work] --> B[End rendering and command buffers]
    B --> C[Submit upload and draw batches]
    C --> D[Wait for submission fence]
    D --> E[Reset staging offsets]
    E --> F[Reset descriptor-set index]
    F --> G[Destroy temporary framebuffers]
    G --> H[Begin next batch]

Loading

pgraph_vk_end_single_time_commands() is stricter:

flowchart LR
    A[One-time command buffer] --> B[vkQueueSubmit]
    B --> C[vkQueueWaitIdle]
    C --> D[Caller continues]

Loading

Candidate path

Retain batch-owned resources until their submission completes. Start later work with another available context instead of waiting only to recycle the first one.

flowchart TD
    A[Record batch N] --> B[Submit and receive completion ticket]
    B --> C[Keep batch N resources immutable]
    C --> D{Batch N result needed by CPU now?}
    D -->|Yes| E[Wait for required ticket]
    D -->|No| F[Record batch N+1 with free resources]
    F --> G[Poll completed tickets]
    G --> H[Retire descriptors, ranges, framebuffers and queries]
    E --> H

Loading

A fence ring or timeline semaphore can represent progress. The important change is ownership, not the specific primitive.

What changes

• Submission no longer means immediate completion.
• Descriptor sets, staging ranges, framebuffers, queries, and referenced cache entries become batch-owned.
• Recycling happens after completion, not directly after vkQueueSubmit().
• CPU-visible reports/readbacks still wait for the exact producer they depend on.

Smaller probe

The current main uses one vkQueueSubmit() call with two submit batches and a binary semaphore between upload and draw work. A bounded probe can place both command buffers in one submit batch while retaining the final fence wait.

That tests submit-batch/semaphore overhead only. It does not test removal of CPU/GPU serialization.

Required guard

A resource cannot be rewritten, reset, destroyed, or evicted until every submitted command that references it has completed.

This includes:

• staging bytes;
• descriptor sets and uniform offsets;
• framebuffers and image views;
• query indices and report queues;
• pipelines, textures, and surfaces referenced by the batch;
• command buffers and synchronization objects.

Risks

• Reusing upload memory before the GPU consumes it.
• Evicting a pipeline or image still referenced by an older batch.
• Publishing report/readback data before its producer completes.
• Unbounded memory growth when retirement stalls.
• Teardown, reset, and snapshot paths stopping workers in the wrong order.
• Moving a wait to another function without increasing overlap.

Evidence

finish_calls{reason}
finish_wait_us{reason}
aux_queue_idle_us{reason}
batches_submitted
batches_in_flight_peak
resource_reuse_waits{resource_type}
retired_batches
staging_bytes_in_flight
GPU idle intervals
fixed-work CPU and frame p95/p99

The key comparison is waits removed because resources were independently owned, not fewer calls to a function named finish.

How to test

1. Record two or three batches that upload visibly different data into the same logical resource.
2. Verify each draw observes its own version while several submissions are in flight.
3. Force descriptor exhaustion, staging growth, query rollover, framebuffer changes, cache eviction, and readback.
4. Exercise reset, renderer teardown, snapshot save/load, and device-loss/error paths.
5. Run Vulkan synchronization validation during correctness tests.
6. Compare an uninstrumented Release candidate using fixed guest progress and wait time by reason.

Related

• Vulkan Throughput Candidates
• Color-Download Folding
• Transient-Buffer Growth
• Vertex Uploads and Render-Pass Lifetime
• Performance Methodology

Full technical reference

Primary reference: This subject page is the design and research reference for Submission and Resource Retirement. Pull requests are implementation/test history only.
Lifecycle: Performance - Unproven · Mechanism: Asynchronous batch ownership architecture
Reviewed source: 322986f4c5502b3fa887a9605ac8cd88e9b2c579 · Acceptance class: intended exact or behavior-preserving for its admitted scope

1. Scope

Decouple queue submission from immediate CPU completion and recycle descriptors, staging ranges, framebuffers, queries, and cache references only after the owning submission completes.

This entry remains Unproven: the source path and opportunity are concrete, but no accepted xemu runtime result has established that the candidate improves representative workloads. The page defines what must be bypassed, what remains mandatory, and how the hypothesis can be rejected.

2. Source seams

File / function	Role
hw/xbox/nv2a/pgraph/vk/draw.c	pgraph_vk_finish() submit/fence/reset path
hw/xbox/nv2a/pgraph/vk/command.c	single-time submit followed by vkQueueWaitIdle()
hw/xbox/nv2a/pgraph/vk/renderer.h	single active resource arrays and lifetime state

3. Current execution path

flowchart TD
    N0["Record upload/draw work"]
    N1["Submit"]
    N0 --> N1
    N2["Wait for fence or queue idle"]
    N1 --> N2
    N3["Reset offsets/set index"]
    N2 --> N3
    N4["Destroy/recycle temporary objects"]
    N3 --> N4
    N5["Record next batch"]
    N4 --> N5

Loading

4. Candidate execution path

flowchart TD
    N0["Record batch N with owned resources"]
    N1["Submit and obtain completion ticket"]
    N0 --> N1
    N2["Record batch N+1 using free context"]
    N1 --> N2
    N3["Poll/wait only for real dependency"]
    N2 --> N3
    N4["Retire N resources on completion"]
    N3 --> N4

Loading

5. Why it may help

The current architecture uses completion as a convenient resource-reuse boundary. A faster GPU cannot overlap with CPU preparation when the CPU waits merely to reset the same arrays. The candidate is not 'delete the wait'; it is make reuse asynchronous and retain ownership until safe retirement.

6. Work removed

• CPU waits caused only by immediate reuse
• Queue-wide idle for auxiliary work that can be ordered
• GPU idle gaps caused by drained producer pipeline

7. Work that remains mandatory

• Waits for genuine CPU-visible results
• Submission ordering
• Resource immutability
• Backpressure when all contexts are in flight
• Teardown and snapshot quiescence

8. Implementation sketch

Batch *batch = acquire_free_batch_context();
record_current_work(batch);
batch->ticket = submit_batch(batch);
enqueue_in_flight(batch);

while ((done = poll_completed_batch()) != NULL) {
    retire_batch_resources(done);
    release_batch_context(done);
}

if (cpu_needs_result(ticket))
    wait_for_ticket(ticket);

The sketch defines the intended boundary; it is not a complete patch and does not replace source review, ownership design, or focused tests.

9. Correctness contract

• Every resource referenced by a submission remains immutable/alive until completion.
• CPU-visible reports/readbacks wait for the exact producer.
• Backpressure is bounded and cannot grow memory without limit.
• Reset/teardown stops producers and retires or waits in a defined order.

10. State, ownership, invalidation, and fallback

Each batch owns command buffers, staging subranges, descriptor sets, query slots, temporary framebuffers/views, and strong references to cached resources. Completion tickets govern reclamation. Global caches need per-entry last-use tickets or retained references.

The maintained reference path remains the fallback whenever admission, capability, lifetime, or validation is uncertain. A fast result with an unproven dependency is not acceptable as an exact optimization.

11. Advantages

• Potential CPU/GPU overlap
• Eliminates waits caused only by reuse strategy
• Creates explicit lifetime model
• Can absorb several existing finish reasons

12. Costs and failure modes

• Large architectural change
• Lifetime bugs can corrupt GPU work
• Higher peak memory
• Driver queue depth/backpressure complexity
• May move waits rather than remove them

13. Evidence counters

Counters must be cheap, subsystem-owned, and snapshotted outside the hot loop. Record at least:

• finish_calls/wait_us_by_reason
• aux_queue_idle_us
• batches_in_flight
• resource_reuse_waits
• staging_bytes_in_flight
• GPU idle intervals
• CPU preparation overlap
• frame p95/p99

A local reduction is not enough. Evidence must connect route reachability and removed work to fixed guest progress, CPU/GPU time where applicable, and p95/p99 behavior.

14. Qualification matrix

• Distinct visible data in several in-flight batches
• Descriptor/staging exhaustion
• Query/report rollover
• Cache eviction and framebuffer changes
• Readback and CPU result waits
• Reset/snapshot/teardown/device failure
• Vulkan validation
• Release fixed-work A/B

Correctness runs may use validation layers, detailed comparisons, and fault injection. Performance acceptance must use optimized Release builds with matching symbols and minimal instrumentation.

15. Acceptance rule

Promote incrementally when independent resource classes can retire correctly and overlap/wait evidence improves without unbounded memory or lifecycle failures.

16. Rejection rule

Reject designs that only rename/move the wait, reuse memory early, require global queue idle for ordinary retirement, or cannot bound in-flight ownership.

17. Related architecture and history

Vulkan-Renderer · Performance-Unproven · Hot-Path-Vulkan-Color-Download-Folding · Hot-Path-Vulkan-Transient-Buffer-Growth

PR pages may be linked as supporting evidence, but they are not substitutes for this reference.