APU Frame Lock Contention

APU Frame-Lock Contention

Lifecycle: Performance - Unproven · Type: Measurement-first synchronization investigation
Reviewed source: 322986f4c5502b3fa887a9605ac8cd88e9b2c579
Question: Does the APU frame thread hold its main mutex long enough to block vCPU MMIO or control activity materially?

APU/DSP investigation directory · Voice-worker dispatch

Verified starting point

mcpx_apu_frame_thread() acquires d->lock before entering its main loop. While holding the mutex, active operation can call throttle() and se_frame(). se_frame() performs VP processing, DSP processing, and monitor/audio work. The thread temporarily drops the mutex only around BQL-protected interrupt update.

GP/EP writes and voice-control operations also acquire the same APU mutex. A plausible interference chain is therefore:

APU thread owns d->lock
    → throttle or wait for audio queue
    → process voices / wait for workers
    → reduce mixbins
    → write/run DSP
    → produce monitor output

vCPU or control path
    → attempts APU MMIO/GP/EP write
    → waits for d->lock

A CPU sampling profile may show the vCPU as waiting rather than attribute time to the APU phase that owns the lock. The first task is instrumentation, not lock splitting.

Source seams

• hw/xbox/mcpx/apu/apu.c
  • mcpx_apu_frame_thread()
  • throttle()
  • se_frame()
  • idle timed wait and BQL interrupt update
  • pause/reset/snapshot quiescence
• hw/xbox/mcpx/apu/vp/vp.c
  • voice lock/unlock
  • worker wait and voice-state commits
• hw/xbox/mcpx/apu/dsp/gp_ep.c
  • GP/EP memory/register writes under d->lock
• APU MMIO and method paths that signal the frame condition or mutate shared control state.

Current ownership flow

sequenceDiagram
    participant V as vCPU / MMIO caller
    participant A as APU frame thread
    participant W as Voice workers
    participant D as GP/EP DSP

    A->>A: lock d->lock
    A->>A: throttle / timed waits
    A->>W: dispatch voices
    A->>W: wait for completion
    W-->>A: worker matrices
    A->>A: reduce / build VP output
    A->>D: write mix block and run DSP
    A->>A: monitor/audio submission
    V->>A: request d->lock for write
    Note over V,A: vCPU may wait until complete frame ownership ends
    A->>A: unlock d->lock
    A-->>V: mutex acquired

Loading

The diagram is a possible timing, not proof of material contention. The instrumented wait distributions decide whether the overlap occurs in real titles.

Phase 1: scoped timing only

Every important mutex acquisition should record caller and wait duration without formatting/logging in the hot path:

typedef enum ApuLockCaller {
    APU_LOCK_FRAME_THREAD,
    APU_LOCK_GP_WRITE,
    APU_LOCK_EP_WRITE,
    APU_LOCK_VOICE_METHOD,
    APU_LOCK_RESET,
    APU_LOCK_SNAPSHOT,
    APU_LOCK_AUDIO_RECONFIG,
} ApuLockCaller;

int64_t wait_start = qemu_clock_get_ns(QEMU_CLOCK_REALTIME);
qemu_mutex_lock(&d->lock);
apu_perf_record_lock_wait(caller,
    qemu_clock_get_ns(QEMU_CLOCK_REALTIME) - wait_start);

The frame thread also needs phase hold timing:

apu_phase_begin(d, APU_PHASE_THROTTLE);
throttle(d);
apu_phase_end(d, APU_PHASE_THROTTLE);

apu_phase_begin(d, APU_PHASE_VP);
mcpx_apu_vp_frame(d, mixbins);
apu_phase_end(d, APU_PHASE_VP);

Suggested phases:

idle/timed wait while lock held
throttle/audio queue wait
voice-list traversal and preparation
worker scheduling/wait
worker reduction
VP→DSP transfer
GP DSP
EP DSP
monitor/audio submission
IRQ update/BQL transition
pause/reset/snapshot quiescence

Record cumulative counters and histograms; do not emit one log line per lock operation.

Evidence model

flowchart TD
    A[Lock wait/hold counters] --> B{vCPU/control waits material?}
    B -->|No| C[Close synchronization redesign; retain evidence]
    B -->|Yes| D[Identify owning phase and caller]
    D --> E{Work can execute outside lock with stable inputs?}
    E -->|No| F[Optimize owning phase first]
    E -->|Yes| G[Prototype shorter ownership or message/snapshot design]
    G --> H[Race/lifecycle testing]
    H --> I[Release A/B]

Loading

If worker or DSP optimization shortens the hold time enough, a lock redesign may become unnecessary. Therefore, contention should be measured before and after other APU work.

Candidate redesigns after evidence

1. Snapshot immutable frame inputs

Copy the guest/control state needed for one 32-sample block while holding the mutex, then process the immutable snapshot outside it. Commit state changes under a shorter ownership window.

qemu_mutex_lock(&d->lock);
ApuFrameInputs in = capture_frame_inputs(d);
qemu_mutex_unlock(&d->lock);

ApuFrameResult out = process_frame(&in);

qemu_mutex_lock(&d->lock);
commit_frame_result(d, &in, &out);
qemu_mutex_unlock(&d->lock);

The hard part is guest writes that occur between capture and commit. The design needs generations/conflict detection or a defined block-boundary model.

2. Message queue for writes

MMIO/control callers enqueue small ordered mutations without waiting for the complete frame. The APU thread applies them at safe boundaries.

This can change when a guest write becomes observable and must be treated as a timing-model change unless the hardware already latches at equivalent boundaries.

3. Separate control and processing locks

Separate rarely contended control/register structures from long-lived processing state. This requires a complete shared-state inventory; splitting one mutex into two without ordering rules can create deadlocks or inconsistent snapshots.

4. Read-side versioning

Use immutable/versioned configuration snapshots for fields read frequently but changed rarely. Writers publish a new version; the frame retains the old version until completion.

5. Shorten owning phases

Worker dispatch, HRTF, VP→DSP transfer, and DSP execution may be optimized without changing synchronization. This is the lowest-risk response when one phase dominates hold time.

Idle wakeups and BQL

When the APU is inactive, the frame thread performs a short timed wait while holding/reacquiring its mutex and may drop it to acquire the BQL for interrupt update.

Keep this as a sub-investigation:

inactive timed wakeups
wakeups that observe no state change
BQL acquisitions
BQL wait/hold time
IRQ update calls
actual IRQ level transitions

Do not create an independent idle-wakeup optimization unless these counts are significant.

Protected invariants

• Ordered application of guest MMIO/method writes.
• Voice lock semantics.
• DSP memory/register visibility.
• DMA control and interrupt timing.
• Condition-variable wakeups and no lost wake.
• Pause/reset/snapshot quiescence.
• No lock-order cycle with BQL, worker mutexes, audio/SDL objects, or DSP callbacks.
• Frame state cannot be committed over newer guest writes silently.
• Teardown cannot leave vCPU/control callers waiting forever.

Required counters

lock acquisitions by caller
wait count / total / max / p50 / p95 / p99 by caller
hold count / total / max by owner phase
vCPU thread waits attributed to APU lock
frame-thread idle wakeups
condition signals/broadcasts
BQL acquisitions, waits and actual IRQ transitions
worker wait time while APU lock is held
GP/EP run time while lock held
audio submission/throttle wait while lock held
APU frame deadline misses
audio underruns

A sampling profiler should be correlated with scoped lock counters. Neither alone is sufficient.

Test matrix

Scenario	Required property
Concurrent voice-register writes	No lost/reordered write
GP/EP memory writes during processing	Defined visibility and no corruption
Voice lock/unlock	Stable ownership and no deadlock
DMA control/status activity	Correct ordering and interrupts
Heavy HRTF/voice frame	Hold phases attributed correctly
Audio queue backpressure	Throttle wait identified separately
Reset and pause/resume	Quiescence completes
Snapshot save/load	Canonical state and thread state safe
Shutdown	No waiter or worker remains blocked
Audio engine reconfiguration	No use-after-free or stale stream state
Artificial lock delay	Counters and vCPU attribution detect it

Advantages

• Can close the issue cheaply if contention is negligible.
• Identifies the actual owning phase instead of blaming a broad lock.
• Supports prioritization among worker, HRTF, DSP, and audio-output work.
• A successful shorter-ownership model can improve vCPU responsiveness without changing local APU throughput.

Costs and risks

• Timing instrumentation can perturb short waits if too detailed.
• Lock splitting is a high-risk race/deadlock project.
• Message queues or frame snapshots may change guest-visible write timing.
• A long hold is not harmful when no other thread needs the lock.
• vCPU waiting may be dominated by another subsystem or scheduler decision.

Acceptance rule

Move to Researched when low-overhead counters show material vCPU/control wait time and identify repeatable owning phases. Any redesign must then pass race/lifecycle tests and improve whole-emulator progress or tails in Release builds.

Rejection rule

If p95/p99 wait time and total blocked vCPU time are negligible, record the negative result and do not split the lock. Optimize only independently hot APU phases.

Related

• APU / DSP Performance Investigations
• Voice-Worker Dispatch and Mix Reduction
• HRTF Optimization
• DSP JIT Qualification
• Peripheral Opportunity Counters