Memory and DMA

Memory & DMA

Memory in xemu is both guest architectural state and a synchronization boundary between the CPU, device models and host-side renderer/audio/network/storage implementations.

The most useful rule is:

A guest physical address tells you where the Xbox thinks data lives; it does not tell you which host representation currently contains the newest copy.

Machine memory map foundation

hw/xbox/xbox.c creates the main Xbox RAM as one QEMU MemoryRegion named xbox.ram and maps it at guest physical address zero.

Guest physical address space
    ├─ Xbox RAM from address 0
    ├─ PCI/MMIO windows
    ├─ ISA / port-I/O space
    ├─ BIOS / MCPX boot ROM mappings near top of address space
    └─ aliases / bridge windows created by chipset devices

hw/xbox/xbox_pci.c then creates the PCI host bridge and a PCI-hole alias above the configured RAM region so PCI devices can expose MMIO in the expected physical range.

QEMU MemoryRegion model

A MemoryRegion can represent different kinds of guest-visible memory:

Region type	Example in xemu	Behavior
RAM	xbox.ram	Backed by host memory and directly stores guest bytes
ROM	Xbox BIOS	Guest-readable firmware image with read-only semantics
Alias	Mirrored BIOS / PCI hole	Maps another region into an additional address range
MMIO / I/O	PCI BARs, SMBus registers	Read/write callbacks implement device behavior instead of plain memory storage

A CPU load from RAM can often remain on a fast translated-memory path. A load from MMIO must enter the device model because the read itself may have side effects or synthesize a register value.

CPU access path

flowchart TD
    A[Guest x86 load/store] --> B[TCG memory access]
    B --> C{Mapped region}
    C -->|RAM| D[Read/write guest RAM backing]
    C -->|ROM| E[Read firmware mapping]
    C -->|MMIO / I/O| F[Device callback]
    F --> G[Device state / IRQ / DMA side effects]

Loading

The cost and synchronization behavior can therefore change sharply when the same guest code switches from ordinary RAM to a device BAR or port-I/O access.

PCI and DMA-visible memory

The Xbox PCI devices share the guest physical memory system. Several major devices are initialized with access to the machine RAM region or QEMU address space so they can consume guest buffers/descriptors.

Examples include:

• NV2A graphics data and pushbuffers.
• MCPX APU audio buffers/state.
• NVNet network descriptors/buffers.
• IDE/storage DMA paths through the PCI device model.
• USB controller transfer descriptors/buffers.

The exact descriptor/register format is device-specific, but the common model is:

Guest writes descriptor/pointer
    ↓
Device model interprets guest address
    ↓
DMA mapping / address-space access
    ↓
Device consumes or produces bytes
    ↓
IRQ/status tells guest about progress

A DMA address is guest data. Validate it against the correct guest address space and transfer length before treating it as a host pointer.

Why host pointers are not guest addresses

xemu can frequently derive a host pointer into RAM for fast access, but the two identities must not be confused:

guest physical / DMA address
        ↓
QEMU memory mapping
        ↓
host backing pointer

The mapping can depend on device DMA objects, aliases, bounds and current guest configuration. Caching only a host pointer while forgetting the guest identity that produced it can cause stale-resource reuse after a remap.

This is especially relevant to texture and vertex caches.

NV2A DMA objects

NV2A does not treat every register value as a raw host-accessible address. Guest GPU state selects DMA objects and offsets that define where vertex, texture, pushbuffer and other data comes from.

A useful conceptual path is:

NV2A method/state
    ↓
DMA object selection
    ↓
guest address + length validation
    ↓
RAM / VRAM bytes
    ↓
PGRAPH / renderer consumer

Changing the DMA object can change the effective resource identity even if another piece of guest state appears unchanged. This is why current Vulkan texture reuse includes source/DMA identity checks rather than trusting a dirty bit alone.

Dirty tracking

QEMU can track writes to memory for different clients. The NV2A renderer uses dirty information to determine when guest CPU-visible memory has changed and a host-side cached/mirrored resource may need synchronization.

The important distinction is:

dirty
    =
guest memory changed relative to a consumer's known state

not necessarily
    =
guest memory is the newest authoritative copy of all overlapping GPU data

A GPU-rendered surface may be newer than guest RAM even if a nearby RAM page is also dirty. Correctness requires knowing both who changed the bytes and which range overlaps the consumer.

Graphics memory ownership

Graphics makes memory ownership most visible because several representations can coexist:

flowchart LR
    A[Guest RAM / VRAM] <--> B[Renderer mirror]
    B <--> C[Staging buffer]
    C <--> D[Host GPU buffer/image]
    D --> E[GPU-authored surface/result]
    E --> A

Loading

The arrows are conditional, not automatic. A transfer should occur only when the next consumer needs a representation that is stale.

CPU/guest becomes newer

Example:

Guest CPU writes vertex bytes
    ↓
NV2A dirty tracking sees changed range
    ↓
renderer updates vertex mirror before GPU use

GPU becomes newer

Example:

GPU renders into surface
    ↓
host surface contains newest pixels/bytes
    ↓
CPU-side consumer needs overlapping guest memory
    ↓
surface must be downloaded/read back first

This is the core coherency problem behind current surface/vertex work.

Vertex mirror

The Vulkan backend keeps a host-side mirror used for guest vertex RAM.

Updating the mirror is not always just a copy problem. If an active Vulkan command buffer already contains a draw that references a page, overwriting the shared mapped mirror can change the data that earlier recorded work observes when the GPU eventually executes it.

The backend therefore needs one of these properties:

1. The update occurs before any active recorded draw depends on the range.
2. Lifetime tracking proves the range has not been consumed by earlier recorded work.
3. An ordered staged copy/barrier preserves the intended sequence.
4. A separate immutable/versioned allocation gives old and new draws different storage.

See Vertex Uploads & Render-Pass Lifetime.

Surface overlap

A broad “surface overlaps this memory” test is not enough by itself. The backend needs the exact fetched/consumed span.

For a vertex attribute, for example, effective bytes depend on:

• Base DMA mapping.
• Attribute offset.
• Minimum/maximum element.
• Stride.
• Element size.

If the coherency check covers far more memory than the draw actually fetches, xemu can trigger needless GPU readbacks. If it covers too little, stale data can be consumed.

PR #85 is active work around this exact-range/freshness boundary.

Staging buffers

A staging buffer gives the CPU a host-visible place to prepare bytes that are then copied into a GPU-consumed resource in command order.

CPU copies bytes to staging allocation
    ↓
flush mapped allocation if required
    ↓
record barrier
    ↓
record vkCmdCopyBuffer / image copy
    ↓
record barrier for later consumer

Advantages:

• Explicit GPU ordering.
• Earlier recorded work can keep seeing the old destination contents until the copy command executes.

Costs:

• CPU copy.
• Staging capacity management.
• Transfer command.
• Barriers.
• Potential render-pass break.
• Potential submission/fence when space is exhausted.

The right optimization question is therefore not “can staging be removed?” but “can the lifetime prove an ordered transfer is unnecessary for this exact range?”

Buffer capacity and rollover

Finite staging/uniform/descriptor resources can force the renderer to retire work before reusing memory.

A capacity event can become:

allocation request
    ↓
no safe free space
    ↓
submit / finish active GPU work
    ↓
wait or retire ownership
    ↓
reuse / grow buffer

Increasing capacity can reduce how often that path runs, but if the workload is ultimately blocked by a report/fence or another resource, the same wait may simply move. Measure total work and guest progress.

Networking and audio DMA

Graphics is not the only DMA consumer.

NVNet

Guest network descriptors/buffers are interpreted by the NVNet device model before packets move to/from QEMU's network backend. Descriptor ownership and IRQ completion are guest-visible semantics.

MCPX APU

The APU is initialized with the Xbox RAM region and processes guest-memory-backed audio state/buffers. DSP/voice processing is therefore coupled to guest memory lifetime just as graphics and networking are, even though the device semantics differ.

These paths should not reuse renderer-specific dirty/freshness assumptions automatically. Each device owns its own descriptor and completion contract.

ROM and boot overlay ownership

Memory ownership also matters during boot.

The BIOS is mapped as ROM, while the MCPX boot ROM is overlaid on the top of the BIOS mapping. The LPC chipset can disable that overlay as boot progresses.

Conceptually:

reset
    ↓
MCPX boot-ROM overlay enabled
    ↓
early boot executes
    ↓
LPC config disables overlay
    ↓
underlying BIOS mapping becomes visible

This is address-space ownership rather than DMA, but the same lesson applies: the byte returned for a guest physical address depends on the active mapping, not only the numerical address.

Memory-access fidelity and fast paths

Not every device reaches guest memory through the same abstraction.

flowchart TD
    A[Guest/device memory request] --> B{Known ordinary RAM?}
    B -->|CPU translated access| C[TCG TLB / RAM fast path]
    B -->|PCI DMA identity required| D[pci_dma_read/write]
    B -->|Device caches stable RAM backing| E[Direct host RAM pointer]
    B -->|MMIO / alias / ROM| F[MemoryRegion dispatch]
    C --> G[Guest bytes]
    D --> G
    E --> G
    F --> G

Loading

Current examples include NVNet using pci_dma_read()/pci_dma_write() for descriptor and packet traffic, while MCPX APU initialization also caches memory_region_get_ram_ptr() for the Xbox RAM region. The latter is faster but relies on the invariant that the device is operating on the stable RAM backing it was given.

Direct pointers are an optimization contract

A direct host pointer removes generic mapping work. It must not be generalized to an address that may resolve to MMIO, ROM, an alias with side effects, or another address space. Likewise, bypassing QEMU dirty/access machinery can break snapshot or renderer coherency if that machinery is part of the consumer's correctness contract.

Memory-controller fidelity

Current machine construction still labels the Xbox memory-controller function with FIXME: Stub the memory controller and realizes a pci-testdev. The RAM map itself is functional, but this means PCI-visible memory-controller behavior is not a complete model. Software that probes undocumented controller registers can therefore behave differently from hardware even when normal RAM accesses work.

See Hardware Fidelity & Emulation Boundaries and MCPX APU & Audio.

Snapshot implications

Snapshotting must preserve guest-visible memory and device state while reconstructing derived host resources.

State	Snapshot expectation
Guest RAM	Serialized/restored as guest memory
Device registers/descriptors	Serialized through device VMState as required
Host GL/Vulkan buffer handles	Recreated, not serialized as guest state
Dirty/freshness metadata	Restore/rebuild only if needed to preserve correct ownership
GPU-authored guest-visible bytes	Must be made coherent before snapshot when required

A renderer reset/load that discards a host surface before downloading guest-visible data can lose state even if guest RAM itself was serialized correctly.

Debugging a memory/coherency bug

Trace the bytes, not just the object names.

1. Identify the exact guest address and byte range.
2. Identify the DMA/resource object that produced that range.
3. Determine the last writer: guest CPU, device DMA, or host GPU.
4. Determine where the newest representation currently lives.
5. Identify the next consumer.
6. Find the synchronization step that should make the consumer's representation current.
7. Verify ordering against already queued/recorded work.
8. Only then optimize copies or overlap checks.

A useful trace record includes:

guest_addr
size
resource / DMA identity
last-writer class
consumer class
dirty/freshness state
copy/readback path
command-buffer generation/batch where relevant

Source map

Source	Responsibility
hw/xbox/xbox.c	Creates xbox.ram, BIOS/MCPX ROM mappings and attaches major RAM-using devices
hw/xbox/xbox_pci.c	PCI address-space windows, PCI hole, buses and MMIO routing
QEMU system/memory* / address-space code	Core MemoryRegion and address-space dispatch infrastructure
hw/xbox/nv2a/*	NV2A DMA, memory dirty tracking and guest GPU access
hw/xbox/nv2a/pgraph/gl/*	OpenGL-side mirrors/surfaces/readback
hw/xbox/nv2a/pgraph/vk/*	Vulkan buffers, staging, surfaces, vertex mirror and coherency
hw/xbox/mcpx/apu/*	Guest-memory-backed APU/audio processing
hw/xbox/mcpx/nvnet/*	Guest network descriptors/buffer DMA behavior

Dirty clients are consumer-specific ordering streams

A dirty bitmap read can be destructive. memory_region_test_and_clear_dirty() acknowledges one client's notification; it is not a durable global history of writes.

Issue #106 showed why independent consumers cannot always share a client:

flowchart LR
    A[CPU or device write] --> B[Mark texture client]
    A --> C[Mark surface client]
    B --> D[Texture consumer clears texture notification]
    C --> E[Surface consumer still sees its notification]

Loading

If texture, vertex, and surface coherency share one client, the first consumer can erase evidence required by another. Document each client's producers, consumers, authority meaning, accelerator coverage, reset behavior, and clear points.

See PGR2 Race-to-Menu Corruption — Lessons Learned for the stale-surface readback failure and Graphics Transition Debugging Tools for the dirty-client audit procedure.

Related Development

• PR #80 — merged guarded texture reuse; demonstrates that DMA/source identity belongs in cache validation.
• PR #85 — active surface/vertex freshness correctness work.
• PR #87 — active direct-write versus ordered-copy lifetime work.
• PR #89 — active immutable/versioned vertex-storage research.

See Development.

Related pages

• System Architecture
• Platform & I/O
• NV2A Graphics Architecture
• Vulkan Renderer
• Vertex Uploads & Render-Pass Lifetime

Recent PR lessons — range validity, derived identity, and consumer spans

Validate the complete access — PR #121

A ZPASS report is 16 bytes. Proving only that its starting offset is inside a DMA object is insufficient. Validate the complete record against the RAMIN descriptor, its inclusive DMA limit, VRAM's exclusive size, and integer overflow. Use subtraction-safe arithmetic and test exact fits plus one-byte-short spans.

Stable handle does not imply stable backing — PR #124

A RAMIN DMA descriptor can be rewritten behind the same handle. Any cache whose correctness depends on the physical source must re-resolve or version that source before reuse.

Stored compressed layout is not sampled LOD — PR #127

BC cubemap storage must count a complete 4x4 block even for 1x1 mips, and every encoded mip contributes to face stride even if sampling later clamps LOD. Compute stored layout once, align the completed face once, and reject overflow before pointer arithmetic.

1x1 BC1 -> one 8-byte encoded block -> nonzero face stride -> six distinct faces

Resolve the full consumed span before pointer formation — PR #125

OpenGL vertex fetch now resolves the complete bytes consumed, reconciles overlapping GPU-authored surfaces, refreshes the vertex mirror, and only then performs CPU-side constant-attribute decoding. A stride of zero consumes one complete element.

flowchart LR
    A[Vertex fetch state] --> B[Checked consumed span]
    B --> C[DMA/VRAM bounds]
    C --> D[Reconcile newest representation]
    D --> E[Form pointer / decode / draw]

Loading

These rules express one principle: numerical address, guest handle, and current byte authority are separate identities.

Related hardware oddities

• DRAM geometry and expansion — revision-1.6 128 MiB correction and the 256 MiB A8/A9 auto-precharge conflict.
• GPU address domains above 64 MiB — software masks, DMA aperture, and installed-memory decoding must remain separate.
• Halo 2 resource aliasing and tiled pitch — multiple interpretations and producers over shared bytes.
• PFIFO calls, payloads, and endpoints and semaphore DMA object quirk — generation-level leads requiring Xbox tests.