Memory Lifetime and Reduction Plan

Memory lifetimes and incremental reduction plan

Current measured target

Merged main d4350b60 retains 408 KiB of kernel image backing versus 180 KiB in the reference, a 228 KiB image gap. At the identical public guest initial checkpoint, available memory differs by 572 KiB. Availability includes runtime and probe allocations; it is not a firmware-only subtotal. These measurements are separate from the older 639-page physical census and must not be combined across revisions.

Three tested fixes have landed: unused storage error backing (#126), overwritten dead-stack retention policy (#127), and unnecessary eager HAL bounce backing (#129). The combined initial recovery is 84 KiB. The detailed function/table ledger is refreshed from this merged source, with exact exports and the conservative static graph recomputed rather than inherited from old addresses.

Owned boot placement versus resident backing

These are exact current header placements, not permanent reservation totals. The loader hands a prepared image to the kernel; INIT/resource and retired bootstrap lifetimes are separate from the mapped address window.

Item	64 MB image	128 MB image	Interpretation
Kernel virtual base	0x84000000	0x84000000	Stable guest virtual addresses for the image.
Kernel initial physical base	0x03400000 (52 MiB)	0x04000000 (64 MiB)	Physical placement, distinct from virtual aliases.
Loader initial physical base/limit	0x03800000..0x03820000	0x07D00000..0x07F00000	Bootstrap placement; not all stays resident.
Initial low page-table base	0x03BC0000	0x07F00000	Installed-table lifetime must be checked before freeing backing.
Low PCR region	0x00030000	0x00030000	Sixteen-page boot mapping.
LoaderBlock/auxiliary pages	0x00040000 / 0x00041000	Same	Handoff placement.
Kernel virtual mapping window	4 MiB	4 MiB	Does not imply 4 MiB resident kernel backing.

Source: boot/nxldr/boot-layout.h, merged reconstructed tree 1277f2b7.... The retained measured kernel is 102 pages / 408 KiB. The header also declares 52 retired bootstrap pages (32 loader plus 20 obsolete tables); this is not a new unreclaimed 208 KiB opportunity. The already-merged reclamation/reset work and actual allocator ownership must govern their status. This page does not infer additional live/free pages from addresses alone.

Storage/PnP: public retirement is insufficient

The owned boot seal changes seven driver objects before initialization memory is freed. Their public PnP handler becomes the resident invalid-request fallback; public AddDevice is cleared. Existing object lookup failures are skipped. The initiating boot thread returns from the discard routine before INIT is freed.

Driver	Retired public PnP target	Retired public AddDevice target
Pci	PciPnpControl	PciAddDevice
pciide	PciIdeXDispatchPnp	PciIdeXAddDevice
atapi	AtaDispatchPnp	AtaAddChannel
Disk	ClassGlobalDispatch	ClassAddDevice
Cdrom	ClassGlobalDispatch	ClassAddDevice
partmgr	PartMgrPnp	PartMgrAddDevice
NxkBus	PdoDispatchPnp	No corresponding assignment

This does not clear every copy. Class drivers retain private initialization callbacks and a private dispatch table containing ClassDispatchPnp. Device extensions refer to that table. ATAPI copies the channel interface's prepare/start/reset/IRQ/DMA callbacks. IDE retains its periodic timer operation and completion DPC. Storage retry, notification, worker and IRP completion callbacks remain published.

The ten explicitly forced-resident storage roots total 17,887 gross compiler input-section extent bytes (potentially including padding) in merged tree 1277f2b7.... That total excludes shared callees, is not a complete current-page deletion candidate, and establishes no reclaimable RAM. Generic PnP worker source is excluded from the Xbox build; it must not be charged merely because it exists in the repository.

Owned references: ntoskrnl/io/iomgr/driver.c boot seal; ntoskrnl/ex/init.c discard ordering; drivers/storage/class/classpnp/{class.c,dispatch.c} private copies; drivers/storage/ide/atapi/{fdo.c,atapi.c,portstate.c} interface/DPC ownership; ntoskrnl/xb/ide.c timer and completion; tools/init-force-resident.list retained roots. The line-level audit inspected candidate tree 17710a83; all ten listed core storage/placement files were checked byte-identical to merged tree 1277f2b7. Size data were freshly remeasured on that merged tree. The complete source audit and identity check stay in the lab evidence, not product main.

Keep, already discard, investigate

Item	Current classification	What would permit reduction
Boot persistence descriptors, 20,200 bytes	Already INIT/discarded	Do not count as a future saving.
Boot-named stack, 12,288 bytes	Live idle stack	Replace its actual runtime role before reclaiming.
Double-fault stack, 4,096 bytes	Live fault handling	A different proven fault-stack contract.
PFN links/pin state, 96 KiB at 64 MB	Live allocator state	Smaller representation preserving pins, overflow and deferred free.
Pool page metadata, 12 KiB	Live run/ownership state	Populated-region representation with equal failure/query behavior.
Ordinary cache plus private reserves, 96 KiB	Runtime I/O state	Drain/lifetime proof or a write-back path that eliminates recursive reserve demand.
Unicode/status/font readonly tables	Readonly placement candidates	Stable ROM mapping, correct addresses/relocations and consumer lifetime; access-cost measurement.
Exported or address-taken functions unobserved in traces	Usage unknown	Complete root/consumer proof; no removal based solely on coverage absence.
Sealed public PnP targets with private copies	New discard unknown	Retire each copy and prove in-flight requests/returns drained.
Storage I/O, IRQ, timers, retries and completions	Keep	A smaller runtime implementation preserving media/errors/reset contracts.

Pool-return policy, repaired churn and remaining qualification

The first immediate-release draft of #131 retained one warm empty page per class/domain. The frozen guest completed all 576 allocation lifetimes and recovered 100 KiB, but repeated 128-object bursts were about 21%, 121% and 249% slower for requests of 64, 512 and 1,024 bytes. That policy released useful backing between bursts and was replaced; these are historical failed-policy measurements.

Current head 5657c2ab63dfbee6544774b66c87bb7de4587868 preserves current/prior class demand and trims quiet surplus with a bounded resident timer/DPC. Allocation/free performs no clock/timer query, and there is no additional worker stack. The revised component ABBA/BAAB reduces the burst overhead to +2.48%, +3.78% and +5.43%, respectively, with all 336 samples matching outputs/iterations. This is guest allocation cost, not a title FPS result.

The integrated native memory guest reconfirms 86 → 61 mapped pool pages, including 58 → 33 bucket pages, the same 27 occupied bucket pages and the same 28 non-bucket pages. Thus 25 pages / 100 KiB are returned after activity; compiled static image backing does not add a page. Earlier public counters reported 70 → 45 pool pages plus 16 ordinary cache pages: do not add these overlapping views together.

The PR is Ready for review, unmerged/HOLD. Final ABBA/BAAB completed eight PGR2 Windows and eight Conker Deck processes, with all exits 0 and all start/end scenes manually checked. PGR2 is effectively neutral. Conker average guest cadence changes −0.24%, while p95/p99 increase +8.78%/+9.29%, with worse tails in both orders. Two B runs have p99 around 99–100 ms versus 83–85 ms for all four A runs; two other B runs are normal. BIOS causality is unresolved. The recovery is not counted as merged main or a cold-start census.

The paired dispatch/allocator diagnostic subsequently reproduced a larger Conker slowdown. Direct trim totaled only 1.988 ms in the measured menu and timer/DPC inclusive elapsed did not rise. The larger observed cost moved into renderer surface-update and upload waits, with approximately the same draw/upload/submit counts. This is an instrumented attribution result, not a qualified release comparison or proof of firmware causality. Resolve that renderer/host dependency and check indirect effects of earlier allocation placement before changing the pool policy or merging.

ROM execution and its limits

The supplied reference traces show startup ROM execution and no post-title ROM execution in the covered workloads. That does not prove no rare reset/error/indirect ROM path exists. There is no established evidence that routine runtime ROM calls explain its smaller image.

An uncompressed readonly executable region can potentially execute from mapped ROM. Compressed firmware bytes cannot. Any writable globals, stacks and mutable tables still require RAM. Export addresses, callback copies, branch/relocation targets and reset mappings must remain valid; all users of a moved table must retain valid addresses. The existing ROM-placement prototype #118 reported a large footprint reduction but failed PGR2 and remains unqualified.

Practical order

1. Retain the merged bounded corrections and maintain the disjoint ledger.
2. Resolve pool burst reuse versus idle recovery; separately examine finer capacities for demonstrated live-object waste.
3. Complete one callback family's final-consumer proof before moving its code/data to INIT.
4. Audit the remaining source-proved live metadata and readonly placement opportunities.
5. Specialize fixed-platform runtime service boundaries where they retain generic code; keep actual runtime I/O behavior.
6. Revisit ROM placement as a separate compatibility-qualified experiment.

Each change gets its own PR and matched outputs, actual page recovery and complete cost measurements. No paper byte sum substitutes for recovered physical pages, and memory recovery alone does not prove a game speed gain.

Deeper research extension

The new Memory-Runtime-Physical-Owners accounts all 332 unavailable pages at the current main's completed frozen allocation-test checkpoint, including 31 wholly empty bucket pages / 124 KiB. Public pool/cache totals overlap raw physical ownership as documented there. The Memory-Resident-Page-Atlas gives all 102 image pages, current physical mappings, and 59 exact global/table sizes plus eight structure-body proofs.

The Memory-ROM-and-Cold-Code audit identifies the actual runtime prefetch patch, reset shadow staging and export/callback address contracts. It also finds 192 extra interrupt stubs / 1,344 bytes beyond the emitted 64-entry IDT. Their arithmetic consumers need a vector-domain proof before deletion; crossing a linker page boundary could recover 4 KiB, but no recovery is claimed yet.

The new Memory-Reference-Content-and-ROM resolves 13 more bytes of exact export metadata, leaving 56,153 retained bytes semantically unknown. Pattern/reference evidence is listed for all 246 prior unknown ranges without pretending to have the original function/structure boundaries. Its two captured ROM maps differ; neither endpoint's known lower aliases appears in the bounded execution traces. Per-event fetch identity and transient mapping history remain missing.

Function code extents, per-call stack exposure and dynamic objects are different memory quantities. The current ledger measures emitted code and object bodies; it does not claim compiler stack-frame bounds or worst nested/interrupt stack depth for every function. Dynamic pool capacity is not automatically the exact requested payload. These remaining measurements should be targeted where a proposed reduction depends on them.

Further object and vector ownership

Memory-Allocation-Object-Ledger lists all 429 live small objects, grouped into 54 tags, and every whole-page pool run. Actual cache pointers and compiler-proved offsets resolve 25 of the 28 whole-page pages: 64 KiB ordinary buffers, 32 KiB private reserves and a 4 KiB shared zero page. Three whole-page allocations remain unidentified. These are components of the existing pool total, not new charges.

Memory-Additional-Table-Lifetimes adds eleven exact global sizes and four structure bodies, raising proved resident content to 367,003 bytes and leaving 36,253 unresolved. Memory-Interrupt-Vector-Lifetimes finds an inherited admission mismatch: unsupported PIC IRQs can translate beyond the 64-entry IDT, with unchecked connect/query paths. Correct software admission is a prerequisite for removing the 1,344-byte extra-stub range or specializing the 256-entry usage arrays. No current game failure is attributed to this finding.