vCPU Windows XMM Register Budget

Windows XMM Register Budget

Lifecycle: Performance - Unproven · Type: Backend optimization
State: Source-backed; runtime value is not established.

Status: Source-validated backend optimization candidate; not implemented or benchmarked.

Mechanism class: Register-allocation and JIT-boundary policy — not a shortcut.

Reviewed source: 322986f4c5502b3fa887a9605ac8cd88e9b2c579

Primary file: tcg/i386/tcg-target.c.inc.

Why this is not a shortcut

The proposal does not bypass a maintained general emulation route under a narrow assumption. It expands the host register resources available to the normal TCG allocator by paying save/restore cost at the C-to-JIT boundary.

It belongs in the Performance Optimization Catalog, not the shortcut-only list.

Current Win64 constraint

The x86-64 allocation order exposes XMM0 through XMM5 on Windows. XMM5 is reserved as TCG_TMP_VEC. Initialization explicitly reserves XMM6 through XMM15 because they are ABI-preserved registers and the current JIT entry/exit path does not save them.

The practical normal allocation budget is therefore:

XMM0–XMM4: normal vector/FP allocation
XMM5: TCG temporary
XMM6–XMM15: reserved on Win64

The source contains an early comment calling XMM6–XMM15 caller-saved, but the later reservation comment describes the actual condition: they are call-saved/nonvolatile and xemu does not preserve them.

Proposed backend experiment

Save a deliberately small selected set at the C-to-JIT boundary, expose those registers to the allocator, and restore them on every exit:

flowchart TD
    A[C code enters translated execution] --> B[Allocate enlarged JIT frame]
    B --> C[Save selected nonvolatile XMM registers]
    C --> D[Run chained translated blocks with larger register budget]
    D --> E{Exit to C/main loop/exception?}
    E --> F[Restore saved registers]
    F --> G[Release frame and return]

Loading

Illustrative boundary:

sub    rsp, enlarged_frame
movdqu [rsp + save_xmm6], xmm6
movdqu [rsp + save_xmm7], xmm7
; execute translated blocks
movdqu xmm6, [rsp + save_xmm6]
movdqu xmm7, [rsp + save_xmm7]
add    rsp, enlarged_frame
ret

Start with XMM6/XMM7 or another small subset. Do not expose XMM6–XMM15 all at once before measuring marginal spill reduction.

Why it may help

Potential beneficiaries include:

• the hard-x87 path retaining multiple stack values;
• direct SSE arithmetic experiments;
• vectorized TCG operations;
• blocks with several simultaneously live scalar/vector temporaries;
• helper-adjacent code where nonvolatile low-128-bit values can survive C calls.

The trade is:

save/restore bytes and instructions per C-to-JIT entry/exit
versus
dynamic spill/reload traffic avoided inside translated execution

Long chained runs may benefit. Workloads that repeatedly leave translated execution may lose.

Important ABI and allocator boundaries

• XMM6–XMM15 low 128 bits are nonvolatile under Win64 and must be preserved for the C caller.
• Upper YMM/ZMM state has different volatility/ABI behavior; do not automatically allocate 256-bit values to newly exposed registers.
• Every normal and exceptional JIT exit must restore the saved state.
• Helper calls, signal/exception unwinding, long exits, and generated epilogues must agree on frame layout.
• Debug/unwind metadata must continue to describe the prologue/epilogue correctly where required.
• TCG_TMP_VEC remains reserved unless a separate design changes it.
• Global call-clobber declarations must match what helpers and the platform ABI actually preserve.

Code and ownership map

Area	Role
tcg_target_reg_alloc_order[]	Registers considered by normal allocation
TCG_TMP_VEC	Current XMM5 reservation
tcg_target_init() reserved-register setup	Reserves XMM6–XMM15 on Win64
tcg_target_qemu_prologue() and return paths	C-to-JIT save/restore seam
TCG register allocator	Spill/reload decisions
JIT debug/unwind frame	Correct stack/register description

Evidence required before changing the prologue

Collect dynamic evidence by value type and block:

Counter	Meaning
vector/FP temporaries simultaneously live	Actual pressure
spills and reloads by register class	Avoidable work
blocks with pressure above five XMM registers	Eligibility
JIT entries/exits	Boundary cost frequency
average/chained TBs per entry	Amortization opportunity
helper calls while values are live	Possible nonvolatile benefit
additional prologue/epilogue bytes and cycles	Candidate cost

A source-level shortage of registers is not enough. If spills are rare, saving more registers on every entry loses.

Test matrix

1. A Win64 C caller loads recognizable sentinel values into every candidate nonvolatile XMM register.
2. Enter translated execution and exercise normal direct-chain exits.
3. Exercise helper calls, interrupts, exceptions, single stepping, and early exits.
4. Return to the caller and verify every sentinel exactly.
5. Force high simultaneous FP/vector liveness and compare generated spills/reloads.
6. Test low-pressure workloads to expose pure boundary overhead.
7. Test 32-, 64-, and 128-bit values separately; reject unsupported wider values initially.
8. Verify stack alignment, shadow space, unwind/debug data, and sanitizer behavior.
9. Run Release ABBA/BAAB on CPU-heavy and retail workloads.

Expected useful signature

measured XMM spills/reloads are substantial
selected extra registers are frequently allocated
JIT entries are amortized across enough translated work
spill/reload reduction exceeds save/restore overhead
caller sentinel and all exit tests pass

Failure modes

• Treating the nonvolatile registers as free without saving them.
• Saving all ten registers before measuring marginal value.
• Ignoring upper YMM/ZMM ABI rules.
• Missing an exceptional or uncommon exit.
• Breaking unwind/debug frame information.
• Globally changing call-clobber behavior for values helpers do not preserve.
• Measuring a large generated block rather than dynamic spill frequency.
• Combining this immediately with native SSE retention, making attribution impossible.

Qualification plan

1. Add allocator spill/reload and liveness evidence.
2. Select a two-register experiment if pressure justifies it.
3. Implement and verify the complete JIT entry/exit save/restore contract.
4. Run sentinel tests for all exit classes.
5. Compare generated code and fixed-work CPU cost.
6. Expand the register set only if each marginal addition remains beneficial.
7. Keep this result separate from shortcut acceptance decisions.

Related pages

• vCPU Shortcut Candidates
• Native SSE Arithmetic
• Helper-Aware x87 Cache Retention
• Performance Optimization Catalog
• CPU & vCPU Execution

Full technical reference

Primary reference: This subject page is the design and research reference for Windows XMM Register Budget. Pull requests are implementation/test history only.
Lifecycle: Performance - Unproven · Mechanism: Win64 JIT register-allocation optimization
Reviewed source: 322986f4c5502b3fa887a9605ac8cd88e9b2c579 · Acceptance class: intended exact or behavior-preserving for its admitted scope

1. Scope

Save selected nonvolatile XMM registers at the C-to-JIT boundary so TCG can allocate more than XMM0-XMM4 during translated execution and potentially reduce spills.

This entry remains Unproven: the source path and opportunity are concrete, but no accepted xemu runtime result has established that the candidate improves representative workloads. The page defines what must be bypassed, what remains mandatory, and how the hypothesis can be rejected.

2. Source seams

File / function	Role
tcg/i386/tcg-target.c.inc	Win64 vector register allocation order/reserved set and JIT prologue/epilogue

3. Current execution path

flowchart TD
    N0["Enter JIT without saving XMM6-XMM15"]
    N1["Reserve nonvolatile XMMs"]
    N0 --> N1
    N2["Allocate from XMM0-XMM4"]
    N1 --> N2
    N3["Spill/reload under pressure"]
    N2 --> N3
    N4["Exit JIT"]
    N3 --> N4

Loading

4. Candidate execution path

flowchart TD
    N0["Save selected XMM6+ once at JIT entry"]
    N1["Expose them to allocator"]
    N0 --> N1
    N2["Run many TBs with larger register pool"]
    N1 --> N2
    N3["Restore saved registers at all exits"]
    N2 --> N3

Loading

5. Why it may help

Windows x64 treats XMM6-XMM15 as nonvolatile. The current JIT avoids them because it does not preserve them, leaving five normal XMM registers after the temporary. Paying boundary saves may reduce dynamic spills when enough translated work occurs before exit. This is a backend optimization, not an assumption shortcut.

6. Work removed

• Dynamic spills/reloads caused by a small vector/FP register pool

7. Work that remains mandatory

• ABI preservation
• Boundary save/restore traffic
• Call-clobber modeling
• Large-vector/YMM rules
• Reference register set

8. Implementation sketch

; Conceptual Win64 JIT entry
sub    rsp, frame_size
movdqu [rsp + save_xmm6], xmm6
movdqu [rsp + save_xmm7], xmm7
; translated execution may allocate xmm6/xmm7
; every normal/exceptional exit restores them
movdqu xmm7, [rsp + save_xmm7]
movdqu xmm6, [rsp + save_xmm6]
add    rsp, frame_size
ret

The sketch defines the intended boundary; it is not a complete patch and does not replace source review, ownership design, or focused tests.

9. Correctness contract

• All exits preserve Win64 nonvolatile registers.
• Allocator/call-clobber masks match the new availability.
• Save cost occurs at C/JIT boundary, not every TB edge.
• 256-bit upper-lane volatility and value widths are handled explicitly.

10. State, ownership, invalidation, and fallback

Saved register slots belong to each JIT entry frame/thread. Start with two added registers to bound cost. Exceptional exits, signals, helper returns and longjmp-like paths require the same restore contract.

The maintained reference path remains the fallback whenever admission, capability, lifetime, or validation is uncertain. A fast result with an unproven dependency is not acceptable as an exact optimization.

11. Advantages

• May reduce spills in FP/vector-heavy TBs
• Can support later native SSE/x87 retention
• Clear ABI sentinel test
• Configurable small experiment

12. Costs and failure modes

• Every JIT entry/exit pays save/restore
• Short runs can regress
• Backend and unwind complexity
• Spill pressure may not be significant
• Upper YMM semantics

13. Evidence counters

Counters must be cheap, subsystem-owned, and snapshotted outside the hot loop. Record at least:

• JIT_entries/exits
• XMM_spills/reloads
• live_vector_values
• added_save_restore_bytes
• translated_cycles_per_entry
• host code size
• vCPU samples

A local reduction is not enough. Evidence must connect route reachability and removed work to fixed guest progress, CPU/GPU time where applicable, and p95/p99 behavior.

14. Qualification matrix

• ABI sentinel values in XMM6+ across all exits
• Different numbers of added registers
• Short vs long translated runs
• FP/vector pressure fixtures
• Helpers/exceptions/signals
• Spill/reload generated-code inspection
• Release games

Correctness runs may use validation layers, detailed comparisons, and fault injection. Performance acceptance must use optimized Release builds with matching symbols and minimal instrumentation.

15. Acceptance rule

Move to Researched when spill counts and boundary-cost experiments establish where the trade wins; implement only with complete ABI/exit coverage.

16. Rejection rule

Reject a global expansion without preservation, or when entry/exit frequency makes save cost exceed dynamic spill savings.

17. Related architecture and history

CPU-and-vCPU-Execution · Performance-Unproven · Hot-Path-vCPU-Native-SSE-Arithmetic · Hot-Path-vCPU-Helper-Aware-x87-Cache-Retention

PR pages may be linked as supporting evidence, but they are not substitutes for this reference.