Skip to content

DXMT: backport dynamic CPU encoding heap#766

Open
moreaki wants to merge 1 commit into
VirtualBox:mainfrom
moreaki:agent/dxmt-cpu-argument-heap-blocks
Open

DXMT: backport dynamic CPU encoding heap#766
moreaki wants to merge 1 commit into
VirtualBox:mainfrom
moreaki:agent/dxmt-cpu-argument-heap-blocks

Conversation

@moreaki

@moreaki moreaki commented Jul 8, 2026

Copy link
Copy Markdown

Summary

Backport upstream DXMT commit 998714300fe1144114fd689ce1f7c36be6c58189, feat(dxmt): dynamically grow & shrink cpu encoding buffer.

VirtualBox vendors DXMT 0.60 in src/libs/dxmt-0.60. In that version, ArgumentEncodingContext::allocate_cpu_heap() logs cpu argument heap overflow, expect error. when a command chunk exceeds the fixed CPU encoding heap, but then still returns a pointer past the backing allocation.

The upstream DXMT fix replaces the single fixed heap with dynamically managed CPU encoding heap chunks and retires underused chunks after repeated non-use. The fix is included upstream starting with DXMT v0.71; current DXMT is newer than the vendored 0.60 snapshot.

Why

A local Windows 11 Arm guest stress test on a macOS/Arm host hit repeated DXMT CPU encoding heap overflows, followed by Metal command-buffer out-of-memory errors and a broken guest display update path. Since DXMT already fixed the allocator upstream, this PR intentionally backports the upstream change rather than introducing a VirtualBox-specific rewrite.

A full DXMT v0.60 -> v0.80 refresh is much larger than this specific crash fix, so this keeps the change narrow and attributable.

Validation

  • Confirmed the upstream DXMT commit applies cleanly onto VirtualBox's vendored src/libs/dxmt-0.60 paths.
  • Rebuilt the integrated macOS/Arm target with:
    kmk VBoxDxMt VBOX_LLVM_PATH=/opt/homebrew/opt/llvm@15 VBOX_LLVM_CONFIG=/opt/homebrew/opt/llvm@15/bin/llvm-config ...
  • The build completed successfully and installed VBoxDxMt.dylib.

Closes #765.

Backport upstream DXMT commit 998714300fe1144114fd689ce1f7c36be6c58189.

The vendored DXMT 0.60 encoder used a single fixed CPU encoding heap and only logged when the heap overflowed, then returned memory past the allocation. Upstream DXMT fixed this by switching to dynamically managed encoding heap chunks and retiring underused chunks after repeated non-use.

Origin: 3Shain/dxmt@9987143
Signed-off-by: Ronnie Nibali <rnibali@gmail.com>
@oracle-contributor-agreement

Copy link
Copy Markdown

Thank you for your pull request and welcome to our community! To contribute, please sign the Oracle Contributor Agreement (OCA).
The following contributors of this PR have not signed the OCA:

To sign the OCA, please create an Oracle account and sign the OCA in Oracle's Contributor Agreement Application.

When signing the OCA, please provide your GitHub username. After signing the OCA and getting an OCA approval from Oracle, this PR will be automatically updated.

If you are an Oracle employee, please make sure that you are a member of the main Oracle GitHub organization, and your membership in this organization is public.

@oracle-contributor-agreement oracle-contributor-agreement Bot added the OCA Required At least one contributor does not have an approved Oracle Contributor Agreement. label Jul 8, 2026
@moreaki

moreaki commented Jul 9, 2026

Copy link
Copy Markdown
Author

DXMT v0.80 audit update:

This is already addressed by newer DXMT. The dynamic CPU encoding heap work landed upstream before v0.80, so updating the vendored DXMT tree to v0.80 would include the equivalent fix.

So this PR is best understood as a narrow backport for VirtualBox's currently vendored DXMT snapshot.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

OCA Required At least one contributor does not have an approved Oracle Contributor Agreement.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Vendored DXMT 0.60 CPU encoding heap can overflow under large command chunks

2 participants