Skip to content

DXMT: avoid initial data for depth/stencil textures#764

Open
moreaki wants to merge 1 commit into
VirtualBox:mainfrom
moreaki:agent/dxmt-depth-stencil-initial-data-crash
Open

DXMT: avoid initial data for depth/stencil textures#764
moreaki wants to merge 1 commit into
VirtualBox:mainfrom
moreaki:agent/dxmt-depth-stencil-initial-data-crash

Conversation

@moreaki

@moreaki moreaki commented Jul 8, 2026

Copy link
Copy Markdown

Summary

Avoid passing initial host-side backing data when creating DX11 depth/stencil textures in the VMSVGA backend.

Depth/stencil resources are normally initialized by later rendering commands such as clears. On the macOS/Arm DXMT path, passing the optional pSurfaceData buffer as initial texture data can crash in Metal's replaceRegion validation when that backing memory is stale or otherwise unsuitable for the newly created depth/stencil texture.

Root cause

vmsvga3dBackSurfaceCreateResource() currently passes paInitialData whenever pSurfaceData is non-null, with exceptions for multisample and decoder resources. The pSurfaceData field is optional and documented as potentially outdated after hardware realization. For depth/stencil textures, that upload is unnecessary in the reproduced path and can hand invalid source bytes to DXMT's Metal texture initialization.

Validation

Tested locally on macOS/Arm with a Windows 11 Arm guest, VMSVGA WDDM, 3D acceleration enabled, and 256 MB VRAM.

Failing run before the change:

  • Microsoft Edge WebGL2 workload reached webgl-context, then VirtualBoxVM aborted.
  • Crash report faulted in the VMSVGA CMD thread.
  • Stack included AGXMetalG14X agxaAssertBufferIsValid, -[AGXG14XFamilyTexture replaceRegion:...], dxmt::InitializeTextureData<D3D11_TEXTURE2D_DESC1>, dxCreateTexture, vmsvga3dBackSurfaceCreateResource, and dxCreateDepthStencilView.

Passing runs after the change:

  • 8 second WebGL2 smoke run completed at 59.9 FPS average.
  • 30 second WebGL2 / ANGLE D3D11 run with 512 instances completed at 58.9 FPS average.
  • The VM remained running after both runs.

Fixes #763.

Signed-off-by: Roberto Nibali <rnibali@gmail.com>
@oracle-contributor-agreement

Copy link
Copy Markdown

Thank you for your pull request and welcome to our community! To contribute, please sign the Oracle Contributor Agreement (OCA).
The following contributors of this PR have not signed the OCA:

To sign the OCA, please create an Oracle account and sign the OCA in Oracle's Contributor Agreement Application.

When signing the OCA, please provide your GitHub username. After signing the OCA and getting an OCA approval from Oracle, this PR will be automatically updated.

If you are an Oracle employee, please make sure that you are a member of the main Oracle GitHub organization, and your membership in this organization is public.

@oracle-contributor-agreement oracle-contributor-agreement Bot added the OCA Required At least one contributor does not have an approved Oracle Contributor Agreement. label Jul 8, 2026
@moreaki moreaki changed the title VMSVGA: avoid initial data for DX depth/stencil textures DXMT/VMSVGA: avoid initial data for depth/stencil textures Jul 9, 2026
@moreaki moreaki changed the title DXMT/VMSVGA: avoid initial data for depth/stencil textures DXMT: avoid initial data for depth/stencil textures Jul 9, 2026
@moreaki

moreaki commented Jul 9, 2026

Copy link
Copy Markdown
Author

DXMT v0.80 audit update:

Updating the vendored DXMT tree to v0.80 would not address this directly, because this fix is in VirtualBox's VMSVGA/D3D11 caller path. The issue is that VirtualBox can pass optional host-side backing data as initial data while creating depth/stencil resources; suppressing that for D3D11_BIND_DEPTH_STENCIL belongs at this call site.

I did not open a DXMT upstream issue for this one, since it is not a DXMT source-tree fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

OCA Required At least one contributor does not have an approved Oracle Contributor Agreement.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

macOS/Arm DXMT crash when creating depth/stencil texture with initial data

1 participant