xorg.xf86videovmware: fix 3d support with mesa

[tricktron]

Fixes #390737.

Context

The xf86videovmware driver needs the gallium-xa driver for 3d support. Otherwise the X server crashes if you have 3d acceleration enabled in vmware workstation/player:

17.628] (II) vmware(0): Driver was compiled without KMS- and 3D support.
[    17.628] (WW) vmware(0): Disabling 3D support.
[    17.628] (WW) vmware(0): Disabling Render Acceleration.
[    17.628] (WW) vmware(0): Disabling RandR12+ support.
...
  19.086] (EE) Backtrace:
[    19.086] (EE) 0: /nix/store/8lfqzlmg0cddfz9j22050mcmw34is7jc-xorg-server-21.1.16/bin/X (OsSigHandler+0x33) [0x5cc453]
[    19.086] (EE) unw_get_proc_name failed: no unwind info found [-10]
[    19.086] (EE) 1: /nix/store/cmpyglinc9xl9pr4ymx8akl286ygl64x-glibc-2.40-66/lib/libc.so.6 (?+0x0) [0x7f7ae43a4f30]
[    19.086] (EE) 2: /nix/store/cmpyglinc9xl9pr4ymx8akl286ygl64x-glibc-2.40-66/lib/libc.so.6 (__memset_avx2_unaligned_erms+0x49) [0x7f7ae44d9b89]
[    19.086] (EE) 3: /nix/store/1l8f8g1q4nczssafpsmzh0mjv75y4ic5-xf86-video-vmware-13.4.0/lib/xorg/modules/drivers/vmware_drv.so (VMWAREScreenInit+0x15c) [0x7f7ae3d7169c]
[    19.086] (EE) 4: /nix/store/8lfqzlmg0cddfz9j22050mcmw34is7jc-xorg-server-21.1.16/bin/X (AddScreen+0xd7) [0x4473c7]
[    19.086] (EE) 5: /nix/store/8lfqzlmg0cddfz9j22050mcmw34is7jc-xorg-server-21.1.16/bin/X (InitOutput+0x27b) [0x48d15b]
[    19.086] (EE) 6: /nix/store/8lfqzlmg0cddfz9j22050mcmw34is7jc-xorg-server-21.1.16/bin/X (dix_main+0x190) [0x44b390]
[    19.086] (EE) 7: /nix/store/cmpyglinc9xl9pr4ymx8akl286ygl64x-glibc-2.40-66/lib/libc.so.6 (__libc_start_call_main+0x7e) [0x7f7ae438e1fe]
[    19.086] (EE) 8: /nix/store/cmpyglinc9xl9pr4ymx8akl286ygl64x-glibc-2.40-66/lib/libc.so.6 (__libc_start_main+0x89) [0x7f7ae438e2b9]
[    19.086] (EE) 9: /nix/store/8lfqzlmg0cddfz9j22050mcmw34is7jc-xorg-server-21.1.16/bin/X (_start+0x25) [0x433245]
[    19.086] (EE) 
[    19.086] (EE) Segmentation fault at address 0x0
[    19.086] (EE) 
Fatal server error:
[    19.086] (EE) Caught signal 11 (Segmentation fault). Server aborting

@K900 Hope one more mesa rebuild is ok despite your hard work to reduce them 😃.

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
  • sandbox = relaxed
  • sandbox = true
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 25.05 Release Notes (or backporting 24.11 and 25.05 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

---

Add a 👍 reaction to pull requests you find important.

[K900]

No, this is very bad and we should not be shipping this. We should probably drop the DDX and just use modesetting.

[tricktron]

No, this is very bad and we should not be shipping this. We should probably drop the DDX and just use modesetting.

@K900 Well for me there is currently no obvious best solution. While modesetting is as fast as the xf86videovmware driver it has two main disadvantages for me:

• no autoresize support: e.g. if you resize the vmware window then the xf86videovmware driver does automatically resize the guest resolution while the modesetting does not.
• noticable input lag from time to time: e.g. you type something, then it freezes, you of course type more stuff in between and then it applies everything creating a big mess.

Switching to Wayland is neither an option because it does not support copy-paste. So I would like to continue to support the xfx86videovmware for some time.

What is the current disadvantage of supporting it at the moment?

[K900]

XA is an old, basically unmaintained, vmware-only code path. This DDX is the only driver that even uses it. Also, as far as I can tell, open-vm-tools should propagate resolution settings to the kernel: https://github.com/vmware/open-vm-tools/blob/739c5a2f4bfd4cdda491e6a6f6869d88c0bd6972/open-vm-tools/services/plugins/resolutionKMS/resolutionKMS.c and

[tricktron]

@K900 XA may be old but the xf86videovmware DDX works really well with the virtual SVGA 3D graphics cards (Do one thing but do it right). Discontinuing it only because it is old does not convince me.

However, I think that we can agree that the current state of having a xf86videovmware driver without 3D support is a bad option.

I exposed the gallium-xa option in a new top-level parameter with the newest commits so that it can stay disabled as default for mesa. Can you live with that?

We can still discuss and decide to completely discontinue the vmware driver for 25.05 (with a warning that you should switch to the modesetting driver) in another pull request.

[K900]

Adding the option makes it worse, actually, because now there's multiple configurations that need to be supported and tested. I am going to try and get a VMWare install and see why auto-resize is not working with modesetting.

[tricktron]

Adding the option makes it worse, actually, because now there's multiple configurations that need to be supported and tested.

Alright. I dropped the commits again.

I am going to try and get a VMWare install and see why auto-resize is not working with modesetting.

@K900
As far as I can see it works like the following:

Screen resize in host -> (RPC) -> vmtoolsd -> (syscall) kernel creates udev screen resize event -> xserver with xfx86-video-vmmware driver reads event -> (kernel mode setting) resizes guest window.

Now the question is why it does not work with the modesetting driver?

[tricktron]

@K900 Did you make some progress?

I actually found another missing feature of the modesetting driver😄: it does not support to use multiple monitors (e.g. the cycle monitor feature or xinerama as the xf86videovmware driver calls it).

So that brings me back to the original question: What is the downside of continuing supporting the DDX for now?
As far as I can tell, it means one more enabled boolean flag for mesa and one more mesa build through the buildInput for the xf86videovmware package.

Can you live with that? Or what do you need so that you could live with supporting this DDX? Do you want to bring more people in to discuss this?

[K900]

No, I was busy with other real life things. Do we know the closure size impact of readding XA?

[tricktron]

No, I was busy with other real life things. Do we know the closure size impact of readding XA?

@K900 Yes, I used the following command: nix path-info -Sh $(nix-build --no-out-link -A mesa):
WIth XA: 859.2 MiB
Without XA: 845.9 MiB

That means a closure size increase of only 1.57% which is in my opinion ok.