Kitty and alacritty cannot launch in vmware workstation pro, 3d accel is enabled

[IceAsteroid]

VMware workstation pro, version 17.5.0.

3d acceleration is enabled for the VM.

The host is windows 10 IoT Enterprise LTSC, version 21H2.

The VM's OS is ArchLinux, kernel 6.18.4-arch1-1.

The graphics information:

VGA compatible controller: VMware SVGA II Adapter (prog-if 00 [VGA controller])
      Subsystem: VMware SVGA II Adapter
      Flags: bus master, medium devsel, latency 64, IRQ 16
      I/O ports at 2040 [size=16]
      Memory at f0000000 (32-bit, prefetchable) [size=128M]
      Memory at fb800000 (32-bit, non-prefetchable) [size=8M]
      Expansion ROM at 000c0000 [disabled] [size=128K]
      Capabilities: [40] Vendor Specific Information: Len=00 <?>
      Capabilities: [44] PCI Advanced Features
      Kernel driver in use: vmwgfx
      Kernel modules: vmwgfx

They launch fine in gnome and niri, but cannot launch in hyprland

kitty's output:

$: kitty -o "linux_display_server=wayland"
  [0.111] [glfw error 65544]: Wayland: fatal display error: Broken pipe
  wl_display#1: error 1: invalid arguments for wl_surface#35.attach

alacritty's output

$: alacritty
  wl_display#1: error 1: invalid arguments for wl_surface#39.attach
  Io error: Invalid argument (os error 22)
  Io error: Invalid argument (os error 22)
  Io error: Invalid argument (os error 22)
  Error: ExitFailure(1)

[kvakulo]

I stumbled upon this patch. It allows hyprland to work together with the vmwgfx driver in VMWare Fusion.

[IceAsteroid]

I'm not currently using Hyprland. However, thank you a lot to let us know!

[Pascal-0x90]

Hihi, I recently ran into this issue too and it dove me down a little rabbit hole of figuring out why the bug even occurs. I, like OP, had an issue with Kitty not wanting to open. The patch by Andrew Bastin offers a way to somewhat ignore why the issue is occurring but I don't think it fully tackles why this is happening.

In Short, Why is it Crashing

I can go into more detail but on VMWare guests using vmwgfx driver in the Linux Kernel, Hyprland and Wayland will kill GPU accelerated clients the moment the compositor attaches a dmabuf since it cant close the dmabuf correctly. This is due to vmwgfx overriding the normally called drm_prime_fd_to_handle_ioctl function with vmw_prime_fd_to_handle which attempts to handle both GEM and TTM surface handles but will just return a TTM object if the reference fed into it happened to be a TTM backed handle by accident. It feels like it's well intended but if an assumption of how this works is made it can cause someone using drmCloseBufferHandle to be unable to properly close the handle since it's not a GEM object.

So the flow ends up being if a Mesa/svga client buffer has an associated guest-backed surface, vmw_prime_handle_to_fd will export it as a non-dumb buffer through its surface handle with ttm_prime_handle_to_fd, producing a TTM prime object rather than a GEM export. So when importing using vmw_prime_fd_to_handle, it will call ttm_prime_fd_to_handle first and sill only call drm_gem_prime_fd_to_handle if that fails. Since its not a dumb buffer, it will return the TTM base-object (a TTM surface) handle. Since this will not exist in the GEM table tracking all the GEM objects, drmCloseBufferHandle will fail to close the object. It is however, tracked in the TTM table but many things try to not use that.

This inevitably needs to be a kernel fix, but we can still perform a proper deallocation in userspace through libdrm and sending the command DRM_VMW_UNREF_SURFACE which will properly handle dropping these handles.

A Fix

Here is my patch atleast for the Hyprland code. I also included the one for Wlroots but that probably won't be something fixed here, just more for completion sake.

Hyprland

--- a/src/protocols/LinuxDMABUF.cpp
+++ b/src/protocols/LinuxDMABUF.cpp
@@ -4,6 +4,7 @@
 #include <tuple>
 #include "../helpers/MiscFunctions.hpp"
 #include <sys/mman.h>
+#include <cstring>
 #include <xf86drm.h>
 #include <fcntl.h>
 #include <sys/stat.h>
@@ -261,6 +262,30 @@
     m_createdBuffer = buf;
 }
 
+#define DRM_VMW_UNREF_SURFACE 10
+
+struct drmVmwSurfaceArg {
+    int32_t  sid;
+    uint32_t handleType;
+};
+
+// vmwgfx prime-imports surface-backed dmabufs as TTM handles, not GEM, so
+// GEM_CLOSE fails with EINVAL; close them via DRM_VMW_UNREF_SURFACE instead.
+static int closeVmwGFXHandle(int fd, uint32_t handle) {
+    bool          isVmwgfx;
+    drmVersionPtr version;
+
+    version  = drmGetVersion(fd);
+    isVmwgfx = version && version->name && strncmp(version->name, "vmwgfx", version->name_len) == 0;
+    if (version)
+        drmFreeVersion(version);
+    if (!isVmwgfx)
+        return -1;
+
+    struct drmVmwSurfaceArg arg = {(int32_t)handle, 0};
+    return drmCommandWrite(fd, DRM_VMW_UNREF_SURFACE, &arg, sizeof(arg));
+}
+
 bool CLinuxDMABUFParamsResource::commence() {
     if (!PROTO::linuxDma->m_mainDeviceFD.isValid())
         return true;
@@ -273,7 +298,7 @@
             return false;
         }
 
-        if (drmCloseBufferHandle(PROTO::linuxDma->m_mainDeviceFD.get(), handle)) {
+        if (drmCloseBufferHandle(PROTO::linuxDma->m_mainDeviceFD.get(), handle) != 0 && closeVmwGFXHandle(PROTO::linuxDma->m_mainDeviceFD.get(), handle) != 0) {
             LOGM(Log::ERR, "Failed to close dmabuf handle");
             return false;
         }

And for wlroots

--- a/types/wlr_linux_dmabuf_v1.c
+++ b/types/wlr_linux_dmabuf_v1.c
@@ -205,6 +205,30 @@
 	wlr_buffer_drop(&buffer->base);
 }
 
+#define DRM_VMW_UNREF_SURFACE 10
+
+struct drmVmwSurfaceArg {
+	int32_t sid;
+	uint32_t handleType;
+};
+
+/* vmwgfx prime-imports surface-backed dmabufs as TTM handles, not GEM, so
+ * GEM_CLOSE fails with EINVAL; close them via DRM_VMW_UNREF_SURFACE instead. */
+static int closeVmwGFXHandle(int fd, uint32_t handle) {
+	bool isVmwgfx;
+	drmVersionPtr version;
+
+	version = drmGetVersion(fd);
+	isVmwgfx = version && version->name && strncmp(version->name, "vmwgfx", version->name_len) == 0;
+	if (version)
+		drmFreeVersion(version);
+	if (!isVmwgfx)
+		return -1;
+
+	struct drmVmwSurfaceArg arg = {(int32_t)handle, 0};
+	return drmCommandWrite(fd, DRM_VMW_UNREF_SURFACE, &arg, sizeof(arg));
+}
+
 static bool check_import_dmabuf(struct wlr_dmabuf_attributes *attribs, void *data) {
 	struct wlr_linux_dmabuf_v1 *linux_dmabuf = data;
 
@@ -219,7 +243,8 @@
 			wlr_log_errno(WLR_ERROR, "Failed to import DMA-BUF FD for plane %d", i);
 			return false;
 		}
-		if (drmCloseBufferHandle(linux_dmabuf->main_device_fd, handle) != 0) {
+		if (drmCloseBufferHandle(linux_dmabuf->main_device_fd, handle) != 0 &&
+				closeVmwGFXHandle(linux_dmabuf->main_device_fd, handle) != 0) {
 			wlr_log_errno(WLR_ERROR, "Failed to close buffer handle for plane %d", i);
 			return false;
 		}

Success and NixOS Patch

This successfully let me patch my NixOS setup and now I can use multiple GPU accelerated programs just fine in Hyprland and Wayland in VMWare. For anyone else using NixOS here is how you can use these patches in NixOS. This is much like that one guy's patch more or less:

{ config, lib, ... }:
let
    cfg = config.patches.vmwgfx-dmabuf;
in {
    options.patches.vmwgfx-dmabuf.enable = lib.mkEnableOption
        "vmwgfx dmabuf import fix for Hyprland/wlroots (VMware guests)";
        
    config = lib.mkIf cfg.enable {
        nixpkgs.overlays = [
            (final: prev: {
                hyprland = prev.hyprland.overrideAttrs (old: {
                    patches = (old.patches or [ ]) ++ [ ./hyprland-vmwgfx-dmabuf.patch ];
                });
                wlroots_0_20 = prev.wlroots_0_20.overrideAttrs (old: {
                    patches = (old.patches or [ ]) ++ [ ./wlroots-vmwgfx-dmabuf.patch ];
                });
            })
        ];
    };
}

[wabarTM]

go away ai

[Pascal-0x90]

its not ai ;-;;

[Pascal-0x90]

I wanted to share a fix I made and why it's breaking. Srry it looks so much like AI

[wabarTM]

impossible, reply too good to be true...

from time to time there is ai spam in old issues, so you got under the same comb

[Pascal-0x90]

Yeaaa that makes sense. I also just kinda mental dumped a bit into the discussion somewhat so it probably felt wayyy too verbose. I was going to post a new discussion thread but found this one existed for the exact issue I was having trying to use Kitty on NixOS with Hyprland.

[carlpe]

I tested the vmwgfx TTM-handle close patch posted above against Hyprland 0.56.2 (efb50993780079460b0cbed1363e2166a2de1d9f) in an Arch/Omarchy guest running under VMware Workstation.

The patch adds a vmwgfx-specific fallback in src/protocols/LinuxDMABUF.cpp: if drmCloseBufferHandle() fails on a TTM surface handle, it attempts to release the handle using DRM_VMW_UNREF_SURFACE. The DMA-BUF is rejected only if both close paths fail.

Only src/protocols/LinuxDMABUF.cpp was changed. Aquamarine, Mesa, the kernel, vmwgfx, open-vm-tools, and Omarchy were left unchanged.

Before the patch, GPU-rendered Wayland clients failed with DMA-BUF / wl_surface.attach errors and a black or unusable session, so the VM required LIBGL_ALWAYS_SOFTWARE=1.

The original 120 FPS Omarchy screensaver provided a convenient repeatable workload and produced approximately:

• 31,545 repeated errors/min
• 5.76 MiB/min of Hyprland log growth

After applying the patch:

• LIBGL_ALWAYS_SOFTWARE is no longer required
• Renderer: VMware SVGA3D via /dev/dri/renderD128
• Native Wayland Chromium works
• Quickshell works
• RustDesk works
• The original 120 FPS Omarchy screensaver works
• DMA-BUF import/close errors: 0/min during the 60-second test
• wl_surface.attach / unknown object errors: 0
• Measured log growth: 0.00 MiB/min
• No black screen or compositor/client crash observed

Versions

• Hyprland source: 0.56.2 (efb50993780079460b0cbed1363e2166a2de1d9f)
• Patched package: 0.56.2.vmwgfx1-1
• Aquamarine: 0.14.0-2
• Mesa: 26.2.1-1
• Kernel: 7.1.9-arch1-2
• vmwgfx: 2.21.0.0
• open-vm-tools: 13.1.0-3
• Omarchy: 4.0.2-1

We observed one RustDesk / xdg-desktop-portal-hyprland capture freeze immediately after the initial test. Restarting RustDesk cleared it, and it did not recur in two subsequent 60-second tests. There is not enough evidence to associate that event with this patch.

This only validates one VMware configuration, but the result strongly supports the proposed vmwgfx TTM surface-handle close path as the fix for the reproduced failure.

[carlpe]

Update on the RustDesk observation: the capture freeze later became reproducible when RustDesk's Auto codec selection chose software-encoded AV1 at 2560×1600. During the freeze, xdg-desktop-portal-hyprland repeatedly logged Out of buffers and tried scheduling on already scheduled cb.

After resetting the capture session and forcing VP8, the initial monitored test completed with zero portal buffer errors and no freeze. Longer testing is still needed before attributing causality, but this appears separate from the vmwgfx DMA-BUF failure and does not change the patch result reported above.

[Pascal-0x90]

Hey thanks for testing! I am glad to hear it works for Omarchy. I had seen similar issues around talking about Omarchy not working nicely in a VM. Hopefully this helps anyone coming across this. I do think that for the short term it makes sense to use the patch but I do think the actual fix should maybe live somewhere in either the vmwgfx implementation in the Linux Kernel or in libdrm.

The two main violations of expectations I see is:

• Hyprland / Mesa assume libdrm properly handles any discrepancies of lower level interfaces
• libdrm makes assumptions that the kernel level DRM functions don't change their functionality and should always do what it expects to do (and in the vmwgfx case, it changes)

So like..maybe libdrm is the right place? I'll likely propagate this issue to libdrm at some point.

[carlpe]

Thanks for the explanation — that makes sense.

I agree the Hyprland-side fallback feels more like a practical workaround than the ideal long-term location for the fix. The test at least confirms that handling the vmwgfx TTM surface handle correctly resolves the failure in this configuration.