libghostty: IOSurfaceLayer display callback not cleared on renderer deinit (use-after-free after ghostty_surface_free)

[mahdiarfrm]

Summary

When embedding libghostty on macOS, the renderer's CALayer keeps a raw function pointer into freed memory after ghostty_surface_free. If the host app keeps the surface's NSView mounted past the free, for example while a pane-close animation finishes, the next CoreAnimation display pass calls into the freed renderer and the process aborts.

Crash signature

BUG IN CLIENT OF LIBPLATFORM: os_unfair_lock is corrupt

Faulting thread: CA::Transaction::commit → IOSurfaceLayer display → renderer drawFrame → draw_mutex lock in freed memory.

Root cause

Verified against main (built 2026-07-01, code unchanged as of 2026-07-08):

• src/renderer/metal/IOSurfaceLayer.zig: the custom CALayer subclass overrides display to call the display_cb/display_ctx ivars — a raw pointer to the Zig Renderer.
• src/renderer/Metal.zig: loopEnter sets the callback; deinit releases the layer reference but never clears the ivars.
• The layer is retained by the host view's layer tree, so it outlives the renderer, and needsDisplayOnBoundsChange = true means a bounds change alone triggers display after teardown.

Ghostty.app doesn't hit this because it frees the surface after its view (and layer) has left the hierarchy. Embedders that free while the view is still mounted hit it reliably.

Reproduction (embedding scenario)

1. ghostty_surface_new with the nsview platform config; leave the view mounted in a window.
2. ghostty_surface_free.
3. Trigger any CA display pass on the still-mounted layer (resize, animation, transaction commit) → abort in drawFrame.

Fix

Clearing the callback during renderer teardown resolves it:

 pub fn deinit(self: *Metal) void {
+    self.layer.setDisplayCallback(null, null);
     self.queue.release();
     self.device.release();
     self.layer.release();
 }

We've been running this patch in a production embedder (Conterm) with no regressions. Happy to open a PR if this is the right shape.

[mitchellh]

Fixed on main