macOS/Metal: currentDrawable can return null, freezing the render loop.

[petrihakkinen]

I'm getting frequent freezing in sokol_gfx.h in my vic20 emulator app running on macOS.

_SOKOL_PRIVATE void _sg_mtl_begin_pass(_sg_pass_t* pass, const sg_pass_action* action, int w, int h) {
    ....
     
    /* if this is the first pass in the frame, create a command buffer */
    if (nil == _sg.mtl.cmd_buffer) {
        /* block until the oldest frame in flight has finished */
        dispatch_semaphore_wait(_sg.mtl.sem, DISPATCH_TIME_FOREVER);   <--- gets stuck here
        _sg.mtl.cmd_buffer = [_sg.mtl.cmd_queue commandBufferWithUnretainedReferences];
    }

The freezing seems to happen only when there's other system activity. For example, when I press space in Finder to preview a file the app usually freezes.

Note that I'm running on an ancient macOS version (10.13.6), so this could be a problem in OS/driver, my app or sokol gfx.

Any ideas?

[floooh]

It's probably completely unrelated to your problem, but (in case you're using sokol_gfx.h together with sokol_app.h) can you check that your version of sokol_app.h doesn't have this "late even polling" which I commented out recently:

sokol/sokol_app.h

Lines 2887 to 2928 in c602d83

	_SOKOL_PRIVATE void _sapp_macos_poll_input_events() {
	/*
	NOTE: late event polling temporarily out-commented to check if this
	causes infrequent and almost impossible to reproduce probelms with the
	window close events, see:
	https://github.com/floooh/sokol/pull/483#issuecomment-805148815
	const NSEventMask mask = NSEventMaskLeftMouseDown |
	NSEventMaskLeftMouseUp|
	NSEventMaskRightMouseDown |
	NSEventMaskRightMouseUp |
	NSEventMaskMouseMoved |
	NSEventMaskLeftMouseDragged |
	NSEventMaskRightMouseDragged |
	NSEventMaskMouseEntered |
	NSEventMaskMouseExited |
	NSEventMaskKeyDown |
	NSEventMaskKeyUp |
	NSEventMaskCursorUpdate |
	NSEventMaskScrollWheel |
	NSEventMaskTabletPoint |
	NSEventMaskTabletProximity |
	NSEventMaskOtherMouseDown |
	NSEventMaskOtherMouseUp |
	NSEventMaskOtherMouseDragged |
	NSEventMaskPressure |
	NSEventMaskDirectTouch;
	@autoreleasepool {
	for (;;) {
	// NOTE: using NSDefaultRunLoopMode here causes stuttering in the GL backend,
	// see: https://github.com/floooh/sokol/issues/486
	NSEvent* event = [NSApp nextEventMatchingMask:mask untilDate:nil inMode:NSEventTrackingRunLoopMode dequeue:YES];
	if (event == nil) {
	break;
	}
	[NSApp sendEvent:event];
	}
	}
	*/
	}

I had seen weird hangs on macOS after this PR was merged, but usually when trying to press the close button (but I guess it could happen with other events). Even this commenting out is just an experiment, I haven't seen the problem since then, but OTH currently I'm mostly working on Windows and Linux stuff.

I don't know how this could lead to getting stuck on that dispatch_wait_semaphore call though.

The semaphore is signalled in the command buffer's completion handler here:

sokol/sokol_gfx.h

Lines 10728 to 10731 in c602d83

	[_sg.mtl.cmd_buffer addCompletedHandler:^(id<MTLCommandBuffer> cmd_buffer) {
	_SOKOL_UNUSED(cmd_buffer);
	dispatch_semaphore_signal(_sg.mtl.sem);
	}];

This is essentially taken from the Xcode "Game" sample project:

• in Xcode: New Project => macOS Tab => Game [Next]
• select "Language: Objective-C" and "Game Technology: Metal"

In the loaded project, search for "dispatch_semaphore" via "Find in Project":

• The semaphore is created with dispatch_semaphore_create(kMaxBuffersInFlight); (where MaxBuffersInFlight is 3, while in sokol_gfx.h it's 2.
• At the start of drawInMTKView(): dispatch_semaphore_wait(_inFlightSemaphore, DISPATCH_TIME_FOREVER);
• And following that the completion handler is added which invokes dispatch_semaphore_signal(block_sema);. In sokol_gfx.h the completion handler is added in _sg_mtl_commit(), but this shouldn't matter as long as it happens before the commit call of the Metal command buffer.

The only thing that appears different is the MaxBuffersInFlight 3 vs 2. You could try setting that to 3 in sokol_gfx.h (search for SG_NUM_INFLIGHT_FRAMES) and if it fixes the problem (which would be strange though), please let me know so we can do this generally.

Also another thing: Are you seeing any Metal validation layer messages? Maybe the command buffer doesn't actually complete because of a validation error? If you're not starting from within Xcode you need to enable the validation layer with an environment variable:

>  export METAL_DEVICE_WRAPPER_TYPE=1
> ./demo

You should see a message 'Metal API Validation Enabled' on the terminal.

That's all I can think of for now. Please keep me posted if you find something :)

[floooh]

PS: Another thing to check would be if the problem can be reproduced in the Xcode Game sample project.

[petrihakkinen]

Thanks for the quick reply!

In case it matters, I'm not using sokol app. Instead I'm using slightly modified version of minimal osxentry.mm from sokol examples, because it was easier to modify and smaller in general (I added always on top windows + some input tweaks).

After enabling METAL_DEVICE_WRAPPER_TYPE and setting SG_NUM_INFLIGHT_FRAMES to 3, I'm getting:
-[_MTLCommandBuffer presentDrawable:]:350: failed assertion `drawable must not be nil.'

Again this happens after a while when I use Finder while the app is running.

I haven't debugged it further yet.

[petrihakkinen]

The problem seems to be that [MTKView currentDrawable] returns nil. Docs mention that this is a valid return value:
"If all drawable objects are in use, the value of this property is nil. Your app should check that currentDrawable isn’t nil before attempting to draw."

So if drawable is nil, the semaphore probably doesn't get signaled in _sg_mtl_commit.

[petrihakkinen]

Found a 100% repro case at least on my mac. If I try to preview a mp4 video file while the app is running, it always crashes/freezes with drawable being nil. Looks like Finder's preview steals the metal drawable or something like that. Maybe on 10.13.6 the number of metal drawables is very limited.

[petrihakkinen]

Added a nil check to start of frame() so that the entire frame gets skipped. After this the app no longer freezes/crashes, but it does not recover from loosing the drawable either. That is, the window stop rendering but I can move it around etc. although it is very sluggish. Maybe some API still needs to be called to make it recover. I guess it's now busy looping or something until currentDrawable returns non-nil which never happens.

I'll stop this spamming for now :)

[petrihakkinen]

One more detail.. If I do something like this in _sg_mtl_commit (which is probably not 100% correct)

    if (cur_drawable) {
        [_sg.mtl.cmd_buffer presentDrawable:cur_drawable];
        [_sg.mtl.cmd_buffer addCompletedHandler:^(id<MTLCommandBuffer> cmd_buffer) {
            _SOKOL_UNUSED(cmd_buffer);
            dispatch_semaphore_signal(_sg.mtl.sem);
        }];
    } else {
        dispatch_semaphore_signal(_sg.mtl.sem); // questionable??
    }

The app does not crash/freeze (but rendering stops) when the mp4 preview opens. Now I if close the mp4 preview, the app is still not rendering (but responding to events). If I now minimize the sokol window by clicking the yellow button next to close button and reactivate the window, the app resumes rendering and everything is back to normal. So close to a fix that it's driving me crazy :)

[floooh]

Ok, thanks a for investigating and the detailed information. I think the key is that currentDrawable can return nil, I hadn't considered this. I need to ponder this for a bit... so far I think if no drawable is available, then the app should pause for a little bit, not call the frame callback (or maybe it's better to just skip the presentation)...

Btw, I'll change the issue title with the new information.

[floooh]

PS: random question that just popped up in my mind, is it guaranteed in your code, that the full sequence sg_begin_default_pass() => sg_end_pass() => sg_commit() is called in each frame in your code, or are there situations where this is either completely or partially skipped (before your workaround for this bug)?

In any case, I'll finish my current work on the window icon stuff, and then I'll look into this issue :)

[floooh]

Btw, I can't seem to reproduce the problem when playing an MP4 (created with the macOS screen recording feature), no matter if double clicking the video or previewing via Space.

I'm on a mid-2014 13"MBP with the current Big Sur beta (11.3 Beta (20E5217a)).

But I think it should be possible to force the problem manually by returning null from sapp_metal_get_drawable(). But as I said, I'll look into it proper in a few days after I finished the icon stuff.

[petrihakkinen]

Hi! The full sequence sg_begin_default_pass => sg_end_pass => sg_commit is always called in the frame callback called by osxentry.mm.

I'm not surprised that the problem does not repro on all setups. The number of available drawables is most likely system dependent (GPU memory, drivers, etc.).

The problem which I haven't been able to solve (yet) is how to recover after currentDrawable has returned nil. The wording in Apple's docs imply that currentDrawable returns a cached value which is only changed by some API calls, but the wording is not clear. So if I skip the whole render loop, currentDrawable always returns nil probably because nothing changes the cached value.

However, if I skip only presentDrawable (see my previous message) minimizing/reactivating the window makes the app recover but I don't know how to do this programmatically (without actually minimizing the window).

[petrihakkinen]

I'm beginning to think a proper solution may require ditching MTKView which is a higher level abstraction on top of CAMetalLayer... CAMetalLayer has nextDrawable which sounds what we should be calling instead of currentDrawable:
https://developer.apple.com/documentation/quartzcore/cametallayer/1478172-nextdrawable?language=objc

[floooh]

Hmm so far I was quite happy with MTKView because it (hopefully) wraps a lot of "best practices" automatically which are hard to figure out. Waiting for a drawable to become available seems to make much more sense indeed though, strange that MTKView doesn't do this on its own.

[floooh]

I've been looking at the Metal example project again, and the drawInMTKView method essentially looks like this:

    ...
    MTLRenderPassDescriptor* renderPassDescriptor = view.currentRenderPassDescriptor;
    if (renderPassDescriptor != nil) {
        ...
        [commandBuffer presentDrawable:view.currentDrawable];
    }
    [commandBuffer commit];

...it's interesting that they don't specifically test if currentDrawable is nil, but they check if the currentRenderPassDescriptor is nil instead, and if yes, skip drawning (but still call [commandBuffer commit].

I think I'll try to copy this 'strategy' in sokol_gfx.h.

[petrihakkinen]

Yep, it checks the same thing basically:
"This property [currentRenderPassDescriptor] is nil if the view’s device property isn’t set or if currentDrawable is nil. Your app should check that currentRenderPassDescriptor isn’t nil before attempting to use it."

[floooh]

Ok, resulting commit:

0c5bc3a

The main thing is that presentDrawable isn't called if the drawable-callback returns nil. I also moved the completedHandler block up into the first begin-pass of a frame, but that shouldn't change the behaviour, it's just closer to the official example code now.

I have tested this with sokol_app.h by returning nil from the sapp_metal_get_renderpass_descriptor() and sapp_metal_get_drawable() functions (in various combinations), and it seems to work fine. The drawing loop doesn't lock up, and the CPU also remains calm (because MTKView continues to call the frame callback at 60Hz, e.g. the frame loop doesn't run unthrottled or something).

Not sure if it fixes anything on your end though, because I can't seem to provoke [MTKView currentDrawable] into returning a nil on my machine.

PS: also no idea how to deal with this "once currentDrawable returns nil, it always returns nil" unfortunately.

[petrihakkinen]

Thank you! I will give it a spin soon.

[floooh]

PS: another interesting tidbit I noticed yesterday when testing the new code: when the sokol_gfx.h Metal backend was new (not sure what macOS version I was on back then), there were situations where Metal functions returned nil object ids when the window is minimized, for instance:

sokol/sokol_gfx.h

Lines 10613 to 10616 in 0c5bc3a

	/* default pass descriptor will not be valid if window is minimized,
	don't do any rendering in this case */
	_sg.mtl.pass_valid = false;
	return;

This no longer seems to be true on newer macOS versions, e.g. I'm still getting a valid render-pass-descriptor back when the window is minimized.

[petrihakkinen]

I no longer get the error about currentDrawable being nil, but unfortunately the app still hangs in the semaphore wait at line 10579 in sokol_gfx.h.

[petrihakkinen]

With the following quick-hack patch the issues seems to disappear. This is probably incorrect & unsafe though...

@@ -10731,6 +10731,8 @@ _SOKOL_PRIVATE void _sg_mtl_commit(void) {
     }
     if (nil != cur_drawable) {
         [_sg.mtl.cmd_buffer presentDrawable:cur_drawable];
+    } else {
+        dispatch_semaphore_signal(_sg.mtl.sem);
     }
     [_sg.mtl.cmd_buffer commit];

[petrihakkinen]

Nope.. I still got it hanging, just seems to be more rare.

[floooh]

Hmm. Could you try if you also get the hangs with the sokol-sapp samples?

You can build without fips like this (to get the sokol-shdc tool, clone this first into the same directory as sokol-samples and sokol: https://github.com/floooh/sokol-tools-bin

cd sokol-samples/sapp
> ../../sokol-tools-bin/bin/osx/sokol-shdc -i cube-sapp.glsl -o cube-sapp.glsl.h -l metal_macos
> cc cube-sapp.c ../libs/sokol/sokol.m -o cube-sapp -DSOKOL_METAL -fobjc-arc -I../../sokol -I ../libs -framework Metal -framework Cocoa -framework MetalKit -framework Quartz -framework AudioToolbox

[floooh]

Btw I'm currently also seeing a similar freezing effect in my multiwindow experiments...

The freeze happens when two windows are open (each with its own MTKView), and always during resizing. The location of the freeze is [MTKView currentRenderPassDescriptor] (AFAIK this needs to aquire a new drawable, and this may block, see: https://developer.apple.com/library/archive/documentation/3DDrawing/Conceptual/MTLBestPracticesGuide/Drawables.html)

I'm now trying to find out why the MTKView is running out of drawables when resizing. I have a hunch that maybe the structure of sokol_gfx.h's Metal backend somehow holds onto the drawable too long by holding a strong reference (which I'm not seeing though). I experimented a bit and thought that I had fixed the freeze by moving the [cmdBuffer presentDrawable] up into begin pass, but that must have been a fluke... at least I'm now seeing the freezes again.

The next step after that would then be to ditch MTKView, drop down to CAMetalLayer and see if this is less opaque and gives me more control.

[floooh]

...interesting, the freeze only occurs when one window is completely obscured while resizing the other window. This explains the "fluke".

Also the 1-second freeze is explained here:

https://developer.apple.com/documentation/quartzcore/cametallayer/1478172-nextdrawable?language=objc

(under "Discussion")...

[oviano]

I have my own framework that uses sokol_gfx and I find I get the 1s freeze inside my renderpass_descriptor_cb whenever I toggle between fullscreen and windowed.

The switch in or out of full screen works, and it's actually a couple of frames later that the freeze occurs when calling...

[view currentRenderPassDescriptor]

...on my MTKView.

It happens every single time, and once the timeout has elapsed everything carries on as normal, so it's aesthetically
displeasing more than anything else. The same doesn't happen with regular resizing (dragging), only when switching in and out of full screen using the green button on the window frame.

I wondered @floooh if you had figured out any more possible reasons for drawables being held onto for too long inside sokol_gfx during your multi-window experiments....?

[floooh]

I haven't found out anything new unfortunately, the problem seems to be down in [CAMetalLayer nextDrawable] and also happens without MTKView, IFIR I only observed this behaviour when a window gets obscured (which isn't much of a problem in a single-window-scenario), maybe switching in and out of fullscreen causes a similar behaviour though.