Capture slows then freezes on Apple Silicon macs

[pnostudiodeveloper]

Describe the bug
Movie capture slows over a period of 3 minutes, before eventually stalling and freezing the app.

Your Setup (please complete the following information):

• Unity version: 2021.2.2f1
• AVPro Movie Capture version: 4.7.5
• Operating system version: MacOS 12.0
• Capture component used: Capture From Camera
• Capture settings (resolution, frame-rate, codec):

Issue is reproducible with a range of settings. However, it is reproduced most quickly with:
Offline Render
(No audio - video only)
3840 x 2160
60 FPS
H.264 Codec

To Reproduce
Steps to reproduce the behavior:

1. Start an offline Capture From Camera capture with a 20,000 frame duration
2. Wait 3-8 minutes
3. The app stalls and does not complete the capture

If you really need me to make a repro project, I can. This seems to be a fundamental issue that you could create the repro on your own? But if you need to repro to look at this more quickly, I can set it up and send it over. This is extremely urgent for my app, and is costing me business.

This was working perfectly previously. The issue seems to have been introduced a few updates ago.

I also provide the Unity framework NatCorder as a capture framework alternative to MovieCapture in the app. The NatCorder capture continues to render out flawlessly without any slowdowns or stalls. So it seems fairly clear that the instability is coming from the MovieCapture framework during the render, rather than my app.

I noticed that something about "intervals" in the release notes for MacOS / iOS - if some changes have been introduced recently to the fundamental capture encoding process recently on MacOS, perhaps it would be possible to closely examine those and see if they could have introduced some instability?

Logs
No errors are thrown in the Logs

Thank you for your assistance

[pnostudiodeveloper]

Forgot to mention that this testing was being done on a new:
2021 Macbook Pro M1 Pro - 2021.2.2f1 Silicon Editor - Intel 64-bit + Silicon build.

It appears that when I build for Intel 64-bit, and run the standalone build through Rosetta, the capture completes successfully without stalling.

Therefore, this may be a Silicon specific issue.

Since Apple is now only selling Silicon computers and Intel is being phased out, hopefully this can still be treated as an urgent issue - I hope you may have an M1 device to test with?

[MorrisRH]

Thanks for the repro project. The issue appears to be that you have not set the capture component to manual update. This means the component is already capturing a frame when you call capture from your co-routine, effectively capturing twice per-frame. At 4k this is a significant memory overhead so I'm not surprised that it's slowing down. To resolve just set the frame update to manual before you start the capture, as shown below:

void Start()
{
	captureFromCamera.FrameUpdate = CaptureBase.FrameUpdateMode.Manual;
}

[MorrisRH]

You can also set this in the inspector as shown here:

[pnostudiodeveloper]

Thank you for the suggested solution. I changed frame update to manual, and unfortunately that does not fix or change the issue.

1. This is an Offline render - so the Frame Update shouldn't be doing anything anyways, regardless of what it chosen there? In any case, it doesn't affect the stall issue I'm seeing. The capture and the Editor just froze at Frame 4665 of capture in the Editor with the Repro project I sent you with that change. It was always set to Manual in my app previously as well.

2. Have you actually been able to test the Repro on an Apple Silicon computer, in the Silicon Editor?

As per my later post above, captures complete perfectly in Intel builds. The stall seems to be replicable only on Silicon architecture.

[MorrisRH]

With the frame update set to manual I was able to run your test scene and capture all 50000 frames with a frame rate consistently around the ~50 fps mark. This was using a Mac mini (M1, 2020) and macOS 12.0.1.

[pnostudiodeveloper]

1. Understood - can you confirm that you were using the Silicon version of the Editor?

2. I'm using a MacBook Pro 2021 M1 Pro - perhaps there is a difference between the M1 and M1 Pro chips.

Please understand that I'm still experiencing a hard freeze ~1 minute into capture, and this is still posing a serious issue for my users. Not trying to get you to spend time on something that's not actually a problem.

[MorrisRH]

This was using Intel and Apple Silicon builds.
I'm installing the Apple Silicon currently, I'll get back to you with the results of that presently.

[pnostudiodeveloper]

Thank you, much appreciated. I understand how frustrating debugging something like this is. I appreciate you looking into it.

[pnostudiodeveloper]

Here is a screen capture of the freeze issue on my 2021 MacBook Pro M1 Pro when I capture in the Silicon Editor:
https://www.dropbox.com/s/cq17a4c7gwd2w4c/MovieCapture%20M1%20Pro%20MacOS%20Freeze.mov?dl=1

Quicktime screen capture replaced the rainbow wheel cursor with a normal cursor at the end of the screen capture - at the end, the cursor is actually the rainbow spinning wheel as soon as the freeze happens.

[MorrisRH]

There seems to be an issue with the Apple Silicon version of the editor and getting a temporary render target that results in the main and graphics threads becoming deadlocked.

This is something that we should be able to work around for now though.

If you could find the method CaptureFromCamera.RequiresResolve() in CaptureFromCamera.cs around line 70 and make it return false. This resolved the issue in our camera capture demo and for the test project you provided. Please let us know if it resolves the issue for your application.

We'll continue to investigate...

[pnostudiodeveloper]

Happy to report this appears to fix the issue in a release build of my app.

Both times I tested with the fix, a 10 minute 4K capture completed successfully in a Silicon build of my app.

Both times I tested with a reverted Silicon build, the 4K capture froze around 6 minutes.

I think that means this is a fix.

Is there any additional instability or issue this code change could cause? Or should we be good to go with this as a long term fix?

Thank you very much for the fix

[MorrisRH]

There should be no instabilities with this fix, in fact it should improve performance slightly as the texture resolve is not necessary on the Mac. The code will be tidied up slightly as the resolve is still required on windows and will make it into the next release.

[pnostudiodeveloper]

• Thank you - what code should I use in RequiresResolve() for my cross platform release today?

• Should that function always return false on iOS as well?

[MorrisRH]

Please use the following:

		private bool RequiresResolve(Texture texture)
		{
#if UNITY_EDITOR_OSX || UNITY_STANDALONE_OSX || (UNITY_IOS && !UNITY_EDITOR)
			// Texture resolve wholly unnecessary on macOS and iOS
			return false;
#else
			bool result = false;

			if (texture is RenderTexture)
			{
				RenderTexture rt = texture as RenderTexture;

				// Linear textures require resolving to sRGB
				result = !rt.sRGB;

				if (!result &&
					(rt.format != RenderTextureFormat.ARGB32) &&
					(rt.format != RenderTextureFormat.Default) &&
					(rt.format != RenderTextureFormat.BGRA32)
					)
				{
					// Exotic texture formats require resolving
					result = true;
				}
			}
			else
			{
				// Any other texture type needs to be resolve to RenderTexture
				result = true;
			}

			return result;
#endif
		}

[pnostudiodeveloper]

Thank you, much appreciated.

Really appreciate the great support!