FaceTime only sees OBS test card, not OBS scene output

[ryandesign]

Platform

Operating system and version: macOS 11.0.1
OBS Studio version: 26.1.0

Expected Behavior

The contents of the OBS scene should be visible in FaceTime.

Current Behavior

The OBS test card appears in FaceTime but the contents of the scene do not appear, regardless of whether or not "Start Virtual Camera" is clicked in OBS.

Steps to Reproduce

1. Open OBS and set up a scene
2. Click Start Virtual Camera
3. Open FaceTime and select OBS Virtual Camera as the video input device

Additional information

The OBS scene does appear in other applications when using the OBS Virtual Camera as video input, such as Zoom, Discord, QuickTime Player, so this problem is unique to a subset of applications including FaceTime.

The Console log contains repeated lines like this:

default	16:13:02.782661-0600	avconferenced	mac-virtualcam(DAL): Server is not available

Looking through your code, "Server is not available" is only printed when OBSDALMachClient's [isServerAvailable] is false, and that should only happen when [[NSMachBootstrapServer sharedInstance] portForName:@MACH_SERVICE_NAME] is nil. This makes me wonder if the com.apple.security.temporary-exception.mach-lookup.global-name entitlement needs to be added to FaceTime.app and/or obs-mac-virtualcam.plugin, but I don't have any experience with entitlements and don't know how to try to add them.

[gxalpha]

FaceTime, like most other macOS system apps, does not support virtual cameras. See this page for further information.

[ryandesign]

I have seen that page but it does not appear to explain this situation. That page talks about an inability to load DAL plugins, but in my case here the DAL plugin is clearly being loaded, as evidenced by the appearance of the OBS test card. What's failing is that the DAL plugin is unable to talk to its Mach client counterpart.

[ryandesign]

Another third party DAL plugin, https://github.com/seanchas116/SimpleDALPlugin, works fine in FaceTime and Photo Booth. So my guess is that whatever restrictions FaceTime is putting on its code, such as maybe using the hardened runtime, prevent the OBS Mac VirtualCam plugin from doing its internal Mach communication.

I tried re-signing OBS.app adding com.apple.security.temporary-exception.mach-lookup.global-name to the entitlements as an array with one element, com.obsproject.obs-mac-virtualcam.server (that's MACH_SERVICE_NAME). Assuming I re-signed correctly, that didn't make a difference. I also tried adding it to a copy of FaceTime.app, which already has com.apple.security.temporary-exception.mach-lookup.global-name set to a list of the 33 Apple Mach ports that it is allowed to talk to. I tried adding com.obsproject.obs-mac-virtualcam.server and re-signing, but then that copy of FaceTime would not launch ("You do not have permission to open the application").

I wonder if there is an alternative method of communication that the OBS Mac VirtualCam plugin could use that does not involve Mach ports. The fact that the entitlement name includes the string "temporary-exception" suggests that Apple would like apps to phase out the use of this method in favor of something else. But I am not familiar with Mach communication or alternatives.

[PatTheMav]

The proper way to do this would be to use an NSXPCConnection which also requires a helper process in the OBS bundle. @johnboiles originally refrained from using this as it would make running OBS/virtualcam without a bundle impossible.

As we need helper app bundles for CEF anyway and an app of OBS' complexity should (IMO) never be run as a lone binary to begin with, I wouldn't mind requiring an app bundle for more advanced features such as browser sources or virtual webcams.

This change however would be something some macOS developer needs to be willing to introduce/maintain, but I see it as inevitable.

With regards to this issue, it sounds like to reproduce this one would have to disable SIP on their machine and that's not something I wish to do (as it should not be disabled in the first place) and that's a use case I wouldn't consider "officially supported".

@ryandesign Would you be able to reproduce this with other apps using virtual webcams that do not require SIP to be disabled (and thus are not considered "unsupported" to begin with)?

[ryandesign]

I haven't disabled SIP on my machines.

$ csrutil status
System Integrity Protection status: enabled.

[PatTheMav]

Interesting, how did you make it pick up the virtual camera? Because by default it shouldn't.

[ryandesign]

Oops, my mistake. On this machine, a 2012 MacBook Pro, I had indeed run a script which disabled SIP in order to install macOS 11. Even when booting this machine to a different volume with macOS 10.14 where SIP is enabled, I can select the OBS virtual cam video input in FaceTime, however csrutil status still reports that some protections are disabled which is probably why it works.

On a different machine, a 2016 MacBook Pro running macOS 10.13, the OBS virtual cam does not appear in FaceTime's Video menu. However a different third-party DAL plugin, EpocCam, is in the menu, can be selected, and it does work completely, showing video from my iPhone's camera in the FaceTime window on my Mac when the EpocCam app is opened on the iPhone.

[ghost]

@ryandesign I see you noted it works for discord did you have todo anything special? I found virtual camera on MacOS Big Sur works in many app's like firefox etc but discord just says "No Video Devices" on my Mac Mini

[ryandesign]

Per my previous comment, it is apparently the disabling of System Integrity Protection that allows it to work and bypass the restrictions imposed by newer OS versions. Big Sur isn't supported on this Mac so I used someone's script to install it anyway, and that script disabled SIP for me, which I had neglected to remember initially.

On a different machine, a 2016 MacBook Pro running macOS 10.13, the OBS virtual cam does not appear in FaceTime's Video menu.

This is probably not a useful datapoint after all because 10.13 doesn't include the hardened runtime, which is seemingly what causes our problem here in the first place.

As far as I've now been able to determine, there is simply no way to do it with apps that use the hardened runtime's library validation feature. Apple says:

We have reviewed your request and have concluded that there is no supported way to achieve the desired functionality given the currently shipping system configurations. If you would like for Apple to consider adding support for such features in the future, please submit an enhancement request via Feedback Assistant

[naleiht]

I have also been looking around and found this thread. I didn't manage to make it works, however, I found out that cloning facetime.app to my desktop enabled an extra Camera selection list in Video menu.
I can see OBS Virtual Camera but selecting it does not do anything. The video stream is still coming from `Facetime HD Camera (Built-in).
I'm running big sur 11.2.3 with OBS 26.1.2, and SIP is enabled.

[ifuchs]

Cloning FT does indeed add the Camera selection but, alas, selecting a camera does nothing.

[ankanpratik]

Interestingly, cloning PhotoBooth allows me to use the Virtual Cam but on Facetime it just shows a blank screen. This is on Monterey 12.0.1.

[gxalpha]

If you're having to clone the app to even get the camera to show up, you're in unsupported territory.

As a note here, we're looking to implement a CoreMediaIO System Extension in the near future that would fix this issue (and probably every other issue with the virtualcam not working in other programs like Discord) as it
a) is the new Apple-backed way to create virtual cameras and
b) doesn't use mach shenanigans.

But for this report, as noted above we consider this scenario to be unsupported.

[Bella-Tim]

I meet the same problem with Microsoft PowerPoint when using the obs plugin code. But it works well with OBS App.

[Bella-Tim]

Oops, my mistake. On this machine, a 2012 MacBook Pro, I had indeed run a script which disabled SIP in order to install macOS 11. Even when booting this machine to a different volume with macOS 10.14 where SIP is enabled, I can select the OBS virtual cam video input in FaceTime, however csrutil status still reports that some protections are disabled which is probably why it works.

On a different machine, a 2016 MacBook Pro running macOS 10.13, the OBS virtual cam does not appear in FaceTime's Video menu. However a different third-party DAL plugin, EpocCam, is in the menu, can be selected, and it does work completely, showing video from my iPhone's camera in the FaceTime window on my Mac when the EpocCam app is opened on the iPhone.

I meet the same problem in PowerPoint,Have you sovled it finally?

[hejun-lyne]

Meet the same problem with "Server is not available".
Re-enable SIP does not solve the problem, Can anyone help?