Apple VT H264 Hardware Encoder not available in Apple Silicon (M1) MacBook Pro

[polamjag]

Platform

Operating system and version: macOS Big Sur version 11.1 (20C69) / MacBook Pro (13-inch, M1, 2020)
OBS Studio version: 26.1.2

Expected Behavior

• Apple VT H264 Hardware Encoder should appear in video encoder selector dropdown like this:
  • 

Current Behavior

• Apple VT H264 Hardware Encoder won't appear in video encoder selector dropdown:
  • 

Steps to Reproduce

1. Install OBS on Mac computers with Apple Silicon
2. Launch OBS and go to settings (preferences) screen > output section > set "Output Mode" to "Advanced" > "Streaming" tab > select "Encoder" dropdown. No "Apple VT H264 Hardware Encoder" option in the dropdown

Additional information

I'm not aware of this problem is reproduced in every Macs with Apple Silicon -- because I only have one MacBook Pro with Apple Silicon.

I searched around web (incl. issues and OBS forum posts) and Twitter, I found only one possibly relevant Twitter post mentioning both M1 Mac and Hardware Encoder of OBS: https://twitter.com/PB_Undesirables/status/1334933759871889409

Some investigations

In OBS source code, Apple VT H264 Hardware Encoder is looked up with ID com.apple.videotoolbox.videoencoder.h264.gva:

obs-studio/plugins/mac-vth264/encoder.c

Line 26 in 9eb32ee

#define APPLE_H264_ENC_ID_HW "com.apple.videotoolbox.videoencoder.h264.gva"

I enumerated VideoToolbox encoders in my Mac with Apple Silicon then I found that H.264 hardware encoder is only available with EncoderID = com.apple.videotoolbox.videoencoder.ave.avc (ref. https://gist.github.com/polamjag/3f7c0e52cf792321dde409f98cccf488#file-gistfile1-txt-L68-L76).

I made some modifications around there to check for EncoderID com.apple.videotoolbox.videoencoder.ave.avc like below:

master...polamjag:apple-vt-hardwareenc-id

This just worked 😌

Can I create a pull request at this point?

[SharkyRawr]

Sadly it does not seem to work on my M1 MacBook Air:
[VideoToolbox streaming_h264: 'h264']: Error in VTCompressionSessionCreate( kCFAllocatorDefault, enc->width, enc->height, kCMVideoCodecType_H264, encoder_spec, pixbuf_spec, NULL, &sample_encoded_callback, enc->queue, &s): The operation couldn’t be completed. (OSStatus error -12908.)
Full log here: https://gist.github.com/SharkyRawr/61a3db69ede9c9d0c402006736012cb3

kVTCouldNotFindVideoEncoderErr = -12908,

Apple VT Software encoder continues to work fine though.

I also had to merge #4105 otherwise it seemed to crash on start though not 100% certain of that.

[SharkyRawr]

This diff makes the hardware encoder work:

diff --git a/plugins/mac-vth264/encoder.c b/plugins/mac-vth264/encoder.c
index 2d4d9bde9..2dd9a7974 100644
--- a/plugins/mac-vth264/encoder.c
+++ b/plugins/mac-vth264/encoder.c
@@ -522,6 +522,11 @@ static void *vt_h264_create_hw(obs_data_t *settings, obs_encoder_t *encoder)
 	return vt_h264_create(settings, encoder, APPLE_H264_ENC_ID_HW);
 }
 
+static void *vt_h264_create_hw_extra(obs_data_t *settings, obs_encoder_t *encoder)
+{
+    return vt_h264_create(settings, encoder, APPLE_H264_ENC_ID_HW_EXTRA);
+}
+
 static void *vt_h264_create_sw(obs_data_t *settings, obs_encoder_t *encoder)
 {
 	return vt_h264_create(settings, encoder, APPLE_H264_ENC_ID_SW);
@@ -1001,7 +1006,8 @@ void register_encoders()
 			   APPLE_H264_ENC_ID_HW_EXTRA) == 0) {
 			info.id = "vt_h264_hw";
 			info.get_name = vt_h264_getname_hw;
-			info.create = vt_h264_create_hw;
+			info.create = strcmp(vt_encoders.array[i].id,
+                                 APPLE_H264_ENC_ID_HW_EXTRA) == 0 ? vt_h264_create_hw_extra : vt_h264_create_hw;
 			obs_register_encoder(&info);
 		} else if (strcmp(vt_encoders.array[i].id,
 				  APPLE_H264_ENC_ID_SW) == 0) {

It's a bit hacky but it does encode and does show a picture on receiving stream player.

[polamjag]

@SharkyRawr

Thanks for trying out my changes! I guess from your log and extra patch that H.264 hardware encoder is available with both of EncoderID com.apple.videotoolbox.videoencoder.h264.gva and com.apple.videotoolbox.videoencoder.ave.avc 🤔

Did you build natively for Apple Silicon? (I've tested my changes under Rosetta 2)

---

By the way, I found that Apple mentions this topic in https://developer.apple.com/documentation/apple_silicon/addressing_architectural_differences_in_your_macos_code#3616876:

• Encoder IDs in the Video Toolbox framework may differ on arm64 and x86_64 architectures and on different versions of macOS. For example, the value in kVTVideoEncoderSpecification_EncoderID may differ between architectures.

[nowcodingaway]

In case it helps.

• Mac M1 Mac mini 16GB
• Mac OS Big Sur 11.2 RC2
• OBS latest version released

SIMPLE SETTINGS and ADVANCED SETTINGS show different drop down options. None of which are hardware.

[SharkyRawr]

Yes I made a fully native arm64 built of OBS. I also made a quick tool to list all VideoToolbox codecs: https://github.com/SharkyRawr/videotoolbox-list
Compiling and running it with -arch arm64 and -arch x86_64 revealed no differences on my MBA M1 Big Sur 11.1:

[...]
Name: Apple H.264 (HW)
Display Name: Apple H.264 (HW)
Id: com.apple.videotoolbox.videoencoder.ave.avc
=========================
Name: Apple H.264 (SW)
Display Name: Apple H.264 (SW)
Id: com.apple.videotoolbox.videoencoder.h264
[...]

I'm missing the .gva suffix on the software encoder. Which is why it failed when attempting to run VTCompressionSessionCreate on a non-existent codec id. Added the vt_h264_create_hw_extra function to create the session with the proper id which the logs reflect:

04:21:20.039: [VideoToolbox streaming_h264: 'h264']: session created with hardware encoding
04:21:20.063: [VideoToolbox streaming_h264: 'h264']: settings:
04:21:20.063: 	vt_encoder_id          com.apple.videotoolbox.videoencoder.ave.avc
04:21:20.063: 	bitrate:               3500 (kbps)
04:21:20.063: 	fps_num:               30
04:21:20.063: 	fps_den:               1
04:21:20.063: 	width:                 1440
04:21:20.063: 	height:                900
04:21:20.063: 	keyint:                0 (s)
04:21:20.063: 	limit_bitrate:         off
04:21:20.063: 	rc_max_bitrate:        2500 (kbps)
04:21:20.063: 	rc_max_bitrate_window: 1.500000 (s)
04:21:20.063: 	hw_enc:                on
04:21:20.063: 	profile:               main

[nowcodingaway]

Wow, you mean we may not be a million miles away from an ARM version? Thank you for all the work and effort you are contributing. Not sure how I can be helpful but I have a Mac mini M1 so if there is anything I can test that helps let me know.

[RustySly]

First of all, thanks! I have a MacBook Air M1 16GB and Hardware Encoding unlocks successfully with the modified encoder.c file but with a caveat; when you chose to limit the bitrate and enter a max bitrate, if you try to record, it does not stop the first time you press the button but the second time and if you try to stream, it fails to send data completely.

[ImBada]

First of all, thanks! I have a MacBook Air M1 16GB and Hardware Encoding unlocks successfully with the modified encoder.c file but with a caveat; when you chose to limit the bitrate and enter a max bitrate, if you try to record, it does not stop the first time you press the button but the second time and if you try to stream, it fails to send data completely.

I've tested your build in here (https://obsproject.com/forum/threads/%E2%80%9Capple-vt-h264-hardware-encoder-unlocked-for-apple-silicon-m1.138433/)
It seems it's not a video issue. It's an audio issue with the the VT Hardware encoder. OBS actually don't send an audio source when you use the VT Hardware encoder.

[vangdfang]

If you’re missing audio, please see some of my comments on #4105 – it’s likely that the reason you’re missing the audio is because OBS doesn’t see the audio and video streams as “synced”. The patch there addresses it.

(Also, as a side note, you would only encounter this sync issue if natively running as an ARM binary — if you’re running under Rosetta, audio should sync in a stock OBS build, but not sure about the hardware encoder build.)

[iRonJ]

Yes I made a fully native arm64 built of OBS. I also made a quick tool to list all VideoToolbox codecs: https://github.com/SharkyRawr/videotoolbox-list
Compiling and running it with -arch arm64 and -arch x86_64 revealed no differences on my MBA M1 Big Sur 11.1:

[...]
Name: Apple H.264 (HW)
Display Name: Apple H.264 (HW)
Id: com.apple.videotoolbox.videoencoder.ave.avc
=========================
Name: Apple H.264 (SW)
Display Name: Apple H.264 (SW)
Id: com.apple.videotoolbox.videoencoder.h264
[...]

I'm missing the .gva suffix on the software encoder. Which is why it failed when attempting to run VTCompressionSessionCreate on a non-existent codec id. Added the vt_h264_create_hw_extra function to create the session with the proper id which the logs reflect:

04:21:20.039: [VideoToolbox streaming_h264: 'h264']: session created with hardware encoding
04:21:20.063: [VideoToolbox streaming_h264: 'h264']: settings:
04:21:20.063: 	vt_encoder_id          com.apple.videotoolbox.videoencoder.ave.avc
04:21:20.063: 	bitrate:               3500 (kbps)
04:21:20.063: 	fps_num:               30
04:21:20.063: 	fps_den:               1
04:21:20.063: 	width:                 1440
04:21:20.063: 	height:                900
04:21:20.063: 	keyint:                0 (s)
04:21:20.063: 	limit_bitrate:         off
04:21:20.063: 	rc_max_bitrate:        2500 (kbps)
04:21:20.063: 	rc_max_bitrate_window: 1.500000 (s)
04:21:20.063: 	hw_enc:                on
04:21:20.063: 	profile:               main

just curious how you get it to build for M1? I can get it to build after swapping out the x86 libs for arm libs, but I'm getting a mach-o binary error trying to run the app after it builds.

[vangdfang]

just curious how you get it to build for M1? I can get it to build after swapping out the x86 libs for arm libs, but I'm getting a mach-o binary error trying to run the app after it builds.

In short, yes, exactly this. I basically followed the steps for building obsdeps, but with ARM libs (and you can stitch the two together to form Universal binaries, too). The error you’re getting is almost certainly caused by the -pagezero_size and -image_base linker arguments. (As noted over in #4105, a newer LuaJIT will resolve the need for this on Intel, but with the current version in obsdeps, omitting these will break LuaJIT on Intel, but not for ARM.)

[iambenmitchell]

just curious how you get it to build for M1? I can get it to build after swapping out the x86 libs for arm libs, but I'm getting a mach-o binary error trying to run the app after it builds.

In short, yes, exactly this. I basically followed the steps for building obsdeps, but with ARM libs (and you can stitch the two together to form Universal binaries, too). The error you’re getting is almost certainly caused by the -pagezero_size and -image_base linker arguments. (As noted over in #4105, a newer LuaJIT will resolve the need for this on Intel, but with the current version in obsdeps, omitting these will break LuaJIT on Intel, but not for ARM.)

Just saw this, where do I specify the Arm libs? in the full-build-macos.sh file? I opened an issue here #4266

[vangdfang]

FYI, #4105 landed this evening, so I believe the issue is at least partially resolved -- or will be when an ARM64-native build of OBS's dependencies are complete, and OBS itself can build as a native binary. (BTW, thanks @polamjag for the callout to the Video Toolbox porting guide specifically -- while I chose a different path, I wouldn't have known about the issue had you and @SharkyRawr tagged my PR!)

[tomlube]

Anyone want to post their m1 build? haha

[iambenmitchell]

Anyone want to post their m1 build? haha

Also want this. I can't figure out how to build it for arm myself manually

[SharkyRawr]

Anyone want to post their m1 build? haha

Also want this. I can't figure out how to build it for arm myself manually

I made one, using brew to install all the deps and compile with cmake and generate the dmg with cpack.
https://github.com/SharkyRawr/m1-builds/raw/main/bin/obs-studio-x64-26.1.2-251-g54892b27d-modified.dmg.zip

[SharkyRawr]

I made one, using brew to install all the deps and compile with cmake and generate the dmg with cpack.
https://github.com/SharkyRawr/m1-builds/raw/main/bin/obs-studio-x64-26.1.2-251-g54892b27d-modified.dmg.zip

Did you test it? I can't open it

I'm not in the apple dev program so it's not signed and macOS freaks out, you can rightlick the Dmg and choose open and then a dialog will ask you if you want to open it anyways. Do the same with the .app bundle and it should work.

[iambenmitchell]

I just built it myself on my x86 build server using the latest source code as of 20 minutes ago and it unlocked the hardware encoder even for M1?

If you want to download it you can, just note it is x86 not Apple Silicon

Download

[paoloestar]

Anyone want to post their m1 build? haha

Also want this. I can't figure out how to build it for arm myself manually

I made one, using brew to install all the deps and compile with cmake and generate the dmg with cpack.
https://github.com/SharkyRawr/m1-builds/raw/main/bin/obs-studio-x64-26.1.2-251-g54892b27d-modified.dmg.zip

Can you tell me what commands you ran exactly (arguments etc) and also did you have to modify any files?

Works for me! -thanks for the build. Looks we are not far away from a final m1 native build!

[polamjag]

#4170 (comment)

This is because the problem is about the logic to look up hardware encoder (4c0bb38 in #4105 is about that) but not the binary architecture of OBS

[richiepreece]

Here is arguably a better solution. You simply replace a .so file in the original application and it enables hardware encoding. My CPU went from having a core pegged and dropping frames, to basically no effort from the computer at all

Looks like @RustySly is part of this thread too. He's the one that supplied this .so file.

[fknorn]

Thank you all. So, being the impatient "expert" I am, I got another step closer. I combined @SharkyRawr's M1 build (Thanks again for that! That one runs native on Apple Silicon and, including the hardware-encoder, but not the Virtual Camera)

https://github.com/SharkyRawr/m1-builds/raw/main/bin/obs-studio-x64-26.1.2-251-g54892b27d-modified.dmg.zip

with Virtual Camera plugin from the official release, copying the file ops-mac-virtualcam.plugin from the OBS.app > Contents > Resources > data folder to the corresponding location under the M1 build, and voilà, it works (in Zoom at least)!

And that seems to be native, since all OBS-related processes show "Apple" as architecture in the system's Activity Monitor.

Now, the only thing that's left is getting the virtual camera to show up in more apps, like Skype, not just Zoom...

[WizardCM]

Hi everyone, please keep discussion in this issue specific to the missing encoder, and not for troubleshooting custom builds or modifying signatures of other apps.

[RustySly]

actually, the issue is not completely resolved yet. now, the apple vt hardware encoder is accessible but if you try to limit the bitrate, you cannot stream or record as expected (i explained it before in this discussion). i built the obs for m1 myself and i can say that the problem is still there for both intel and apple builds.

[iambenmitchell]

Anyone want to post their m1 build? haha

Also want this. I can't figure out how to build it for arm myself manually

I made one, using brew to install all the deps and compile with cmake and generate the dmg with cpack.
https://github.com/SharkyRawr/m1-builds/raw/main/bin/obs-studio-x64-26.1.2-251-g54892b27d-modified.dmg.zip

Could you let me know the exact commands you used to compile? I can't figure it out. I also tried to make an Xcode project but it said there was some file missing? not sure I am doing it from the right dir

[RytoEX]

With the merging of #4105, this should be fixed as of OBS Studio 27.0.0-rc1. As such, I'm going to close this. If the Apple VT Hardware Encoder is still not available in the latest OBS Studio 27 Release Candidate (currently OBS Studio 27.0.0-rc2), please comment and we can reopen this if necessary.

[DevAndArtist]

Is it incorrect that this patch does not enabled hardware encoder in Simple Output Mode? I do have the option with the advanced mode, but not with the simple one. 🤔

• Mac mini M1 (macOS 11.3 (20E232))
• OBS: 27.0.0-rc3

[geerlingguy]

@DevAndArtist - Even with non-M1 macs, the hardware encoder option was always hidden behind the 'advanced' mode tab, because its use has some limitations that can cause issues with certain types of streams.

[paulreimer]

@geerlingguy I curious what those limitations might be (I've been happily using the Hardware Encoder; hopefully I haven't been excluding viewers or presenting a poor experience!)

[Alvt5]

I only see x264 software encoder option (greyed out) for 27.2.2 , there is no other option for hardware. I uploaded the modified .so into the plugins dir and also no change to the option.
I am on m1 pro, Monterey with latest ops. Any ideas?

[koenhendriks]

When enabling the Hardware Encoder under output > advanced > streaming, does it also enable it for the Virtual Camera?

[gxalpha]

Virtual camera is a raw output, so no. The encoder doesn't matter there.

[koenhendriks]

So for now on Mac there is no way of hardware encoding virtual cam? Looking at 140% CPU now ☹️

[gxalpha]

OBS itself doesn't encode the virtualcam output at all. This means that any encoding would be done by whatever app your using the virtualcam in.

[harryqt]

As of v28.0.1, if I only record (not streams) then should I use Software h264 encoding or Hardware Apple h264?

[RytoEX]

As of v28.0.1, if I only record (not streams) then should I use Software h264 encoding or Hardware Apple h264?

Use whatever works best for your needs. Please use our forums or Discord server for support/usage questions.