Support H265 for WebRTC in Safari with go2rtc

[AlexxIT]

go2rtc is one of the first projects in the World, that support HEVC/H.265 for WebRTC in web browser.

Maybe you can see articles about WebRTC H265 earlier. But I haven't seen any examples where the video was displayed in a popular browser. Some projects can support WebRTC H265 if they are used on the receiving and transmitting sides. But what if there is a real browser on one side?

I tried streaming the RTSP from the Dahua camera to Safari with Janus without success. This works for some H264 cameras, but not all of them. This can work for streaming H265 between two Apple devices. But this does not work for H265 cameras.

All because Apple doesn't respect world standards. You can forgot RTP Payload Format for High Efficiency Video Coding (HEVC) (RFC 7798) then work with Apple devices. They invented their own payload:

1. You need a correct H265 stream: VPS, SPS, PPS, I-frame, P-frame(s).
2. You need it with Annex-B headers 00 00 00 01 before each NAL unit.
3. You need split your stream on RTP payloads with one byte header:
   • 03 for payload with VPS, SPS, PPS, I-frame start
   • 01 for all next packets from this I-frame
   • 02 for payload with P-frame
   • 00 for all next packets from this P-frame
4. Don't forget set marker flag only for last packet of each Access Units

That's all. Now anyone can try to watch WebRTC H265 in Safari even if don't have H265 cameras. This is because go2rtc has optional transcode feature for almost any stream source:

streams:
  sonoff_camera: ffmpeg:rtsp://rtsp:12345678@192.168.1.123:554/av_stream/ch0#video=h265#audio=copy

If you have H265 camera, you can stream it without transcoding, even from Xiaomi Gateway 3. Because go2rtc is a small application that you can install almost anywhere.

Unfortunately Safari is the only browser at the moment that supports H265 for WebRTC.

Codecs madness

Device	WebRTC	MSE	MP4
latency	best	medium	bad
Desktop Chrome	H264	H264, H265*	H264, H265*
Desktop Safari	H264, H265*	H264	no
Desktop Edge	H264	H264, H265*	H264, H265*
Desktop Firefox	H264	H264	H264
Desktop Opera	no	H264	H264
iPhone Safari	H264, H265*	no	no
iPad Safari	H264, H265*	H264	no
Android Chrome	H264	H264	H264
masOS Hass App	no	no	no

[Frend13]

Спасибо Вам,Вы делаете огромную и трудоемкую работу!За Getwei 3 Отдельное СПАСИБИЩЕ!

[xiangxud]

use webcodecs

[itispip]

Edge support H265 as well.

[AlexxIT]

@itispip can you show SDP that confirms your words?

[compressionist]

[Assistance needed]: Only audio played during WebRTC test via RTSP stream from H.265 file on Safari

Description: I encountered an issue while testing the WebRTC via RTSP stream from an H.265 file on Safari. When I tried to stream from a file encoded with libx265 and hevc_nvenc with opus for audio, only the audio is played for WebRTC, and only the first picture is shown for MSE.
Here's what I did:

• Transcoded the file to an mp4 using libx265 and hevc_nvenc:

ffmpeg -re -i puppets_with_greenbg_hevc.mov -pix_fmt yuv420p -c:v libx265 -g 50 -preset:v superfast -tune:v zerolatency -c:a libopus -ar:a 48000 -ac:a 2 puppets_with_greenbg_x265_opus.mp4
ffmpeg -re -i puppets_with_greenbg_hevc.mov -c:v hevc_nvenc -g 50 -level:v auto -strict -2 -c:a libopus -ar:a 48000 -ac:a 2 puppets_with_alpha_hevc_nvenc_opus.mp4

• Allowed WebRTC h265 in "Experimental Features" and tried H.264 streaming from a file, which worked well using the same settings in the go2rtc.yaml and ffmpeg commands for streaming.

• Streamed RTSP from a file to go2rtc:
  ffmpeg -re -stream_loop -1 -i puppets_with_greenbg_hevc_nvenc_opus.mp4 -c copy -f rtsp rtsp://localhost:8554/stream1

• Used the following go2rtc.yaml:

streams:
  stream1: ""

• info: http://127.0.0.1:1984/api/streams?src=stream1

{
  "producers": [
    {},
    {
      "type": "RTSP source",
      "url": "rtsp://localhost:8554/stream1",
      "remote_addr": "[::1]:65513",
      "user_agent": "Lavf59.27.100",
      "medias": [
        "video, sendonly, 96 H265/90000",
        "audio, sendonly, 97 OPUS/48000/2"
      ],
      "tracks": [
        "96 H265/90000, sinks=1",
        "97 OPUS/48000/2, sinks=1"
      ],
      "recv": 6700364
    }
  ],
  "consumers": [
    {
      "type": "WebRTC client",
      "remote_addr": "udp4 prflx 192.168.2.10:60248 related :0",
      "user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.3 Safari/605.1.15",
      "send": 2613086
    }
  ]
}

I would appreciate your guidance on what I might be doing wrong. Thank you in advance for your help

[AlexxIT]

H264 and H265 from files may not be compatible for streaming. They may have B-frames or some others unsupported NAL types.

I don't support hardware encoding for H265 NVENC exact because it produce some kind of "bad" stream. Haven't had time to figure out the problem and fix it yet.

The best stream is made by cameras of good brands, such as Dahua.

[compressionist]

Dear AlexxIT,

Thank you for your prompt response. Do you have an example of an H.265+opus file that works well with go2rtc? Alternatively, could you provide me with a test stream from a Dahua camera or any other HEVC RTSP test stream that I can use for testing purposes? I ask this as I am currently unable to purchase a Dahua camera.

Also, regarding NVENC, I don't require a hardware video encoder. Instead, a software video encoder such as libx265 would suffice for my current needs.

Thank you for your assistance.
Best regards, Evgeny

[compressionist]

Dear AlexxIT,

I hope you are doing well. I have been attempting to create an RTSP stream without audio, and while it worked for MSE, I have yet to make it work for WebRTC. So I have a question about MSE - is it considered a Progressive Download?

Please provide me with some assistance with WebRTC. I would appreciate your help.
Thank you and best regards, Evgeny

Here are the ffmpeg commands I used for creating the RTSP stream without audio:

ffmpeg -re -i puppets_with_greenbg_hevc.mov -pix_fmt yuv420p -c:v libx265 -g 50 -preset:v superfast -tune:v zerolatency -bsf:v hevc_mp4toannexb -c:a libopus -ar:a 48000 -ac:a 2  -f mp4 -y puppets_with_greenbg_libx265_opus.mp4

ffmpeg -re -stream_loop -1 -i puppets_with_greenbg_libx265_opus.mp4 -c copy -an -rtsp_transport tcp -f rtsp rtsp://localhost:8554/stream1

Thank you.

[AlexxIT]

Are you sure you using WebRTC in Apple Safari with enabled H265 for WebRTC checkbox?

[compressionist]

Hi AlexxIT,
I enabled the "WebRTC H265 codec" feature under "Develop" > "Experimental Features." After that, I attempted to restart Safari, but should I do anything else?

[compressionist]

Hi AlexxIT,

I'm having trouble finding a working RTSP H.265 stream for Safari browser. I've tested my RTSP stream on Safari versions 16.3 (Apple M1, macOS Ventura v13.2.1) and 14.1.2 (macOS Mojave v10.14.6), but haven't had any success with WebRTC. Do you know where I can find a working stream?

Thank you, Evgeny

[compressionist]

Hi AlexxIT,

I hope this message finds you well. I have successfully tested the rtsp stream from go2rtc on the VLC media player. Here is the URL: rtsp://192.168.2.30:8554/stream1

I have tried it on both my MacBook Air and iPad, and everything appears to be working well, including the original HEVC file with alpha channel "puppets_with_alpha_hevc.mov" with AAC from Apple.

ffmpeg -re -stream_loop -1 -i puppets_with_alpha_hevc.mov -c copy -f rtsp rtsp://localhost:8554/stream1

However, I am experiencing an issue making WebRTC with HEVC work on the Safari browser. I would like your assistance in resolving this issue. I would appreciate your prompt response and help.

Thank you very much.
Best regards,
Evgeny

[alexpokotilo]

I implemented the same at June, 2022 https://blog.wmspanel.com/2022/07/hevc-webrtc-apple.html
I tried to handle H265 RTP from Apple using standard Pion unpackager and of course failed.
I had to process it manually the same way and was reading RFC 7798) again and again thinking I am doing something wrong. I tried to emphasize this issue here pion/webrtc#2271 and here https://groups.google.com/g/discuss-webrtc/c/Zc3-3hddEn0 but nobody cares. Unfortunately I was the first who implemented this.
If I wait a few month and found your post this would save me a lot of time.

[AlexxIT]

@alexpokotilo fair enough. I have removed phrase about "first in the World" from readme. Go2rtc has many others "first in the World". No any problem :)

[alexpokotilo]

@AlexxIT it's not about who made it first. I just started to implement whep and found your habr article and this post and send a message to support you. When I faced with that case I thought I do something wrong.
You are the first who described this problem in details. That is for sure!

[AlexxIT]

@alexpokotilo Thank you. You are developing a really cool project. This is the first time I've seen it. Very powerful features. From the video it looks pretty handy.

[alexpokotilo]

Here is what Sergio Garcia Murillo just replied about this problem

There is an PR already available for fixing the h265 packetization at WebKit:
WebKit/WebKit#15494

As a I side note, we are working on contributing h265 support to libdatachannel/OBS WHIP:
paullouisageneau/libdatachannel#932

[jimm98y]

I can confirm that in Safari 17.5 you have to use RFC 7798 to stream H265 over WebRTC.