Try h265 #615

mpromonet · 2023-08-20T17:56:53Z

Description

This pull request try to use recently support h265 parsing in webrtc SDK, with -o it forward h265 packet from RTSP to WebRTC.
It needs a web browser that support webrtc with h265 to validate and improve it.

Related Issue

#600
#611

Motivation and Context

Types of changes

Docs change / refactoring / dependency upgrade
Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)

okteto-cloud · 2023-08-20T17:56:55Z

Preview environment mpromonet-webrtc-streamer-pr-615 is being deployed...

liusuyi2021 · 2023-09-05T04:41:06Z

谷歌已经支持HEVC的H265了，使用最新版本0.8.2 还是无法播放呢？

yangjieshao · 2023-09-23T11:36:52Z

所以现在已经支持H265了？
(So now it supports H265?)
congratulations
有人可以提供一下支持的浏览器版本号么？
(Can someone provide a supported browser version?)

yangjieshao · 2023-09-23T11:46:30Z

是否需要更新 webrtcstreamer.js
(is that need to update webrtcstreamer.js？)

yangjieshao · 2023-09-23T11:57:38Z

可能是我chrome版本不对，画面并未在浏览器显示
(Perhaps my Chrome version is incorrect and the video is not displayed in the browser)

yangjieshao · 2023-09-23T12:16:25Z

看上去是要更新 webrtcstreamer.js ？
(t seems like we need to update webrtcstreamer.js?)

mpromonet · 2023-09-23T14:07:13Z

Hi,

The H265 support is not limited by webrtcstreamer.js, you can use another way to display webrtc stream it will be the same.
You can print video codec supported by webrtc stack of your browser with :

console.log(RTCRtpReceiver.getCapabilities('video').codecs)

Best Regards,
Michel.

strong0716 · 2023-10-17T02:53:48Z

I used the same JavaScript code to set the RTSP URL, and updated the webrtc-streamer executable to the latest version (v0.8.3). I utilized the "-o" option in the command line to pass the stream to the web. When the stream is in h264 format, I can view the video in the browser; however, I am unable to do so when the stream is in h265 format. My JavaScript code involves importing "webrtcstreamer.js" for use. T_T

(ps.) my chrome version is 118.0.5993.71

mpromonet · 2023-10-17T18:23:48Z

Hi @strong0716,

I am sorry to do the same answer than before, but it is needed that the browser support h265, you can check this with console.log(RTCRtpReceiver.getCapabilities('video').codecs). You may also read this AlexxIT/Blog#5

Best Regards,
Michel.

strong0716 · 2023-10-22T17:21:33Z

I tested the command console.log(RTCRtpReceiver.getCapabilities('video').codecs) in my Chrome browser and indeed, I didn't see support for h265.
Initially, I thought that the Chrome browser had announced support for h265, but it appears that the support is for general video streaming. It seems that webrtc does not have support for h265 yet.

Thank you!

LucaArteco · 2023-11-07T16:38:34Z

I've checked with the Tech Preview on Safari (Mac) (Release 182 (Safari 17.4, WebKit 19618.1.4.1)) and from the capabilities call should be supported but I didn't get it working neither.

Basically I'm not getting errors, but checking the decoded frames

pc.getStats().then(res => {
    res.forEach(report => {
	let bytes;
	let headerBytes;
	let packets;
	if (report.type === 'inbound-rtp') {
		if(report.framesDecoded > 0 && !firstFrame) {									
			firstFrame = true;

...
framesDecoded is 0.
What else should I try?
Many thanks!
Luca

mpromonet · 2023-11-07T20:25:02Z

Hi @LucaArteco,

If you are using -o option, the H265 frame should be forwarded through webrtc. Is it working with a H264 stream ?

Best Regards,
Michel.

LucaArteco · 2023-11-10T09:31:10Z

Hi @mpromonet yes everything's fine on H264, and I kept the -o option.

Recap:

WebRTC H.264: Working
Tested my RTSP out with VLC: Working
RTSP out video info: confirmed HEVC Codec
RTCRtpReceiver.getCapabilities('video').codecs: Confirmed presence video/H.265 in list (ONLY current Safari Tech Prev)
WebRTC HEVC: No errors, but no decoded frame (the screen is just black)

Did someone actually see the HEVC WebRTC stream working on any browser? I would like to replicate a working example to see if maybe the issues are in my RTSP out (VLC can play it but VLC eats everything usually)

EDIT: Just adding some logs after comparing two sessions. One working H.264 and one not working (H.265)

I made a little amendment in case of H.265 in order to force the priority for that code. It worked (for what I got) and now in case of h265 source I got the H265 (104 in my SDP) as first codec for both Local and Remote Description.

webrtc-streamer log
H.264
Start playing sink for "video/H264" subsession
H.265
Start playing sink for "video/H265" subsession

[247:283][14236] (PeerConnectionManager.h:183): OnStateChange channel:ServerDataChannel state:open
[247:284][14236] (PeerConnectionManager.h:274): OnDataChannel
[247:286][14236] (PeerConnectionManager.h:190): OnMessage channel:ServerDataChannel msg:remote channel openned
[247:286][14236] (PeerConnectionManager.h:183): OnStateChange channel:ClientDataChannel state:open
[247:287][14236] (PeerConnectionManager.h:190): OnMessage channel:ClientDataChannel msg:local channel openned

Codec stats (from pc.getStats())
H.265

clockRate: 90000
id: "COT01_104"
mimeType: "video/H265"
payloadType: 104
timestamp: 1699632833081
transportId: "T01"
type: "codec"

H.264

clockRate: 90000
id: "COT01_98_level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f"
mimeType: "video/H264"
payloadType: 98
sdpFmtpLine: "level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f"
timestamp: 1699633000524
transportId: "T01"
type: "codec"

I've noticed that additional parameter, sdpFmtpLine, which is not appearing for H265

Thanks!
Luca

mpromonet · 2023-11-10T20:16:16Z

Hi @LucaArteco,

I am still waiting for a support on windows or linux to make tests...
Maybe parsing https://github.com/mpromonet/webrtc-streamer/blob/master/inc/VideoDecoder.h#L74C1-L75C1 is not working well ? Increasing loglevel may helps to understand problem.
This commit d0bef6c may helps too.

Best Regards,
Michel.

LucaArteco · 2023-11-13T08:20:36Z

Hi @mpromonet thanks, we're now testing your commit.
What kind of support are you waiting for? If you need test cases with different platform and camera brands we're available, feel free to propose.
Now we're checking your suggestions and let you know.
Many thanks,
Luca

mpromonet · 2023-11-14T19:42:51Z

Hi Luca,
The problem is to identify where it is stop working... maybe server doesnot send the webrtc stream, maybe browser doesnot support what it receive.... Enabling server log may helps to identify if negociation works well and send data....
Are you able to negociate H265 webrtc stream with another tools, like between 2 webbrowsers ? go2rtc ?
Best Regards,
Michel.

LucaArteco · 2023-11-17T08:54:14Z

Hi Michel
following your refs, i've stumbled upon this PoC https://github.com/jimm98y/SharpRTSPtoWebRTC and I've been able to play H265 on Safari (17.4, also not tech preview).

I think the key reference is still AlexxIT/Blog#5

I've tested your last commit but the problem is still I'm receiving bytes, but no frames were decoded on client side.
On server side I got these logs

[152:668][3528] (h265_sps_parser.cc:516): Error in stream: invalid value, expected sps_max_num_reorder_pics[i] to be in range [0:0] found 3 instead
[152:668][3528] (livevideosource.h:192): cannot parse sps
[152:698][1420] (video_stream_encoder.cc:1774): Video frame parameters changed: dimensions=0x0, texture=1.
[152:698][1420] (VideoEncoderFactory.h:26): Create Null Encoder format:Codec name: H265, parameters: { }
[152:699][1420] (video_stream_encoder.cc:1196): ReconfigureEncoder:
Simulcast streams:
0: 0x0 min_kbps: 30 target_kbps: 600 max_kbps: 600 max_fps: 60 max_qp: 51 num_tl: 1 active: true

[152:700][1420] (NullEncoder.h:24): InitEncode
[152:701][1420] (resource_adaptation_processor.cc:117): Registered resource "EncoderUsageResource".
[152:701][1420] (quality_scaling_experiment.cc:41): QP thresholds: low: 24, high: 37
[152:701][1420] (quality_scaler.cc:216): QP thresholds: low: 24, high: 37
[152:702][1420] (resource_adaptation_processor.cc:117): Registered resource "QualityScalerResource".

On client side, I see Bytes flowing but no decoding

Many thanks,
Luca

mpromonet · 2023-11-17T21:09:21Z

Hi Luca,

In the log there is an error parsing SPS, but there is also an error forwarding PPS, I fixed this in 093e8b2.

Best Regards,
Michel.

LucaArteco · 2023-11-20T10:04:01Z

Hi Michel,
still getting the sps parsing error

Best,
Luca

mpromonet · 2023-11-20T19:47:34Z

Hi luca,
Sure, the fix allow to forward the pps that was broken, parsing SPS is the same. Maybe webrtc code doesnot support camera SPS, maybe a problem in https://webrtc.googlesource.com/src/+/refs/heads/main/common_video/h265/h265_sps_parser.cc#516.
Not sure the safari annexb stuff is implemented in webrtc...
Best Regards
Michel

mpromonet added 4 commits August 19, 2023 20:15

first try of h265

165377a

rework

4c989c5

forward frametype

f618b62

send sps even decoding fails

1a88adc

mpromonet merged commit 48b8207 into master Aug 20, 2023
10 of 31 checks passed

mpromonet deleted the try_h265 branch January 27, 2024 11:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Try h265 #615

Try h265 #615

mpromonet commented Aug 20, 2023

okteto-cloud bot commented Aug 20, 2023

liusuyi2021 commented Sep 5, 2023

yangjieshao commented Sep 23, 2023

yangjieshao commented Sep 23, 2023

yangjieshao commented Sep 23, 2023

yangjieshao commented Sep 23, 2023

mpromonet commented Sep 23, 2023 •

edited

strong0716 commented Oct 17, 2023 •

edited

mpromonet commented Oct 17, 2023 •

edited

strong0716 commented Oct 22, 2023

LucaArteco commented Nov 7, 2023

mpromonet commented Nov 7, 2023

LucaArteco commented Nov 10, 2023 •

edited

mpromonet commented Nov 10, 2023 •

edited

LucaArteco commented Nov 13, 2023

mpromonet commented Nov 14, 2023 •

edited

LucaArteco commented Nov 17, 2023

mpromonet commented Nov 17, 2023

LucaArteco commented Nov 20, 2023

mpromonet commented Nov 20, 2023 •

edited

Try h265 #615

Try h265 #615

Conversation

mpromonet commented Aug 20, 2023

Description

Related Issue

Motivation and Context

Types of changes

okteto-cloud bot commented Aug 20, 2023

liusuyi2021 commented Sep 5, 2023

yangjieshao commented Sep 23, 2023

yangjieshao commented Sep 23, 2023

yangjieshao commented Sep 23, 2023

yangjieshao commented Sep 23, 2023

mpromonet commented Sep 23, 2023 • edited

strong0716 commented Oct 17, 2023 • edited

mpromonet commented Oct 17, 2023 • edited

strong0716 commented Oct 22, 2023

LucaArteco commented Nov 7, 2023

mpromonet commented Nov 7, 2023

LucaArteco commented Nov 10, 2023 • edited

mpromonet commented Nov 10, 2023 • edited

LucaArteco commented Nov 13, 2023

mpromonet commented Nov 14, 2023 • edited

LucaArteco commented Nov 17, 2023

mpromonet commented Nov 17, 2023

LucaArteco commented Nov 20, 2023

mpromonet commented Nov 20, 2023 • edited

mpromonet commented Sep 23, 2023 •

edited

strong0716 commented Oct 17, 2023 •

edited

mpromonet commented Oct 17, 2023 •

edited

LucaArteco commented Nov 10, 2023 •

edited

mpromonet commented Nov 10, 2023 •

edited

mpromonet commented Nov 14, 2023 •

edited

mpromonet commented Nov 20, 2023 •

edited