Enable low latency mode #169

jianjunz · 2022-12-09T09:25:27Z

This PR picks commits for low latency mode. Some changes will be submitted in separate PRs.

This change enables HEVC support after rebasing to M108. HEVC related structures are always declared, but the codec is only implemented when GN arg rtc_use_h265 is enabled (macro WEBRTC_USE_H265 is defined).

* Add HEVC support for iOS/Android * Some changes for building with OWT * Enable openssl * Add create_peerconnection_factory to WebRTC.framework. (open-webrtc-toolkit#46) * Set kVTCompressionPropertyKey_RealTime to true. (open-webrtc-toolkit#51) * H265 packetization_mode setting fix (open-webrtc-toolkit#53) * add H.265 QP parsing logic (open-webrtc-toolkit#47) * Fix linux build error. (open-webrtc-toolkit#54) * Add h264 prefix NAL parser implmentation for enabling frame-marking for h.264 (open-webrtc-toolkit#58) * Make hevc rtp depacketizer/tracker conforming to h.264 design Co-authored-by: jianjunz <jianjun.zhu@intel.com> Co-authored-by: Cyril Lashkevich <notorca@gmail.com> Co-authored-by: Piasy <xz4215@gmail.com> Co-authored-by: ShiJinCheng <874042641@qq.com>

…t lower/upper bounds of bitrate for gpra (open-webrtc-toolkit#83)

…-webrtc-toolkit#91)

)

…tc-toolkit#94)

* Set generic frame descriptor extension for HEVC * Enable more rtcp feedbacks for loss/reference indication

There is code to set different protection levels for keyframes vs. delta frames, but because this set_is_key_frame was never called, the distinction was never used. Also, clear old media packets from a previous frame, which can happen if a keyframe request comes from the receiver.

…-webrtc-toolkit#128) Enable flex-fec by application instead of hard-code.

…kit#160)

This reverts commit 61912d5. This change introduced too much code in header, and some code doesn't with 108. Revert it and apply it manually later if it's needed.

api/video/builtin_video_bitrate_allocator_factory.cc

call/rtp_transport_controller_send.cc

taste1981 · 2022-12-13T01:28:13Z

modules/congestion_controller/goog_cc/BUILD.gn

@@ -8,6 +8,8 @@

 import("../../../webrtc.gni")

+defines = []
+
 config("bwe_test_logging") {
  if (rtc_enable_bwe_test_logging) {
    defines = [ "BWE_TEST_LOGGING_COMPILE_TIME_ENABLE=1" ]


Checked with the latest code, defines=[] can be removed from line 11 because all paths below has defines.

modules/congestion_controller/goog_cc/goog_cc_network_control.cc

modules/rtp_rtcp/source/rtp_format.cc

taste1981 · 2022-12-13T01:39:37Z

pc/used_ids.h

@@ -168,8 +168,8 @@ class UsedRtpHeaderExtensionIds : public UsedIds<webrtc::RtpExtension> {
        }
      }
    }
-    RTC_DCHECK(next_extension_id_ >= min_allowed_id_);
-    RTC_DCHECK(next_extension_id_ <= max_allowed_id_);
+    //RTC_DCHECK(next_extension_id_ >= min_allowed_id_);


These two lines were commented out in old cg branch only. Instead of removing them, I'm recovering it in this new branch. If we still hit these DCHECK failures, let's see if we can fix it then.

test/scenario/video_stream.cc

video/video_stream_encoder.cc

https://bugs.webkit.org/show_bug.cgi?id=258794 Reviewed by Youenn Fablet. The current WebRTC HEVC is using generic packetization instead of RFC 7789 Packetization. So this is about implementing the RFC 7798 Packetization. Fix HEVC depacketizer issues. (WebKit#185) Enalbing low latency mode for RTC (WebKit#169) Enable HEVC support. (WebKit#165) Fix out-of-bounds write in H265VpsSpsPpsTracker (WebKit#163) Apply fix bitstream logic to RtpVideoStreamReceiver2 (WebKit#142) Add missing CODEC_H265 switch case (WebKit#136) Add HEVC support for iOS/Android (WebKit#68) H265 packetization_mode setting fix (WebKit#53) Add H.265 QP parsing logic (WebKit#47) This patch is extracted from following Open WebRTC Toolkit code changes: <open-webrtc-toolkit/owt-deps-webrtc#185> <open-webrtc-toolkit/owt-deps-webrtc#169> <open-webrtc-toolkit/owt-deps-webrtc#165> <open-webrtc-toolkit/owt-deps-webrtc#163> <open-webrtc-toolkit/owt-deps-webrtc#142> <open-webrtc-toolkit/owt-deps-webrtc#136> <open-webrtc-toolkit/owt-deps-webrtc#68> <open-webrtc-toolkit/owt-deps-webrtc#53> <open-webrtc-toolkit/owt-deps-webrtc#47> co-authoured by: taste1981 <jianlin.qiu@intel.com> jianjunz <jianjun.zhu@intel.com> Cyril Lashkevich <notorca@gmail.com> Piasy <xz4215@gmail.com> ShiJinCheng <874042641@qq.com> Andreas Unterhuber <andreas.unterhuber@keepinmind.info> dong-heun <63987238+dong-heun@users.noreply.github.com> * Source/ThirdParty/libwebrtc/Configurations/libwebrtc.xcconfig: * Source/ThirdParty/libwebrtc/Source/webrtc/api/video/video_codec_type.h: * Source/ThirdParty/libwebrtc/Source/webrtc/api/video_codecs/video_codec.cc: * Source/ThirdParty/libwebrtc/Source/webrtc/api/video_codecs/video_codec.h: (webrtc::VideoCodecH265::operator!= const): * Source/ThirdParty/libwebrtc/Source/webrtc/api/video_codecs/video_decoder_software_fallback_wrapper.cc: * Source/ThirdParty/libwebrtc/Source/webrtc/api/video_codecs/video_encoder.cc: * Source/ThirdParty/libwebrtc/Source/webrtc/api/video_codecs/video_encoder.h: * Source/ThirdParty/libwebrtc/Source/webrtc/call/rtp_payload_params.cc: * Source/ThirdParty/libwebrtc/Source/webrtc/common_video/h265/h265_common.cc: * Source/ThirdParty/libwebrtc/Source/webrtc/common_video/h265/h265_common.h: * Source/ThirdParty/libwebrtc/Source/webrtc/common_video/h265/h265_pps_parser.cc: * Source/ThirdParty/libwebrtc/Source/webrtc/common_video/h265/h265_pps_parser.h: * Source/ThirdParty/libwebrtc/Source/webrtc/common_video/h265/h265_sps_parser.cc: * Source/ThirdParty/libwebrtc/Source/webrtc/common_video/h265/h265_sps_parser.h: * Source/ThirdParty/libwebrtc/Source/webrtc/common_video/h265/h265_vps_parser.cc: * Source/ThirdParty/libwebrtc/Source/webrtc/common_video/h265/h265_vps_parser.h: * Source/ThirdParty/libwebrtc/Source/webrtc/logging/rtc_event_log/rtc_event_log_parser.cc: * Source/ThirdParty/libwebrtc/Source/webrtc/media/base/media_constants.cc: * Source/ThirdParty/libwebrtc/Source/webrtc/media/base/media_constants.h: * Source/ThirdParty/libwebrtc/Source/webrtc/media/engine/internal_decoder_factory.cc: * Source/ThirdParty/libwebrtc/Source/webrtc/modules/rtp_rtcp/source/create_video_rtp_depacketizer.cc: * Source/ThirdParty/libwebrtc/Source/webrtc/modules/rtp_rtcp/source/rtp_format.cc: * Source/ThirdParty/libwebrtc/Source/webrtc/modules/rtp_rtcp/source/rtp_format_h265.cc: * Source/ThirdParty/libwebrtc/Source/webrtc/modules/rtp_rtcp/source/rtp_format_h265.h: (webrtc::RtpPacketizerH265::PacketUnit::PacketUnit): (webrtc::VideoRtpDepacketizerH265::~VideoRtpDepacketizerH265): Deleted. * Source/ThirdParty/libwebrtc/Source/webrtc/modules/rtp_rtcp/source/rtp_sender_video.cc: * Source/ThirdParty/libwebrtc/Source/webrtc/modules/rtp_rtcp/source/rtp_video_header.h: * Source/ThirdParty/libwebrtc/Source/webrtc/modules/video_coding/codecs/h265/include/h265_globals.h: * Source/ThirdParty/libwebrtc/Source/webrtc/modules/video_coding/encoded_frame.cc: * Source/ThirdParty/libwebrtc/Source/webrtc/modules/video_coding/h265_vps_sps_pps_tracker.cc: * Source/ThirdParty/libwebrtc/Source/webrtc/modules/video_coding/h265_vps_sps_pps_tracker.h: * Source/ThirdParty/libwebrtc/Source/webrtc/modules/video_coding/include/video_codec_interface.h: * Source/ThirdParty/libwebrtc/Source/webrtc/modules/video_coding/packet_buffer.cc: * Source/ThirdParty/libwebrtc/Source/webrtc/rtc_base/experiments/min_video_bitrate_experiment.cc: * Source/ThirdParty/libwebrtc/Source/webrtc/sdk/objc/components/video_codec/nalu_rewriter.cc: * Source/ThirdParty/libwebrtc/Source/webrtc/sdk/objc/components/video_codec/nalu_rewriter.h: * Source/ThirdParty/libwebrtc/Source/webrtc/test/scenario/video_stream.cc: * Source/ThirdParty/libwebrtc/Source/webrtc/test/video_codec_settings.h: (webrtc::test::CodecSettings): * Source/ThirdParty/libwebrtc/Source/webrtc/video/config/video_encoder_config.cc: * Source/ThirdParty/libwebrtc/Source/webrtc/video/config/video_encoder_config.h: * Source/ThirdParty/libwebrtc/Source/webrtc/video/encoder_overshoot_detector.cc: * Source/ThirdParty/libwebrtc/Source/webrtc/video/rtp_video_stream_receiver2.cc: * Source/ThirdParty/libwebrtc/Source/webrtc/video/rtp_video_stream_receiver2.h: * Source/ThirdParty/libwebrtc/Source/webrtc/video/send_statistics_proxy.cc: * Source/ThirdParty/libwebrtc/Source/webrtc/video/video_stream_encoder.cc: * Source/ThirdParty/libwebrtc/libwebrtc.xcodeproj/project.pbxproj: Canonical link: https://commits.webkit.org/267677@main

https://bugs.webkit.org/show_bug.cgi?id=258794 Reviewed by Youenn Fablet. The current WebRTC HEVC is using generic packetization instead of RFC 7789 Packetization. So this is about implementing the RFC 7798 Packetization. Fix HEVC depacketizer issues. (#185) Enalbing low latency mode for RTC (#169) Enable HEVC support. (#165) Fix out-of-bounds write in H265VpsSpsPpsTracker (#163) Apply fix bitstream logic to RtpVideoStreamReceiver2 (#142) Add missing CODEC_H265 switch case (#136) Add HEVC support for iOS/Android (#68) H265 packetization_mode setting fix (#53) Add H.265 QP parsing logic (#47) This patch is extracted from following Open WebRTC Toolkit code changes: <open-webrtc-toolkit/owt-deps-webrtc#185> <open-webrtc-toolkit/owt-deps-webrtc#169> <open-webrtc-toolkit/owt-deps-webrtc#165> <open-webrtc-toolkit/owt-deps-webrtc#163> <open-webrtc-toolkit/owt-deps-webrtc#142> <open-webrtc-toolkit/owt-deps-webrtc#136> <open-webrtc-toolkit/owt-deps-webrtc#68> <open-webrtc-toolkit/owt-deps-webrtc#53> <open-webrtc-toolkit/owt-deps-webrtc#47> co-authoured by: taste1981 <jianlin.qiu@intel.com> jianjunz <jianjun.zhu@intel.com> Cyril Lashkevich <notorca@gmail.com> Piasy <xz4215@gmail.com> ShiJinCheng <874042641@qq.com> Andreas Unterhuber <andreas.unterhuber@keepinmind.info> dong-heun <63987238+dong-heun@users.noreply.github.com> * Source/ThirdParty/libwebrtc/Configurations/libwebrtc.xcconfig: * Source/ThirdParty/libwebrtc/Source/webrtc/api/video/video_codec_type.h: * Source/ThirdParty/libwebrtc/Source/webrtc/api/video_codecs/video_codec.cc: * Source/ThirdParty/libwebrtc/Source/webrtc/api/video_codecs/video_codec.h: (webrtc::VideoCodecH265::operator!= const): * Source/ThirdParty/libwebrtc/Source/webrtc/api/video_codecs/video_decoder_software_fallback_wrapper.cc: * Source/ThirdParty/libwebrtc/Source/webrtc/api/video_codecs/video_encoder.cc: * Source/ThirdParty/libwebrtc/Source/webrtc/api/video_codecs/video_encoder.h: * Source/ThirdParty/libwebrtc/Source/webrtc/call/rtp_payload_params.cc: * Source/ThirdParty/libwebrtc/Source/webrtc/common_video/h265/h265_common.cc: * Source/ThirdParty/libwebrtc/Source/webrtc/common_video/h265/h265_common.h: * Source/ThirdParty/libwebrtc/Source/webrtc/common_video/h265/h265_pps_parser.cc: * Source/ThirdParty/libwebrtc/Source/webrtc/common_video/h265/h265_pps_parser.h: * Source/ThirdParty/libwebrtc/Source/webrtc/common_video/h265/h265_sps_parser.cc: * Source/ThirdParty/libwebrtc/Source/webrtc/common_video/h265/h265_sps_parser.h: * Source/ThirdParty/libwebrtc/Source/webrtc/common_video/h265/h265_vps_parser.cc: * Source/ThirdParty/libwebrtc/Source/webrtc/common_video/h265/h265_vps_parser.h: * Source/ThirdParty/libwebrtc/Source/webrtc/logging/rtc_event_log/rtc_event_log_parser.cc: * Source/ThirdParty/libwebrtc/Source/webrtc/media/base/media_constants.cc: * Source/ThirdParty/libwebrtc/Source/webrtc/media/base/media_constants.h: * Source/ThirdParty/libwebrtc/Source/webrtc/media/engine/internal_decoder_factory.cc: * Source/ThirdParty/libwebrtc/Source/webrtc/modules/rtp_rtcp/source/create_video_rtp_depacketizer.cc: * Source/ThirdParty/libwebrtc/Source/webrtc/modules/rtp_rtcp/source/rtp_format.cc: * Source/ThirdParty/libwebrtc/Source/webrtc/modules/rtp_rtcp/source/rtp_format_h265.cc: * Source/ThirdParty/libwebrtc/Source/webrtc/modules/rtp_rtcp/source/rtp_format_h265.h: (webrtc::RtpPacketizerH265::PacketUnit::PacketUnit): (webrtc::VideoRtpDepacketizerH265::~VideoRtpDepacketizerH265): Deleted. * Source/ThirdParty/libwebrtc/Source/webrtc/modules/rtp_rtcp/source/rtp_sender_video.cc: * Source/ThirdParty/libwebrtc/Source/webrtc/modules/rtp_rtcp/source/rtp_video_header.h: * Source/ThirdParty/libwebrtc/Source/webrtc/modules/video_coding/codecs/h265/include/h265_globals.h: * Source/ThirdParty/libwebrtc/Source/webrtc/modules/video_coding/encoded_frame.cc: * Source/ThirdParty/libwebrtc/Source/webrtc/modules/video_coding/h265_vps_sps_pps_tracker.cc: * Source/ThirdParty/libwebrtc/Source/webrtc/modules/video_coding/h265_vps_sps_pps_tracker.h: * Source/ThirdParty/libwebrtc/Source/webrtc/modules/video_coding/include/video_codec_interface.h: * Source/ThirdParty/libwebrtc/Source/webrtc/modules/video_coding/packet_buffer.cc: * Source/ThirdParty/libwebrtc/Source/webrtc/rtc_base/experiments/min_video_bitrate_experiment.cc: * Source/ThirdParty/libwebrtc/Source/webrtc/sdk/objc/components/video_codec/nalu_rewriter.cc: * Source/ThirdParty/libwebrtc/Source/webrtc/sdk/objc/components/video_codec/nalu_rewriter.h: * Source/ThirdParty/libwebrtc/Source/webrtc/test/scenario/video_stream.cc: * Source/ThirdParty/libwebrtc/Source/webrtc/test/video_codec_settings.h: (webrtc::test::CodecSettings): * Source/ThirdParty/libwebrtc/Source/webrtc/video/config/video_encoder_config.cc: * Source/ThirdParty/libwebrtc/Source/webrtc/video/config/video_encoder_config.h: * Source/ThirdParty/libwebrtc/Source/webrtc/video/encoder_overshoot_detector.cc: * Source/ThirdParty/libwebrtc/Source/webrtc/video/rtp_video_stream_receiver2.cc: * Source/ThirdParty/libwebrtc/Source/webrtc/video/rtp_video_stream_receiver2.h: * Source/ThirdParty/libwebrtc/Source/webrtc/video/send_statistics_proxy.cc: * Source/ThirdParty/libwebrtc/Source/webrtc/video/video_stream_encoder.cc: * Source/ThirdParty/libwebrtc/libwebrtc.xcodeproj/project.pbxproj: Canonical link: https://commits.webkit.org/267677@main

jianjunz and others added 26 commits December 7, 2022 18:02

Enable HEVC support.

0cb2252

This change enables HEVC support after rebasing to M108. HEVC related structures are always declared, but the codec is only implemented when GN arg rtc_use_h265 is enabled (macro WEBRTC_USE_H265 is defined).

Fix unit test build failure on msvc-17

40fd20c

enalbing low latency mode for RTC

9864dc2

Fix clang build failure caused by external VCM (open-webrtc-toolkit#73)

604c6e8

Renable gpra with new interface (open-webrtc-toolkit#74)

5ffdbb4

Fix pacer issue with low latency mode (open-webrtc-toolkit#75)

7398683

Disable frame-dropper for cloudgaming (open-webrtc-toolkit#76)

d4b1f59

Handle RTP extension PictureId. (open-webrtc-toolkit#77)

bf6056e

Fix issue with BWE value proposition (open-webrtc-toolkit#81)

74556ee

Only set marker bit on last slice. (open-webrtc-toolkit#82)

3965c80

Update bwe logic to seperate gpra with gcc. Also allow application se…

0bfe81a

…t lower/upper bounds of bitrate for gpra (open-webrtc-toolkit#83)

Field trials of bitrate control follows field trial convention. (open…

795ecd5

…-webrtc-toolkit#91)

Enable automatic audio playback device switching (open-webrtc-toolkit#92

fe021f4

)

Do not check last_fragment flag for non h.264/h.265 stream (open-webr…

3b8b0de

…tc-toolkit#94)

Set generic frame descriptor extension for HEVC (open-webrtc-toolkit#95)

8110732

Lntf (open-webrtc-toolkit#100)

5dd188c

* Set generic frame descriptor extension for HEVC * Enable more rtcp feedbacks for loss/reference indication

Changes for fake audio on headless platform

4302c52

Link to static gpra library only on Windows. (open-webrtc-toolkit#127)

f79f503

Allow flex-fec support enabling/disabling from application side (open…

be10c3d

…-webrtc-toolkit#128) Enable flex-fec by application instead of hard-code.

Fix gpra crash caused by INTEL_GPRA macro (open-webrtc-toolkit#129)

adbe4e2

Avoid depending on ffmpeg by default (open-webrtc-toolkit#131)

52e6584

Fix build failure on Ubuntu 22.04 (open-webrtc-toolkit#159)

ca294a1

Enable TCAE information population on client stack. (open-webrtc-tool…

05ff903

…kit#160)

Revert "Changes for fake audio on headless platform"

e32b58c

This reverts commit 61912d5. This change introduced too much code in header, and some code doesn't with 108. Revert it and apply it manually later if it's needed.

jianjunz added the enhancement New feature or request label Dec 9, 2022

jianjunz force-pushed the cg branch 2 times, most recently from a3cbef5 to a0eef49 Compare December 9, 2022 14:04

jianjunz marked this pull request as ready for review December 9, 2022 14:11

jianjunz requested a review from taste1981 December 9, 2022 14:12

jianjunz force-pushed the cg branch from a0eef49 to 5b3e694 Compare December 12, 2022 09:56