Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add WebRTC (WHIP) output support #7926

Merged
merged 5 commits into from
Jun 10, 2023

Conversation

kc5nra
Copy link
Contributor

@kc5nra kc5nra commented Dec 14, 2022

Co-authored-by: Sean DuBois sean@siobud.com
Co-authored-by: DDRBoxman colin@recursivepenguin.com
Co-authored-by: tt2468 tt2468@gmail.com

First and foremost, thanks to the original author of the draft PR for webrtc Colin Edwards (DDRBoxman).

Description

This PR adds WebRTC support to OBS. WebRTC is an alternative to RTMP that is being adopted by more services. Adding WebRTC support to OBS would benefit OBS and its users greatly. These are some of the highlights. Following this I also have a FAQ for questions we have already received regarding this PR.

This initial version is intentionally simplified to only support output. Later PRs will add source support.

Specifically, this adds a plugin that extends support for:

  • WebRTC Service type with recommended settings for maximum compatibility with webrtc peers
  • WebRTC Output

Motivation and Context

Subsecond Latency

With WebRTC we are able to achieve sub second latency. With our initial measurements we see ~120 Milliseconds from Broadcaster to Playback. We believe we can continue to bring down this number as well.

image

Interactive latency lets broadcasters create new experiences they couldn’t before. OBS users can now create interactive experiences with their viewers. This would allow talk shows and other productions that require conversational latency to use OBS.

Remove FTL from OBS

FTL is a protocol that was created for Mixer. It provided sub-second latency and was used by a few Open Source projects and streaming services. The protocol has been abandoned and it would be nice to remove it from OBS. Adding WebRTC would make that possible since it provides the latency required in addition to the many benefits inherent in the WebRTC stack like encryption, network topology strategies, robust congestion control, etc.

Latest in Video Compression

WebRTC will let OBS users experiment with newer codecs. WebRTC is flexible in what codecs it supports so users could use H265 and AV1.

AV1 users are reporting a 50% reduction in bandwidth required.

Broadcast all angles

WebRTC supports multiple video streams in the same session. OBS could experiment with broadcasting multiple scenes at once! Viewers could then switch between scenes or different views in the same game.

Simulcast

WebRTC allows broadcasters to upload multiple streams of different quality. OBS users could upload ‘high’, ‘med’, and ‘low’ streams themselves. This would reduce the server costs greatly, allowing more

Peer-to-Peer (if you need it)

With WebRTC OBS users could share their video without needing a video server at all! WebRTC can establish a P2P connection so users could send video from OBS directly to their users. No more worrying about setting up special servers to ingest and transcode.

In the future we could add a WebRTC source. Then broadcasters could add WebRTC sources to their scenes. This means low latency collaboration with no backend required!

How Has This Been Tested?

This has currently been tested on Linux and Mac with locally run broadcast-box as well as remotely hosted WHIP endpoints.

Configurations tested:

  • Mac OSX M1 Monterey
  • Debian Sid

Original draft PR: #7192
Closes #7192

Types of changes

  • New feature (non-breaking change which adds functionality)

Checklist:

  • My code has been run through clang-format.
  • I have read the contributing document.
  • My code is not on the master branch.
  • The code has been tested.
  • All commit messages are properly formatted and commits squashed where appropriate.
  • I have included updates to all appropriate documentation.

@kc5nra
Copy link
Contributor Author

kc5nra commented Dec 14, 2022

FAQ

How can I use this today?

Multiple companies provide WebRTC Broadcast/Playback today. Cloudflare and Millicast are two popular options.

Gilmesh has created and is hosting a public WebRTC server at https://b.siobud.com/. You can run this locally for developing and testing as well https://github.com/glimesh/broadcast-box

Why libdatachannel as the WebRTC library?

WebRTC has many libraries that we could have used. The first one we evaluated was Google's implementation known as libwebrtc.
The time required to fetch+download would have been too much of a burden for the OBS code base. These metrics come from a Macbook M1 with a 400 Mb/s connection.

Fetching the source of libwebrtc: 11 minutes 20 seconds
Installing the dependencies: 3 minutes and 30 seconds
Building libwebrtc: 7 minutes
Size added by libwebrtc: 35 megabytes

This caused us to evaluate other options. We arrived at https://github.com/paullouisageneau/libdatachannel and it had everything we needed! It is a C++ WebRTC implementation and uses CMake. It adds very litle impact to the project.

Additional Build time: 9 seconds
Size Added: 11 Megabytes

What is WHIP?

WHIP is the process that WebRTC uses to establish a session between two WebRTC Agents (Client/Server in our case). The actual definition of it can be found at https://www.ietf.org/archive/id/draft-ietf-wish-whip-05.txt

UI/window-basic-settings-stream.cpp Outdated Show resolved Hide resolved
plugins/obs-webrtc/webrtc-output.c Outdated Show resolved Hide resolved
plugins/obs-webrtc/webrtc-service.c Outdated Show resolved Hide resolved
plugins/obs-webrtc/webrtc-services-main.c Outdated Show resolved Hide resolved
plugins/obs-webrtc/CMakeLists.txt Outdated Show resolved Hide resolved
plugins/obs-webrtc/webrtc-services-main.c Outdated Show resolved Hide resolved
plugins/obs-webrtc/webrtc-output.c Outdated Show resolved Hide resolved
plugins/obs-webrtc/webrtc-output.c Outdated Show resolved Hide resolved
@tt2468 tt2468 added Work In Progress Seeking Testers Build artifacts on CI Request for Comments More feedback & discussion is requested New Feature New feature or plugin UI/UX Anything to do with changes or additions to UI/UX elements. labels Dec 14, 2022
phillipmiller0044

This comment was marked as spam.

@phillipmiller0044

This comment was marked as spam.

@tt2468 tt2468 changed the title Add WebRTC output support Add WebRTC (WHIP) output support Dec 15, 2022
.gitignore Outdated Show resolved Hide resolved
Sean-Der and others added 3 commits June 9, 2023 17:03
This adds a new OBS_SERVICE_CONNECT_INFO_BEARER_TOKEN which is needed for WHIP output.

Signed-off-by: pkv <pkv@obsproject.com>
This adds a WHIP output & associated service.
- Code inspiration from DDRBoxman
- Implemented by Sean DuBois & tt2468
- Various fixes and contributions by pkv.

Co-authored-by: tt2468 <tt2468@irltoolkit.com>
Co-authored-by: DDRBoxman <colin@recursivepenguin.com>
Co-authored-by: pkv <pkv@obsproject.com>
Signed-off-by: pkv <pkv@obsproject.com>
This provides the UI glue to enable the WHIP service introduced in the
obs-webrtc plugin.

Co-authored-by: John Bradley <jocbrad@twitch.tv>
Signed-off-by: pkv <pkv@obsproject.com>
Copy link
Member

@DDRBoxman DDRBoxman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

@DDRBoxman DDRBoxman merged commit dd39218 into obsproject:master Jun 10, 2023
@RytoEX
Copy link
Member

RytoEX commented Jun 10, 2023

For some reason, build-aux/modules/50-libdatachannel.json is using libdatachannel paullouisageneau/libdatachannel@f66f281 while obs-deps is using paullouisageneau/libdatachannel@506961b. Is that intentional?

@pkviet
Copy link
Member

pkviet commented Jun 10, 2023

For some reason, build-aux/modules/50-libdatachannel.json is using libdatachannel paullouisageneau/libdatachannel@f66f281 while obs-deps is using paullouisageneau/libdatachannel@506961b. Is that intentional?

That's an oversight
That being said, this is not very consequential. I plan on working on a full msvc compile using Pat's scripts which will allow syncing to last libdatachannel commit on all 3 platforms.

@ma3uk
Copy link

ma3uk commented Jun 10, 2023

It works well, but periodically the image is showered with artifacts (tested with NVENC, QSV and x264). Great job, I'm waiting for support for B-Frames and other codecs

@norihiro
Copy link
Contributor

norihiro commented Jun 10, 2023

After the commit dd39218, obs gets crashed when opening the settings dialog. Steps to reproduce is as below.
OS: Linux

  1. Build with -DENABLE_WEBRTC=OFF.
  2. Start obs. At the first-time wizard, configure with 'Optimize just for recording, I will not be streaming'. Exit obs.
  3. Start obs and open settings.

Log shows a line below and obs crashed immediately.

error: Service 'whip_custom' not found

Backtrace is as below.

#0  obs_service_get_supported_video_codecs (service=0x0) at /home/kamae/repo/obs-studio/libobs/obs-service.c:465
#1  0x00000000007f2a91 in OBSBasicSettings::ResetEncoders(bool) (this=0x7ffe97518168, streamOnly=false)
    at /home/kamae/repo/obs-studio/UI/window-basic-settings-stream.cpp:1552
#2  0x000000000072087a in OBSBasicSettings::LoadOutputSettings() (this=0x0, this@entry=0x7ffe97518168)
    at /home/kamae/repo/obs-studio/UI/window-basic-settings.cpp:2477
#3  0x00000000006f8518 in OBSBasicSettings::LoadSettings(bool) (this=0x7ffe97518168, changedOnly=false)
    at /home/kamae/repo/obs-studio/UI/window-basic-settings.cpp:3315

@Sean-Der
Copy link
Contributor

Sorry about that @norihiro i will have that fixed today!

@RytoEX
Copy link
Member

RytoEX commented Jun 10, 2023

That's an oversight
That being said, this is not very consequential.

In general, we prefer that the deps are the same version across all platforms whenever possible so that we aren't guessing at behavior differences. This can be done separately as a one-line change.

paullouisageneau/libdatachannel@v0.19.0-alpha.1...v0.19.0-alpha.3

@pkviet
Copy link
Member

pkviet commented Jun 10, 2023

That's an oversight
That being said, this is not very consequential.

In general, we prefer that the deps are the same version across all platforms whenever possible so that we aren't guessing at behavior differences. This can be done separately as a one-line change.

paullouisageneau/libdatachannel@v0.19.0-alpha.1...v0.19.0-alpha.3

Yeah, i was just suggesting that i'd make a new change to the commit which is checked out, before the first beta, in order to remove the current libdatachannel patch so maybe it's not necessary to multiply the PRs ? Your call

@RytoEX
Copy link
Member

RytoEX commented Jun 10, 2023

That's an oversight
That being said, this is not very consequential.

In general, we prefer that the deps are the same version across all platforms whenever possible so that we aren't guessing at behavior differences. This can be done separately as a one-line change.
paullouisageneau/libdatachannel@v0.19.0-alpha.1...v0.19.0-alpha.3

Yeah, i was just suggesting that i'd make a new change to the commit which is checked out, before the first beta, in order to remove the current libdatachannel patch so maybe it's not necessary to multiply the PRs ? Your call

Personally, I'd prefer the deps were aligned sooner rather than later. Less time for people to run into obscure issues. Want CI builds to be in the best shape we can manage, and a one-line PR is simple enough to review.

@fairbairn
Copy link

Would you mind sharing your OBS settings used to achieve this 120ms glass to glass latency? We've set up OBS to aggressively do the same, similar setup (we think), but cannot get under 200ms (localhost, with broadcast-box). We'd like to replicate your setup for our experiments.

@Sean-Der
Copy link
Contributor

@fairbairn What encoder (and settings) are you using?

I saw this latency when I did a tune of zerolatency with x264

@fairbairn
Copy link

fairbairn commented Jun 16, 2023

x264, main, 5000K, CBR, 1080p, 30fps, ultrafast, zerolatency

did you use main or baseline?

also, were you streaming audio as well, or just video?

@fairbairn
Copy link

@Sean-Der

We have been unable to replicate this latency, even locally while transmitting through broadcast-box to remove all network influences.

We're unable to get below 210-260ms in our various tests, and we always tune for low latency on the video encoder by default (x264 as well as NVIDIA).

Can we get the precise setup you used (audio and video) from your test where you achieved 120ms so that we may reproduce in our own lab?

In our tests, we're accompanying the video with audio, which may have an influence, but we'd very much like to reproduce your baseline before we tune from there.

Thank you.

@Sean-Der
Copy link
Contributor

Hi @SetoKaiba I have a build of Simulcast available here

I am actively working on it so will have bugs, thank you!

@Sean-Der
Copy link
Contributor

@fairbairn

Sorry I missed this! It might be better if you reach out on OBS discord. I get a lot of github notifications, easy to miss things :/

  • M1 Mac
  • x264 enc
  • tune=zerolatency
  • Run Broadcast Box on localhost

@RytoEX RytoEX added this to the OBS Studio (Next Release) milestone Aug 10, 2023
winlinvip added a commit to ossrs/srs that referenced this pull request Sep 21, 2023
RFC for WHIP: https://datatracker.ietf.org/doc/draft-ietf-wish-whip/

RFC for WHEP: https://datatracker.ietf.org/doc/draft-murillo-whep/

Please note that SRS 5.0 already had WHIP support. I didn't write a
document about WHIP, because WHIP is not a RFC right now, but there are
clues in
[srs-unity](https://github.com/ossrs/srs-unity#usage-publisher). SRS
WHIP url for publisher:
`http://localhost:1985/rtc/v1/whip/?app=live&stream=livestream`

This PR is for WHEP, the url for player is
`http://localhost:1985/rtc/v1/whep/?app=live&stream=livestream`

PS: There is a great PR for OBS to have WHIP support, see
obsproject/obs-studio#7926 and #3581

PS: WHIP for FFmpeg ossrs/ffmpeg-webrtc#1

See #3170


---------

Co-authored-by: Haibo Chen <495810242@qq.com>
Co-authored-by: john <hondaxiao@tencent.com>
Co-authored-by: ChenGH <chengh_math@126.com>
winlinvip added a commit to ossrs/srs that referenced this pull request Sep 21, 2023
RFC for WHIP: https://datatracker.ietf.org/doc/draft-ietf-wish-whip/

RFC for WHEP: https://datatracker.ietf.org/doc/draft-murillo-whep/

Please note that SRS 5.0 already had WHIP support. I didn't write a
document about WHIP, because WHIP is not a RFC right now, but there are
clues in
[srs-unity](https://github.com/ossrs/srs-unity#usage-publisher). SRS
WHIP url for publisher:
`http://localhost:1985/rtc/v1/whip/?app=live&stream=livestream`

This PR is for WHEP, the url for player is
`http://localhost:1985/rtc/v1/whep/?app=live&stream=livestream`

PS: There is a great PR for OBS to have WHIP support, see
obsproject/obs-studio#7926 and #3581

PS: WHIP for FFmpeg ossrs/ffmpeg-webrtc#1

See #3170

---------

Co-authored-by: Haibo Chen <495810242@qq.com>
Co-authored-by: john <hondaxiao@tencent.com>
Co-authored-by: ChenGH <chengh_math@126.com>
duiniuluantanqin added a commit to duiniuluantanqin/srs that referenced this pull request Oct 11, 2023
RFC for WHIP: https://datatracker.ietf.org/doc/draft-ietf-wish-whip/

RFC for WHEP: https://datatracker.ietf.org/doc/draft-murillo-whep/

Please note that SRS 5.0 already had WHIP support. I didn't write a
document about WHIP, because WHIP is not a RFC right now, but there are
clues in
[srs-unity](https://github.com/ossrs/srs-unity#usage-publisher). SRS
WHIP url for publisher:
`http://localhost:1985/rtc/v1/whip/?app=live&stream=livestream`

This PR is for WHEP, the url for player is
`http://localhost:1985/rtc/v1/whep/?app=live&stream=livestream`

PS: There is a great PR for OBS to have WHIP support, see
obsproject/obs-studio#7926 and ossrs#3581

PS: WHIP for FFmpeg ossrs/ffmpeg-webrtc#1

See ossrs#3170

---------

Co-authored-by: Haibo Chen <495810242@qq.com>
Co-authored-by: john <hondaxiao@tencent.com>
Co-authored-by: ChenGH <chengh_math@126.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
New Feature New feature or plugin Request for Comments More feedback & discussion is requested Seeking Testers Build artifacts on CI UI/UX Anything to do with changes or additions to UI/UX elements. Work In Progress
Projects
None yet
Development

Successfully merging this pull request may close these issues.