[META] WebTransport or Full WebSocket support / Investigate external libraries over GStreamer for transport protocols

[ehfd]

https://web.dev/i18n/en/webtransport/
https://www.w3.org/TR/webtransport/
https://web.dev/webcodecs/
https://www.w3.org/TR/webaudio/
https://web.dev/media-mse-basics/

WebTransport is a promising new web protocol based on HTTP3. This combined with Media Source Extensions, WebCodecs, and Web Audio API, will enable Selkies to provide equal performance compared to the native UDP protocols used by many high-performance remote desktop protocols such as Parsec. A TURN server will no longer be required to be used, as WebTransport establishes a client-server connection.

This will enable more codecs not restrained to WebRTC specifications, such as H.265 HEVC and AV1, as well as bringing the performance equal to existing desktop client implementations such as Sunshine or Parsec.

However, a WebTransport plugin for GStreamer is not yet ready. Look below.

This feature requires funding for their developers to add. Please reach out if you are interested in funding.

[ehfd]

I am starting to believe that for both WebTransport and WebRTC, using custom Python solutions outside of GStreamer such as https://github.com/aiortc/aioquic or https://github.com/aiortc/aiortc is becoming plausible, since as long as an RTP stream is generated with GStreamer, WebRTC/WebTransport operations can definitely be separated.

This can be done soon instead of waiting for GStreamer and with much less funding.

This is because using GStreamer plugins for transport is slowly showing less flexibility and performance issues, because they are mainly focused on media processing rather than protocols, and browser issues are fixed much slower. Using a dedicated library for WebRTC or WebTransport QUIC will solve transport specific issues more versatilely.

While the core developers ourselves are short of time to do this in the meantime, and we will focus maintaining the GStreamer native webrtcbin/webrtcsink pipelines, people are welcome to set bounties or help out in this way instead of implementing a GStreamer plugin. A substantial bounty may allow us to implement this ourselves.

This will likely solve #7 for once and for all.

[ehfd]

https://www.w3.org/TR/webcodecs-hevc-codec-registration/
HEVC is slowly coming.

[ehfd]

Dependent on:

https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/1571

[ehfd]

https://www.phoronix.com/news/Mozilla-Firefox-114

Firefox now supports WebTransport as of 114. WebGPU is also recently implemented and prototyped as of 116.

Firefox still requires the implementation of the WebCodecs API, but that makes Safari the only major browser engine without WebTransport.

[ehfd]

https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/1261#note_2053233

A hypothesis I have is that Issue #7 might indeed be something from the web browser client.

If true, we need our own audio stack going through MSE using RTCDataChannel or an equivalent WebTransport interface, it would be sensible to use external WebRTC libraries as well as a new signalling server. Since the only thing we can use from GStreamer's webrtcbin would be the video sink.
Solved differently in #7. Left for historical information.

[ehfd]

#7 is fixed. But WebTransport support in web browsers made things very hard for serving in localhost or for self-signed certificates. Good technology but needs waiting.

[ehfd]

There are restricted environments where only Connection: Upgrade to WebSocket from plain HTTP is available (such as banks).
This might require aiohttp as well as python-websockets.

https://developer.mozilla.org/en-US/docs/Web/HTTP/Protocol_upgrade_mechanism#upgrading_to_a_websocket_connection

It might be a good idea to pair WebSocket + WebCodecs, perhaps instead of the widely intended WebTransport + WebCodecs pairing.

WebCodecs is better than MSE or WASM in terms of latency, and it can be used to be paired with WebTransport as well. That way, both WebSocket and WebTransport will use the same media decoder if we implement both.

In conclusion, I think implementing good old WebSockets has its point.

[ehfd]

Things that are relevant for using WebRTC without exposing various ports for the TURN server, perhaps.
(Continuously updated)

https://github.com/amitv87/turn_ws_proxy
https://groups.google.com/g/turn-server-project-rfc5766-turn-server/c/un2u2xsrjLA

[ehfd]

Archive of relevant discussions from #7

@ehfd

Opus + MP4? https://chromestatus.com/feature/5100845653819392

https://web.archive.org/web/20211003152946/https://parsec.app/blog/game-streaming-tech-in-the-browser-with-parsec-5b70d0f359bc

• https://github.com/parsec-cloud/parsec-sdk/blob/master/sdk/parsec.h
• https://github.com/evshiron/parsec-web-client
• https://github.com/robterrell/web-client

@ehfd

https://www.w3.org/2018/12/games-workshop/slides/21-webtransport-webcodecs.pdf

Looks like WebTransport + WebCodecs (in implementation within Firefox) is the modern alternative to WebAssembly or MSE. In progress to be implemented in Firefox.

I’ve been tracking WebTransport closely and prototyping it locally off and on the past year.

@danisla

It’s interesting but not ready for the Selkies Project yet. Right now, it works well in a tightly controlled production environment. However, today it requires a lot of configuration, valid certificates, browser command line arguments and OS exceptions that make working it locally very challenging.

@ehfd

Self-signed certificates look possible but complicated. By the way, is mainly using WebSocket that bad (for a fall-back protocol) compared to WebRTC? - Since it's what Guacamole, NoVNC, etc uses.

https://docs.libp2p.io/concepts/transports/webtransport/

When connecting to a WebSocket server, browsers require the server to present a TLS certificate signed by a trusted CA (certificate authority). Few nodes have such a certificate, which is the reason that WebSocket never saw widespread adoption in the libp2p network. libp2p WebTransport offers a browser API that includes a way to accept the server’s certificate by checking the (SHA-256) hash of the certificate (using the serverCertificateHashes option), even if the certificate is “just” a self-signed certificate. This allows us to connect any browser node to any server node, as long as the browser knows the certificate hash in advance (see WebTransport in libp2p for how WebTransport addresses achieve this).

WebTransport in libp2p
WebTransport multiaddresses are composed of a QUIC multiaddr, followed by /webtransport and a list of multihashes of the node certificates that the server uses.

For instance, for multiaddress /ip4/192.0.2.0/udp/123/quic/webtransport/certhash/, a standard local QUIC connection is defined up until and including /quic. Then, /webtransport/ runs over QUIC. The self-signed certificate hash that the server will use to verify the connection.

The WebTransport CONNECT request is sent to an HTTPS endpoint. libp2p WebTransport server use /.well-known/libp2p-webtransport. For instance, the WebTransport URL of a WebTransport server advertising /ip4/192.0.2.0/udp/1234/quic/webtransport/ would be https://192.0.2.0:1234/.well-known/libp2p-webtransport?type=noise (the ?type=noise refers to the authentication scheme using Noise).

@wanjohiryan

Yeah, i can second what @danisla is saying... local development for webtransport is hectic at the moment especially for anything working on with the browser (server-side quic development is the easiest).

For example, for warp we must generate self-signed certs every 14 days, as well as a fingerprint hash as a workaround to having to execute bash scripts each time we need to refresh our site in order to force Chrome to allow QUIC connections.

Also, there is currently no way to load balance QUIC connections, UDP load balancers are great and all but they take away the QUIC's ability for roaming support (you can change internet connections, e.g mobile data to Wifi, without having to reinitiate the handshake to the server)

QUIC is shiny, promising and all... however we are not there yet

[ehfd]

I have been informed by @zenomt that RTWebSocket is a pretty interesting option for the WebSocket portion of this project.

https://github.com/zenomt/rtwebsocket
https://github.com/zenomt/rtmfp-cpp

https://www.rfc-editor.org/rfc/rfc7016.html

Return flow association (bi-directional information):
https://www.rfc-editor.org/rfc/rfc7016.html#section-2.3.11.1.2

https://www.rfc-editor.org/rfc/rfc7425.html

Return flow association (bi-directional information):
https://www.rfc-editor.org/rfc/rfc7425.html#section-5.3.5

Along with WebTransport, the possible reason to use this is to show frames as soon as they arrive instead of going through internal jitterbuffers which WebRTC has limited control over (but the current approach of having separate audio and video streams should still make the frame display as fast as possible).

[zenomt]

two things to keep in mind about RTWebSocket when looking at the above links:

• RTWebSocket uses the same primary abstraction as RTMFP: unidirectional ordered message-oriented "flows" named by arbitrary binary metadata, where each flow can have an independent priority/precedence (that you can change at any time), each message can have an arbitrary transmission deadline (that you can change at any time), and "return flow association" generalizes bidirectional communication into arbitrarily-complex trees of unidirectional flows
• to fully realize the "Real-Time" benefits of RTWebSocket (or RTMFP), you need to avail yourself of the real-time facilities of priority/precedence and transmission deadlines.

note that you can revise a message's transmission deadline after it's been queued (for example, you can revise the transmission deadlines of previously-queued video messages when you get a new keyframe).

the link to Section 5.3.5 of RFC 7425 above is to an illustration of multiple levels and many more than 2 flows of a bidirectional flow tree.

RFC 7016 describes RTMFP (a UDP-based transport protocol suspiciously similar to, but significantly predating, Quic). RFC 7425 describes how to send RTMP video, audio, data, and RPC messages over RTMFP. the same method can be used to send RTMP messages over RTWebSocket, and the https://github.com/zenomt/rtwebsocket and https://github.com/zenomt/rtmfp-cpp repos have demonstrations of doing that.