Skip to content

moq-rtc: WebRTC (WHIP/WHEP) gateway, both ingest and egress#1528

Merged
kixelated merged 14 commits into
devfrom
claude/magical-hypatia-e2644c
May 30, 2026
Merged

moq-rtc: WebRTC (WHIP/WHEP) gateway, both ingest and egress#1528
kixelated merged 14 commits into
devfrom
claude/magical-hypatia-e2644c

Conversation

@kixelated
Copy link
Copy Markdown
Collaborator

@kixelated kixelated commented May 28, 2026 •
edited
Loading

Summary

New rs/moq-rtc crate bridging WebRTC ↔ MoQ. Library split along two orthogonal axes — HTTP role (server accepts incoming, client dials out) and direction (publish = WHIP, subscribe = WHEP) — so all four combinations compose from the same Session driver and per-codec adapters in codec::Bridge / codec::Track.

Subcommand WebRTC role Direction Status
server publish accept WHIP publishes RTP → MoQ working
client subscribe dial remote WHEP RTP → MoQ working
server subscribe serve WHEP subscriptions MoQ → RTP working
client publish dial remote WHIP MoQ → RTP working

All four work end-to-end for Opus / H.264 / VP8 / VP9.

Why

WebRTC is the de facto contribution and last-mile distribution protocol for browsers, OBS, mobile SDKs, capture tools, and most camera vendors. Today there's no WebRTC bridge into MoQ. moq-rtc lets any conformant WHIP/WHEP peer feed (or pull from) a MoQ relay without shipping a custom MoQ client.

CLI shape

Mirrors moq-cli:

# server publish (WHIP server): accept publishes into MoQ
moq-rtc --relay https://relay.example.com --broadcast foo \
        server --listen 0.0.0.0:8088 publish

# client subscribe (WHEP client): pull from a remote WHEP source
moq-rtc --relay https://relay.example.com --broadcast cam0 \
        client --url https://camera.example.com/whep/cam0 subscribe

# server subscribe (WHEP server): serve a MoQ broadcast over WHEP
moq-rtc --relay https://relay.example.com --broadcast foo \
        server --listen 0.0.0.0:8088 subscribe

# client publish (WHIP client): push a MoQ broadcast to a remote WHIP endpoint
moq-rtc --relay https://relay.example.com --broadcast foo \
        client --url https://twitch.tv/whip publish

Architecture

  • str0m 0.19 for the sans-IO WebRTC stack. axum 0.8 + axum-server for HTTP signaling. reqwest for the client-side dial.
  • Server::publish_router() / Server::subscribe_router() — axum routers.
  • Client::publish(url, ...) / Client::subscribe(url, ...) — reqwest dialers.
  • session::Session drives str0m::Rtc over a UdpSocket. The main loop select!s on socket recv + egress write requests + str0m timeout.
  • ingest::IngestSink is the RTP-in adapter (used by server publish and client subscribe).
  • egress::EgressSource is the RTP-out adapter (used by server subscribe and client publish). Subscribes to the catalog, picks a rendition per MediaAdded, spawns a pump task that feeds the session loop via an mpsc channel.

Codec adapters

  • Opus: passthrough. moq-mux's Opus importer ingests, codec::Track::opus egresses.
  • H.264: ingest uses moq-mux's Avc3 importer so Annex-B input passes through with SPS/PPS lifted into the catalog. Egress handles both storage shapes: avc3 passthrough, or avc1 length-prefix → Annex-B with SPS/PPS prefixed on every keyframe (covered by the moq-mux regression test added in this PR).
  • VP8 / VP9: passthrough both directions.

AV1 / H.265 aren't in str0m 0.19's Codec enum, so they're not negotiated. Tracked as a follow-up.

Cross-package sync

Branch targeting

Targeting dev per CLAUDE.md: new public API surface that should bake before shipping to main.

Test plan

  • cargo clippy -p moq-rtc --all-targets -- -D warnings clean.
  • cargo fmt --check -p moq-rtc clean.
  • cargo test -p moq-rtc: 5/5 bitstream tests pass (H.264 catalog publishing, Opus catalog publishing, VP9 keyframe detection, egress Opus passthrough, egress H.264 avc3 passthrough).
  • cargo doc -p moq-rtc --no-deps clean with RUSTDOCFLAGS="-D warnings".
  • cargo check --workspace --no-default-features and --all-features clean.
  • cargo test -p moq-mux 157/157 pass (new SPS/PPS-on-keyframe tests + existing).
  • Manual smoke tests against real WebRTC peers (OBS WHIP publish, gst-launch WHIP/WHEP, browser WebRTC) — reviewer or follow-up.

Known limitations

  • WebRTC peers expect a keyframe within ~2 s of joining. If the upstream MoQ broadcast uses long GOPs, freshly-connected WHEP subscribers see a black screen until the next natural keyframe. KeyframeRequest events from the peer are logged but not propagated upstream; PLI-to-MoQ back-pressure is a future enhancement.
  • WHIP-client codec list: the offer advertises every codec str0m supports (Opus, H.264, VP8, VP9), not just what's in the catalog. The remote answer picks the intersection. Could be tightened later by restricting str0m's CodecConfig at offer-creation time.

(Written by Claude)

@kixelated kixelated force-pushed the claude/magical-hypatia-e2644c branch from dd9eaeb to 60d3c52 Compare May 28, 2026 20:11
kixelated and others added 4 commits May 28, 2026 15:21
@kixelated @claude
moq-rtc: add WebRTC (WHIP/WHEP) gateway crate
fb7fd1c 
Introduces a new rs/moq-rtc crate that bridges WebRTC and MoQ so any
WHIP-conformant publisher (OBS, browsers, mobile SDKs) can feed a MoQ
relay without a custom MoQ client. The crate is shaped like moq-relay:
a reusable library with an axum-based HTTP layer, plus a thin binary
that dials an upstream relay.

WHIP ingest is wired end-to-end for Opus, H.264, VP8, and VP9; the H.264
path reuses moq-mux's Avc3 importer so Annex-B input passes through with
the catalog SPS/PPS lifted directly from the bitstream. WHEP egress
returns 501 for now (RTP re-packetization is the follow-up).

Targets dev per CLAUDE.md branch targeting: this adds a new public API
surface that should bake before shipping.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@kixelated @claude
moq-rtc: use moq_net::Timestamp after dev's runtime-Timescale change
6a734ae 
#1473 on dev replaced the fixed `moq_mux::container::Timestamp` alias with
`moq_net::Timestamp` so each container keeps its source scale. Rebasing onto
dev surfaced the four call sites in moq-rtc's codec bridges. Same shape,
new path.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@kixelated @claude
moq-rtc: split into server/client roles with publish/subscribe direct…
50d5fa8 
…ions

Refactor the library and binary into the same 2x2 shape moq-cli uses, so
the 4 combinations (server/client x publish/subscribe) compose from a
shared Session driver and per-codec bridges.

Library:
- Gateway -> moq_rtc::server::Server (axum routers).
- New moq_rtc::client::Client (reqwest dialer).
- Lift IngestSink to a top-level moq_rtc::ingest so both
  server::whip and client::whep can construct it.
- Add moq_rtc::egress placeholder for the deferred RTP-out path.
- Session now takes a MediaRole enum (Ingest now, Egress later).

Binary subcommands mirror moq-cli's verb conventions:
  moq-rtc --relay ... --broadcast foo server --listen ... publish
  moq-rtc --relay ... --broadcast foo client --url ... subscribe
  (server subscribe / client publish are wired but return 501 until the
  per-codec re-packetizer lands)

Implements `client subscribe` (WHEP client) along the way. Reuses
IngestSink wholesale; only the SDP exchange direction is new.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@kixelated @claude
moq-rtc: adapt to moq-mux thiserror and OriginConsumer split
c53c00f 
Rebasing onto dev picked up #1495 (moq-mux::Error is thiserror, not
anyhow) and #1434 (OriginConsumer lost announced_broadcast, gained
get_broadcast/announced). Track both in our error enum and binary;
the WHIP-client path is gated on the per-codec re-packetizer anyway,
so first-miss is good enough until that lands.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@kixelated kixelated force-pushed the claude/magical-hypatia-e2644c branch from 60d3c52 to c53c00f Compare May 28, 2026 22:41
@kixelated kixelated changed the title moq-rtc: add WebRTC (WHIP/WHEP) gateway crate moq-rtc: WebRTC (WHIP/WHEP) gateway with client/server x publish/subscribe shape May 28, 2026
kixelated and others added 3 commits May 28, 2026 19:38
@kixelated
Merge remote-tracking branch 'origin/dev' into claude/magical-hypatia…
7fe09c8 
…-e2644c
@kixelated @claude
moq-mux: lock down SPS/PPS-on-keyframe in the H.264 Annex-B exporter
ee1441f 
The avc1 -> Annex-B export path already injects the cached SPS+PPS prefix
ahead of every keyframe (via `frame.keyframe.then(...)` into
`annexb::from_length_prefixed`), and the Consumer marks the first frame
of each group as a keyframe by protocol invariant. The contract works
end-to-end, but there was no test pinning it down.

Add two layers of coverage:

- Unit tests on `from_length_prefixed` for the with-prefix and
  without-prefix paths, plus multi-NAL frames.
- An integration test on `Export` that runs a two-group Legacy-stored avc1
  broadcast through the full Consumer -> Export pipeline and asserts the
  SPS+PPS prefix lands on both group-starting frames (IDR + P-slice, since
  the Consumer treats the first frame of every group as a keyframe).

Surfaces the contract that the egress side of `moq-rtc` will rely on once
the RTP packetizers land.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@kixelated @claude
moq-rtc: implement the RTP packetizer (egress) for all 4 subcommands
4cf2648 
`server subscribe` (WHEP server) and `client publish` (WHIP client) now
work end-to-end. Both source frames from a MoQ broadcast and push them as
RTP via str0m's Frame API; the per-codec adapters live in `codec::Track`
and are the same regardless of HTTP role.

Library:
- `codec::Track`: subscribed moq-mux track normalized to the bitstream
  shape str0m's Frame API expects. Passthrough for Opus / VP8 / VP9 and
  avc3-stored H.264; for avc1 H.264 it length-prefix -> Annex-B and
  prepends the cached SPS+PPS on every keyframe (covered by the moq-mux
  regression test landed in ee1441f).
- `egress::EgressSource`: subscribes to the catalog at session start,
  picks a rendition per `MediaAdded` event, spawns a pump task that
  forwards frames over an mpsc channel into the session loop.
- `egress::dispatch`: drains write requests into `Rtc::writer(mid).write`,
  converting microsecond timestamps into the negotiated codec clock.
- `session::Session::egress(...)` constructor; main loop `select!`s on
  socket recv + write-request rx + timeout.
- WHEP server: real handler. Look up broadcast via `OriginConsumer::get_broadcast`,
  build Rtc, accept offer, spawn egress session.
- WHIP client: real handler. Mint a `SendOnly` offer (str0m default
  CodecConfig), POST to remote, accept answer, spawn egress session.

Tests:
- `egress_opus_passthrough`: ingest then egress, byte-for-byte round-trip
  including timestamp.
- `egress_h264_avc3_passthrough`: ingest then egress, asserts Annex-B
  start codes and SPS/IDR NALs make it through unchanged.

Docs: doc/bin/rtc.md flipped to mark all four subcommands working, plus a
note about WHEP keyframe latency until PLI-to-MoQ back-pressure lands.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@kixelated kixelated changed the title moq-rtc: WebRTC (WHIP/WHEP) gateway with client/server x publish/subscribe shape moq-rtc: WebRTC (WHIP/WHEP) gateway, both ingest and egress May 29, 2026
@kixelated @claude
moq-rtc: restrict offered codecs to what the catalog can source
93404d8 
Previously the WHIP-client offer advertised every codec str0m supports
(Opus, H.264, VP8, VP9), and the WHEP-server answer accepted whatever
codec the peer offered, even if the local catalog had no matching
rendition. If the negotiated codec didn't intersect with the catalog,
`EgressSource::pick_track` returned None, the pump never started, and the
m-line went silent for the lifetime of the session.

Fix it by restricting `RtcConfig` to `source.catalog_codecs()` before the
SDP exchange in both paths. The remote answer (WHIP) and the local
answer (WHEP) now both intersect with what we can actually deliver, so
`MediaAdded` only fires for codecs we can pump.

Bonus: also flip the doc/bin/rtc.md CLI examples for the server-subscribe
and client-publish rows that were still labeled "not yet implemented".

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
kixelated
kixelated commented May 29, 2026
View reviewed changes
Comment thread rs/moq-rtc/bin/moq-rtc.rs Outdated
/// Repeat the flag for multi-homed deployments. Falls back to the
/// kernel-picked address when empty (loopback testing only).
#[arg(long = "ice-candidate", env = "MOQ_RTC_ICE_CANDIDATE", value_delimiter = ',')]
ice_candidates: Vec<SocketAddr>,
Copy link
Copy Markdown
Collaborator Author

@kixelated kixelated May 28, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

str0m shoooould use peer reflexive candidates right, so these are optional?

Maybe call this public-addr instead to advertise candidates and make it optional.

Comment thread rs/moq-rtc/bin/moq-rtc.rs Outdated
.context("failed to load TLS cert/key")?;
axum_server::bind_rustls(bind, config).serve(service).await?;
}
_ => {
Copy link
Copy Markdown
Collaborator Author

@kixelated kixelated May 28, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Error if they're both not None?

@kixelated kixelated force-pushed the claude/magical-hypatia-e2644c branch from 6dc0b04 to 86ef695 Compare May 29, 2026 04:06
kixelated and others added 5 commits May 28, 2026 21:16
@kixelated
Merge remote-tracking branch 'origin/dev' into claude/magical-hypatia…
0293d91 
…-e2644c
@kixelated @claude
libmoq: switch struct literals to Track::new after compress field landed
b823662 
#1531 added `compress: bool` to `moq_net::Track` but two struct literals
in libmoq's lib still listed only `name` and `priority`, so the lib test
build (which is part of `cargo check --all-targets`) errored with
E0063 once #1531 reached this branch via the dev merge.

Swap both to `Track::new(name)` since neither needed a non-default
priority. Same construction the rest of the workspace uses.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@kixelated
Merge remote-tracking branch 'origin/dev' into claude/magical-hypatia…
a973b0f 
…-e2644c
@kixelated @claude
moq-rtc: address PR review
3225e25 
- Rename --ice-candidate (Vec<SocketAddr>) to --public-addr
  (Option<SocketAddr>): str0m already discovers peer-reflexive candidates
  via STUN, so an explicit host candidate is only needed when the gateway
  sits behind a NAT that requires an external address. Drop the multi-arg
  shape; one public address is enough for any realistic deployment.
- Make the TLS arg pairing explicit in serve(): (Some, Some) does TLS,
  (None, None) does plain HTTP, and the mismatched arms bail with a clear
  error. clap's `requires` already gates this at parse time; the explicit
  arm is belt-and-suspenders.
- doc/bin/rtc.md: reflect the new flag name and semantics.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@kixelated @claude
moq-rtc: keep --public-addr as Vec for dual-stack v4/v6
2a096f1 
A single SocketAddr loses the case where the gateway has both an IPv4
and an IPv6 external address. Switch back to Vec<SocketAddr> with
value_delimiter = ',' so a dual-stack deployment can advertise both
candidates in one pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@kixelated kixelated merged commit 24a8cb6 into dev May 30, 2026
1 check passed
@kixelated kixelated deleted the claude/magical-hypatia-e2644c branch May 30, 2026 00:41
kixelated added a commit that referenced this pull request May 30, 2026
@kixelated @claude
moq-rtc: with_publish takes a Producer now (post-dev-merge fixup)
2fb938a 
Picked up #1528 (moq-rtc) and #1531 (compress field on Track) by
merging origin/dev. The new moq-rtc binary still passes
publisher.consume() to with_publish, which used to take an
OriginConsumer but now takes an OriginProducer. Drop the .consume().

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@kixelated