local video example enhancements#1067
Conversation
ChangesetThe following package versions will be affected by this PR:
|
|
local_video should not be in the changeset since it is not a versioned package configured in knope.toml. |
There was a problem hiding this comment.
Pull request overview
Enhances the examples/local_video publisher/subscriber demos with test-pattern generation, local preview rendering, improved timing/latency overlays, and macOS Metal zero-copy NV12 rendering; also exposes room creation playout-delay options via livekit-api.
Changes:
- Add
RoomClient::create_room_with_playout_delayand use it from the publisher to recreate rooms with explicit subscriber playout delay. - Extend local video examples with SMPTE test pattern publishing, optional local preview window, aspect-constrained viewport resizing, and richer timing overlays.
- Add macOS NV12 CVPixelBuffer → Metal texture import path and shader support for NV12 vs I420 sampling.
Reviewed changes
Copilot reviewed 11 out of 12 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| livekit-api/src/services/room.rs | Refactors room creation request building and adds a playout-delay-specific room creation API. |
| examples/local_video/src/yuv_shader.wgsl | Adds yuv_layout to support NV12 (RG) vs I420 (separate U/V) sampling. |
| examples/local_video/src/viewport_aspect.rs | Introduces an aspect-constrained viewport helper for smoother window resizing. |
| examples/local_video/src/video_display.rs | Adds a local preview app for displaying published I420 frames plus publisher timing HUD. |
| examples/local_video/src/timestamp_burn.rs | Reworks burned-in overlay rendering and adds LatencyDisplay with tests. |
| examples/local_video/src/test_pattern.rs | Adds SMPTE-style 75% color bars generator in I420 format. |
| examples/local_video/src/subscriber.rs | Adds viewport constraint logic, timing overlay updates, jitter buffer logging, and macOS zero-copy NV12 render path. |
| examples/local_video/src/publisher.rs | Adds test-pattern source, room playout-delay recreation, local preview window, and updated capture/timing pipeline. |
| examples/local_video/README.md | Documents new flags and recommends --release for smoother rendering. |
| examples/local_video/Cargo.toml | Adds macOS dependency on metal. |
| Cargo.lock | Locks metal dependency. |
| .changeset/add_room_playout_delay_options.md | Records the API and example enhancement as a patch change. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| if stride_y >= dims.0 { | ||
| queue.write_texture( | ||
| wgpu::TexelCopyTextureInfo { | ||
| texture: &state.y_tex, | ||
| mip_level: 0, | ||
| origin: wgpu::Origin3d::ZERO, | ||
| aspect: wgpu::TextureAspect::All, | ||
| }, | ||
| &state.upload_y, | ||
| data_y, | ||
| wgpu::TexelCopyBufferLayout { | ||
| offset: 0, | ||
| bytes_per_row: Some(upload_row_bytes.0), | ||
| bytes_per_row: Some(stride_y), | ||
| rows_per_image: Some(dims.1), | ||
| }, |
| let source_frame_started_at = Instant::now(); | ||
| let frame_wall_time_us = unix_time_us_now(); | ||
| let mut buffer = create_i420_buffer(width, height, align_buffers_for_display); | ||
| let (stride_y, stride_u, stride_v) = buffer.strides(); | ||
| let (data_y, data_u, data_v) = buffer.data_mut(); |
alan-george-lk
left a comment
alan-george-lk
left a comment
There was a problem hiding this comment.
LGTM: my review is more of a rubber stamp since there's a lot of context I'm missing, but was a good Rust exercise for me
stephen-derosa
left a comment
stephen-derosa
left a comment
There was a problem hiding this comment.
Nice, i appreciate the new additions, they will definitely come in handy!
|
|
||
| Publisher flags (in addition to the common connection flags above): | ||
| - `--camera-index <n>`: Camera index to use (default: `0`). Use `--list-cameras` to see available indices. | ||
| - `--test-pattern`: Generate a standard SMPTE 75% color-bar test pattern instead of capturing from a camera. `--camera-index` is ignored when this is set; `--width`, `--height`, and `--fps` still control the output resolution and frame rate. |
| .or_else(|| env::var("LIVEKIT_API_SECRET").ok()) | ||
| .expect("LIVEKIT_API_SECRET must be provided via --api-secret or env"); | ||
|
|
||
| if args.min_playout_delay.is_some() || args.max_playout_delay.is_some() { |
There was a problem hiding this comment.
suggestion: This can be applied from the token (no need for server API requests); see with_room_config.
There was a problem hiding this comment.
ooh I didn't know that. Do you know if this must be set in the token that triggers room creation? In the publisher example, I explicitly destroy the room then recreate it w/ the options to ensure they are set since it's a room level config.
There was a problem hiding this comment.
I can't seem to find that documented anywhere. For other properties that can be set on the token (such as agent dispatches), it doesn't have to be the token that creates the room. However, this might be different for something like playout delay (i.e., might not get updated after the room is created).
There was a problem hiding this comment.
I opted for creating the room via server API before the publisher joins to ensure it has the right room config.
--test-patternflag to publisher example to generate SMPTE color bars.--display-videoon publisher example to render video being published.