fix: vision mixer A/V sync and multiview latency#502
Merged
Conversation
The audiomixer used start-time-selection=first while the vision mixer compositor uses start-time-selection=zero. When audio and video pass through separate aggregators (e.g. audio mixer + vision mixer in the same flow), this mismatch causes a permanent A/V offset because the two aggregators pick different time bases for their output. Change audiomixer to start-time-selection=zero to match. This also guards against the same GStreamer 1.26 race condition that motivated the compositor change. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
GStreamer defaults tsdemux latency to 700ms for PCR synchronization. This adds a tsdemux_latency property (default 100ms) to the MPEGTSSRT input block, reducing end-to-end latency in live pipelines while keeping enough margin for PCR-based A/V sync. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Two issues caused the multiview compositor (mv_comp) to add excessive latency: 1. Latency query stacking: the PGM feed from the distribution compositor (mixer) is tee'd into mv_comp. The latency query from mv_comp traversed back through mixer, causing both compositors' latencies to stack. Fixed by suppressing the latency query on the queue_pgm_mv sink pad so mv_comp's peer latency is determined by its direct input paths instead. 2. Overlay stall: the overlay appsrc pushed at ~1fps (clock updates) with PTS=0. The aggregator saw the overlay pad as perpetually stale and waited up to its full deadline on every frame. Fixed by pushing at the multiview framerate (re-pushing the last sample when content hasn't changed) and using do-timestamp=true instead of PTS=0. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The tsdemux "latency" property only affects the reported latency in GStreamer's latency query, not internal buffering. For live MPEG-TS over SRT, 0ms is appropriate since SRT already handles jitter. Add an "Ignore PCR" boolean property (default true) that sets ignore-pcr on tsdemux. For live pipelines where SRT handles clock recovery, PCR-based timestamp adjustments are unnecessary and can interfere with timing. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add a "TS Offset (ms)" property that applies a timestamp offset to all clocksync elements inside whepserversink. A negative value makes the output play out earlier than the pipeline latency dictates. This enables multiview outputs to display with minimal delay (by setting a negative offset that compensates for upstream pipeline latency) while PGM outputs retain full pipeline latency for proper sync. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The ts-offset approach on clocksync elements was ineffective because webrtcsink's internal appsink has sync=true, which re-applies clock synchronization downstream of clocksync — the buffer is released earlier by clocksync but then waits in the appsink until the pipeline latency elapses. Replace with a "Low Latency" boolean property that sets sync=false on the internal appsink elements via deep-element-added. This makes buffers flow through immediately without waiting for the pipeline's negotiated latency. A/V sync is maintained on the receiver side via RTCP sender reports. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…anel Replace the low_latency bool on WHEP output with a ts_offset_ms integer for precise playout timing control. Add SessionThreadConfig to propagate thread priority/affinity to dynamically created WebRTC session pipelines via pad probes. Add a WebRTC stats overlay to the WHEP player (bitrate, jitter buffer, packet loss, A/V sync offset). Clean up stale comments and remove try_elevate_scheduling_privileges code. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Overlay repush was re-pushing the same GstSample, forcing BaseSrc to copy the entire buffer (8 MB at 1080p) on every non-dirty frame due to shared refcount. Store pixel data as Arc<[u8]> and wrap in a fresh buffer per push — only an Arc refcount bump, no pixel copy. Replace sleep(frame_interval) with deadline-based sleep so render/push time doesn't accumulate as drift. Clamp tsdemux_latency i64→i32 to GStreamer's [-1, i32::MAX] range and use saturating_mul for WHEP ts-offset ns conversion. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…overlay timer spin The is_active() guard on SessionThreadConfig was evaluated at block-build time, before populate() is called in start(). The OnceLock was always empty so session thread priority was never installed on WHEP session pipelines. Move the check inside the consumer-pipeline-created callback where it runs after populate(). Also clamp the overlay timer when it falls behind by more than one frame to avoid a catch-up spin burst, and lower latency query log to trace level. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
firsttozeroto match compositor, fixing A/V offset when audio and video pass through separate aggregatorstsdemux_latencyproperty (default 100ms) to MPEGTSSRT input block, down from GStreamer's 700ms defaultTest plan
cargo check— compiles cleanly🤖 Generated with Claude Code