Fix/surface pts shares audio origin - #2181
Conversation
In SURFACE mode VideoEncoder.checkBuffer rebases every frame on firstTimestamp — the PTS of the first frame this encoder produced. AudioEncoder.calculatePts rebases on presentTimeUs, the origin StreamBase hands to every encoder in startSources(). The two timelines therefore start at different moments: audio at the instant the stream started, video at the instant its first frame came out of the encoder. Whatever sits between them — opening the camera, warming up GL — becomes a constant offset, with the picture running ahead of the sound. Measured on a Samsung SM-A065F (Android 15, camera source, built-in mic, RTMP): video ran 470 ms ahead of audio, 18 clap pairs, spread 427-517 ms. Lips move, speech follows. Both TimestampMode branches perform the same rebase, so switching CLOCK/BUFFER makes no difference — verified by measuring both. Rebasing on presentTimeUs is safe here: GlStreamInterface stamps frames through GlTimestamp, which anchors the first frame to TimeUtils and clamps later frames to within one frame of that clock, and StreamBase reads the same TimeUtils to start the encoders. Both sides are on CLOCK_BOOTTIME microseconds, so the subtraction is meaningful and matches what the audio path already does. Non-surface paths are untouched, and the old behaviour still applies when the encoder was started without a shared origin. With the same test the 490 ms cluster disappears; what remains is inside the resolution of a clap measurement.
|
Tested on device. Your version works, and it works without the workaround we had been carrying. Setup: Samsung SM-A065F (Android 15), camera source, built-in mic, RTMP to a local Camera, your patch applied, our own fix disabled: 47 pairs, median −87 ms, spread −143…−53. Unpatched it was +470 ms with the picture ahead of the sound (18 clap pairs, 427–517 ms). So the half-second is gone. The residual ~87 ms in the other direction is not the origin bug — we measured the same figure with our own fix in place, and it matches the delay between the sensor exposing a frame and the GL interface stamping it. Different problem, much smaller, outside the scope of this PR. Screen capture: no regression. 45 pairs, median +93 ms against +57 ms unpatched — both within one to three frames at 30 fps. I built your commit on top of the Thanks for the fix, and for catching the case my patch would have broken — |
|
Unrelated to this PR, but found while measuring the same builds, in case you want to look at both together: #2182. Portrait broadcasts announce the wrong shape. |
2 participants
No description provided.