A/V sync regression after dynamic ad break on HLS stream #4589
Comments
joeyparrish
added
component: HLS
|
Still an issue in 4.3.0 |
|
Thanks for the update. HLS issues are the top priority right now, but we are still short-handed and the list of HLS issues is long. If you want to investigate and/or send a PR, that would be most welcome. If not, we appreciate your patience. |
|
Having done some bisecting, these are my findings:
I'll see if I can find out more |
|
Taking over for @theodab. |
|
This content has no EXT-X-PROGRAM-DATE-TIME, which is critical for AV sync. I strongly recommend using EXT-X-PROGRAM-DATE-TIME. I caught one instance of the ads themselves getting badly out of sync. I'm going to analyze the ad segments, and I expect to find that they are misaligned at at least one point. There was an ad in English showing interviews about 66% through the break, which was very helpful in this case. If an ad is not aligned on the right-hand side (audio ends earlier than video or vice-versa), we might need to realign the SourceBuffers after the discontinuity. I'm seeing "physical" evidence of this, too, in the debugger: player.mediaSourceEngine_.sourceBuffers_.audio.timestampOffset -
player.mediaSourceEngine_.sourceBuffers_.video.timestampOffset
2.0372580000000653This number grows over time during ad breaks, and seems like a plausible measurement of desync. This would also make sense with my theory above (that audio ends earlier than video or vice-versa in certain ads). In sequence mode, if an application does not touch timestampOffset during playback, it behaves as an accumulator, showing how much data has actually been appended to the buffers. Here we see different amounts appended to audio and video over time. Eventually, this number is great enough that we start getting warnings from the player: |
joeyparrish
added
priority: P0
|
Good news! I've watched this several times, and my Swedish is improving. 🇸🇪 It appears that desync increases by roughly 50-100 ms per ad. Joining mid-ad-break results in less desync that joining during the main program and sitting through an entire ad break. This content appears to be a loop of the same program and ad content each time. Each ad break seems to add to the desync by a little more than 1 second. I'm now experimenting with a change that resynchronizes the SourceBuffers across discontinuity bounds, so that we don't accumulate errors in |
|
I've watched the loop several more times testing my fix, and I'm certain I heard Ingvar Kamprad mentioned twice on the antiques show. 🫖 🛋️ 🛒 My fix appears to work. The gist is that we track discontinuity sequence numbers for segment references, and resync This resync is only done at these boundaries, though, and not on every segment. This way, if segment durations in the playlist are somewhat inaccurate, we don't introduce sync issues by slightly moving the segments within discontinuity boundaries. Sequence mode is allowed to work as it should up until a discontinuity. |
If server-side ad segments aren't aligned, AV could get out of sync by accumulating errors in the timestampOffset of the SourceBuffers. This fixes the issue by tracking discontinuity boundaries and resetting timestampOffset to theoretical segment start times when a boundary is crossed. Closes shaka-project#4589
|
My fix works, however, after watching several more rounds, there is another source of accumulated error. This time, in the playlist itself. I'm syncing with theoretical segment start times, but those are based on summed segment durations only. What we really need is to resync against @fredrik-telia, can |
|
Thank you so much for your findings, hopefully we can add |
|
I believe the config for the test url has been updated now, might take a few minutes for the change to take effect |
|
I'm not seeing PROGRAM-DATE-TIME in the playlists yet. |
|
Not present when joining main content, and not added after ad discontinuities, either. |
|
I'll go ahead and update the tests affected by the fix I have so far, and get that through review today if I can. That fix should be in an update this week. If I still don't have DATE-TIME in the playlists at that point, I'll start working on a change to use that for continuous sync (instead of one-time startup sync as it stands today), but that change will be purely theoretical until I have content to verify it. This second fix may not make it into this week's updates, and may get pushed out to January while we are on holiday. |
If server-side ad segments aren't aligned, AV could get out of sync by accumulating errors in the timestampOffset of the SourceBuffers. This improves the issue by tracking discontinuity boundaries and resetting timestampOffset to theoretical segment start times when a boundary is crossed. Issue #4589
Discontinuity tracking was broken and test coverage was insufficient to catch this. This fixes the parsing and counting of discontinuities, and replaces two outdated and useless tests with a new one that covers the counting properly. One of the old tests was checking that a timestamp offset was set for each discontinuity, but this had become irrelevant since the test was written. Discontinuities do not have anything to do with timestamp offsets in current versions of Shaka Player. The other old test was checking that after a discontinuity, we didn't fetch a segment to parse out the timestamp, but we stopped doing that entirely in v4. The new test checks that the initial discontinuity sequence number is honored, and that after a discontinuity, the number goes up. It also checks that the correct number is extracted after an update. This test fails without the fix, and passes with it. This bug affected v4.2.6 and v4.3.2 only. Issue #4589
After seeking or adaptation, StreamingEngine will create a new SegmentIterator. However, if the buffered ranges are different enough from the references in the iterator, the lookup in the index can produce a duplicate segment. If this happens when sequence mode is active, StreamingEngine will append the duplicate segment instead of overwriting the end of the original one. Any time a new iterator is created, we force the stream to resync, so that unaligned segments or slightly-inaccurate segments are written to the correct place in the buffer. Issue #4589
Before, tiny segments (~33ms in practice) and overwritten segments (sometimes necessary) would register as encoding problems. This makes the logic more robust and the error logs more reliable. Issue #4589
Make SegmentIndex robust against rounding errors so that we don't end up with duplicate segments on merge. In sequence mode, this would cause a massive AV sync issue of 1 segment duration. Issue #4589
Closes #4589 Backported to v4.2.x: - patched in lowestSyncTime_ from v4.3.x in HlsParser
Closes #4589 Backported to v4.2.x: - patched in lowestSyncTime_ from v4.3.x in HlsParser
|
Perfectly in sync after 45 minutes with the latest fixes and |
|
And I've seen these ad breaks so many times now, I really want to watch "Marko & Irma". |
|
Thanks so much for the great work! |
|
#4589 (comment) We are seeing AV sync issues in 4.3.4 as well. |
|
it is a query parameter on the url for this particular type of stream, it has nothing to do with shaka |
If server-side ad segments aren't aligned, AV could get out of sync by accumulating errors in the timestampOffset of the SourceBuffers. This improves the issue by tracking discontinuity boundaries and resetting timestampOffset to theoretical segment start times when a boundary is crossed. Issue shaka-project#4589
…t#4869) Setting the timestampOffset requires an abort() in some cases on Chromecast. Issue shaka-project#4589
Discontinuity tracking was broken and test coverage was insufficient to catch this. This fixes the parsing and counting of discontinuities, and replaces two outdated and useless tests with a new one that covers the counting properly. One of the old tests was checking that a timestamp offset was set for each discontinuity, but this had become irrelevant since the test was written. Discontinuities do not have anything to do with timestamp offsets in current versions of Shaka Player. The other old test was checking that after a discontinuity, we didn't fetch a segment to parse out the timestamp, but we stopped doing that entirely in v4. The new test checks that the initial discontinuity sequence number is honored, and that after a discontinuity, the number goes up. It also checks that the correct number is extracted after an update. This test fails without the fix, and passes with it. This bug affected v4.2.6 and v4.3.2 only. Issue shaka-project#4589
…ject#4886) After seeking or adaptation, StreamingEngine will create a new SegmentIterator. However, if the buffered ranges are different enough from the references in the iterator, the lookup in the index can produce a duplicate segment. If this happens when sequence mode is active, StreamingEngine will append the duplicate segment instead of overwriting the end of the original one. Any time a new iterator is created, we force the stream to resync, so that unaligned segments or slightly-inaccurate segments are written to the correct place in the buffer. Issue shaka-project#4589
Before, tiny segments (~33ms in practice) and overwritten segments (sometimes necessary) would register as encoding problems. This makes the logic more robust and the error logs more reliable. Issue shaka-project#4589
…4884) Make SegmentIndex robust against rounding errors so that we don't end up with duplicate segments on merge. In sequence mode, this would cause a massive AV sync issue of 1 segment duration. Issue shaka-project#4589
github-actions
bot
added
the
status: archived
|
@fredrik-telia Can you review if the issue persist in the main branch even with the revert? Thanks! |
avelad
added
the
status: waiting on response
|
Closing due to inactivity. If this is still an issue for you or if you have further questions, the OP can ask shaka-bot to reopen it by including |
shaka-bot
removed
the
status: waiting on response
Have you read the FAQ and checked for duplicate open issues?
yes
What version of Shaka Player are you using?
nightly
Can you reproduce the issue with our latest release version?
yes, 4.2.2
Can you reproduce the issue with the latest code from
main?yes
Are you using the demo app or your own custom app?
demo app
If custom app, can you reproduce the issue using our demo app?
What browser and OS are you using?
Chrome, macOS
For embedded devices (smart TVs, etc.), what model and firmware version are you using?
What are the manifest and license server URIs?
Will send in separate email
What configuration are you using? What is the output of
player.getConfiguration()?default demo app config
What did you do?
Add custom content, play, wait until ad break is over (there will be a timestamp on screen when the "real" program is playing)
Observe A/V sync
What did you expect to happen?
A/V to be in sync after ad break
What actually happened?
Audio and video are out of sync, and will get more out of sync after successive ad breaks.
We were hoping the issues would be solved by #4308 but unfortunately this doesn't seem to be the case.
Using an older version ( https://v3-3-11-dot-shaka-player-demo.appspot.com/demo/ ) A/V are still in sync after the ad break.
Using the nightly version ( https://nightly-dot-shaka-player-demo.appspot.com/demo/ ) A/V are out of sync after the ad break.
The text was updated successfully, but these errors were encountered: