Hls.Events.ERROR is never received unless HTMLVideoElement is attached to the DOM #5759
Comments
bes
added
Bug
Needs Triage
|
Unfortunately I can't share the stream. |
|
So the element is attached to the player, but not the DOM? Have you looked for an escape in gap-controller |
|
Thanks, I will try GapController. Is an unattached video element otherwise unsupported? |
|
I was able to play from a disconnected media element and throttle the network to produce a stall that triggered: hls.js/src/controller/gap-controller.ts Line 155 in 68c6046 |
|
Is there some way to share the stream privately with you? EDIT: GitHub app didn't show your above message until it was too late. Sorry. |
video-dev Slack https://www.video-dev.org |
|
@robwalch maybe I was too slow to click the link? When I click "Join the Slack community" it says "This link is no longer active To join this workspace, you’ll need to ask the person who originally invited you for a new link." |
|
@bes - Please give it another try now. Thanks. |
I'm a little confused by the description. If the As far as I can tell this issue (not reporting stall errors) is a result of the stall check exiting early here: hls.js/src/controller/gap-controller.ts Lines 94 to 100 in 164100c This may be easier to reproduce with the element removed from the DOM as the video buffer is likely ignored and stalls only occur when reaching the end of the audio buffer which is more than not ahead of the video buffer. |
|
Ah I see you are copying the video to a texture with OpenGL... Unfortunately it looks like the browser will ignore the video buffer and continue playing through audio when the element is not attached to the DOM. You might want to try attaching the element to the DOM but hiding it with CSS in one of several ways ( There is still an issue with stall check exiting early when (audio) playback actually does stall. I have a fix for that (#5779). For detecting playback getting ahead of the video buffer without playback stalling ( |
|
Thanks @robwalch I tried attaching the video element both with It only starts working once I scroll the video element into view 😭 (Chrome) So I guess this is a browser issue. Do you think it would be worth it to report this to Chrome? It doesn't stall at all in Firefox or Safari. My thinking is that Chrome should treat copying of video frames to OpenGL in the same way as an attached and visible video element. |
|
You should file an issue, but for now I suggest you implement you own tracking of currentTime in the buffered ranges. The branch I shared may help. You can do this using only video.currentTime, video.buffered.end(n), and the timeupdate event. |
|
So I did like this to nudge the videoElement forward after 2 seconds of the video buffer stalling. private lastSmallestEnd = 0;
private checkVideoBufferedRanges = () => {
if (this.videoElement === null) {
return;
}
if (this.hlsJs === undefined) {
return;
}
const currentTime = this.videoElement.currentTime;
let smallestEnd = Number.MAX_SAFE_INTEGER;
try {
// This is an array of the buffered ranges of all sources [video, audio].
// This call can throw
const timeRanges = this.videoElement.buffered;
if (timeRanges.length > 0) {
for (let i = 0; i < timeRanges.length; i++) {
const end = timeRanges.end(i);
console.log("Index and end", { i, end });
if (end < smallestEnd) {
smallestEnd = end;
}
}
}
} catch {
// Noop
}
if (this.lastSmallestEnd !== smallestEnd && this.lastSmallestEnd !== 0) {
this.lastSmallestEnd = 0;
}
if (currentTime > smallestEnd + 2) {
if (this.lastSmallestEnd !== smallestEnd) {
console.log("Nudging!");
this.setTime(currentTime);
this.lastSmallestEnd = smallestEnd;
}
}
};But I am surprised that after nudging, and seeing the video start moving again, the Log: |
|
I thought the issue was that playback was advancing past the end of the video buffer like this:
So why do you want to nudge playback? I would either pause playback so that audio stops until video frames are available (the user experience will be audio and video stalling together), or let audio playback continue as it does now, but track that video is stalled and frames are being dropped, and not copy from the video element since there is nothing new to copy. |
|
I can't tell you what is happening, I can only see the symptoms:
The graphs available at https://hlsjs.video-dev.org/demo/ are they available in the HLS.js package somewhere, so that I can just use them to get a feeling for what is happening? |
|
Also, if I add a 1x1px video on the screen, visible, everything "just works", which is irritating, but will probably have to do. EDIT: This adds about 10% percentage points of load to the page when running. |
There is a video buffer gap between the first and second segment. Related issues:
|
|
@robwalch Do you have any publicly available examples of stalling video (but audio continues), that I can use when creating a proof-of-concept bug report for Chrome? I want to illustrate the difference between using a video element & using opengl canvas, but I don't have any public videos available for that. |
|
Wrote an issue directed at Chrome https://bugs.chromium.org/p/chromium/issues/detail?id=1478681 |
|
Update: I got an answer from a Chromium developer here: https://bugs.chromium.org/p/chromium/issues/detail?id=1478681#c6 Would there be a chance of HLS.js getting the option of enabling such a buffer checking, so that everyone doesn't have to reimplement it? And the follow-up question being; why isn't the |
The buffered TimeRange end advances only after audio and video are loaded and appended to the MediaSource audio and video SourceBuffers. If you have a gap in either buffer, you have more than one time range, and it is the last time range which is extended after each append.
Some players accept a "min buffer time" or "buffering goal" and stop playback (set rate to 0 or pause) when reaching the buffered end, and only resume when buffer "has enough" or reaches the min/goal. A feature request and PR contributing this functionality would be accepted as long as it is optional, not on by default (for backwards compatibility), well documented, and aligns with existing player API (HLS.js and similar libraries). |
|
Thanks, I'll see what I can manage. You say:
But as I wrote in #5759 (comment) -
This is extremely surprising to me, super counter-intuitive. If possible, can you explain why am I seeing that behavior? Like, I can't build anything that "gets me out of that situation" if the buffer information can't be relied on. I'm at a loss. |
It's buffering at 11.9 -> 15.9. There is a gap in the buffer starting at 3.9 which will always be reproduced when loading and appending the same video. Why? Likely an issue with misaligned or overlapping video. See related issue #5743. To continue playback, you need to seek over (or nudge past) the gap. |
|
Yes, and when I do that, buffer[0].end is still stuck at Maybe we're saying the same thing, but I don't get why buffer[0] (which I assume is video) is stuck at Are we saying the same thing? Am I being thick? |
|
Thank you for our slack chat @robwalch , it was very helpful. I just wanted to update my latest understanding here: The different indexes of It doesn't seem that I can detect this stall using |
|
With the video removed from the DOM or display: none, Chrome does not pick up the second video TimeRange after the gap for decoding, unless a seek is performed. There are two video TimeRanges because of a gap in the video buffer between the first and second segment append. When attached to the DOM and displayed, currentTime advances past the gap while audio is played but video is frozen for 1-2 seconds. Eventually currentTime stalls and audio stops waiting for that gap to filled (this shows up as As we have three issues with similar gap issues I'd like to find a solution that removes the gap altogether, but I don't see a path forward when we are passing media to MSE as provided by the source asset. I event tried putting the video SourceBuffer in "sequence" mode and still got a gap, so I'm inclined to think this is unavoidable with certain segments, but I can't tell you why. The media panel offers no clues why an append produces a gap (removing frames?) when it was made at the end of last append. I can look into having the gap-controller proactively step over video gaps. My concern is that could produce skips in audio and introduces seek operations on platforms perfectly capable of transitioning over these gaps without the same late demuxer underflow that only Chrome seems to exhibit. |
robwalch
added
browser: Chrome
Stream Issue
and removed
Needs Triage
|
The cause of the video buffer gap is overlapping decode timestamps between the first and second segments:
HLS.js does not shift the samples to prevent overlapping appends when the next segments presentation time is greater than the expected start decode time. If we make an exception for chrome the gap goes away. Here's the code in question: hls.js/src/remux/mp4-remuxer.ts Lines 514 to 528 in d0d5204 |
|
Unfortunately, adjusting the first sample of the second segment doesn't work for all related issues. It prevented media from being appended in one, so this is not a fix I think we should make. These gaps are caused by samples at the end of the first segment overlapping with the start of the first. |
|
@robwalch I tried it in my WebGL setup (pulled & built your branch locally), and it works! Big thanks! |
What version of Hls.js are you using?
1.4.10
What browser (including version) are you using?
Chrome 116.0.5845.96 (Official Build) (arm64)
What OS (including version) are you using?
macOS 13.5.1 (22G90)
Test stream
No response
Configuration
Additional player setup steps
Standard setup, but the
HTMLVideoElementis never attached to the DOMChecklist
Steps to reproduce
HTMLVideoElementmanually:document.createElement("video")hlsJs.on(Hls.Events.ERROR, ...Expected behaviour
Hls.Events.ERRORto be received.See previous discussion: #5758
What actually happened?
Hls.Events.ERRORis never receivedIf I attach the
HTMLVideoElementto the DOM, it will throw anHls.Events.ERRORafter a few seconds.Console output
Chrome media internals output
No response
The text was updated successfully, but these errors were encountered: