fancy scrub bar UI #32
Labels
Milestone
Comments
Closed
scottlamb
added a commit
that referenced
this issue
Jun 9, 2020
This is useful for a combo scrub bar-based UI (#32) + live view UI (#59) in a non-obvious way. When constructing a HTML Media Source Extensions API SourceBuffer, the caller can specify a "mode" of either "segments" or "sequence": In "sequence" mode, playback assumes segments are added sequentially. This is good enough for a live view-only UI (#59) but not for a scrub bar UI in which you may want to seek backward to a segment you've never seen before. You will then need to insert a segment out-of-sequence. Imagine what happens when the user goes forward again until the end of the segment inserted immediately before it. The user should see the chronologically next segment or a pause for loading if it's unavailable. The best approximation of this is to track the mapping of timestamps to segments and insert a VTTCue with an enter/exit handler that seeks to the right position. But seeking isn't instantaneous; the user will likely briefly see first the segment they seeked to before. That's janky. Additionally, the "canplaythrough" event will behave strangely. In "segments" mode, playback respects the timestamps we set: * The obvious choice is to use wall clock timestamps. This is fine if they're known to be fixed and correct. They're not. The currently-recording segment may be "unanchored", meaning its start timestamp is not yet fixed. Older timestamps may overlap if the system clock was stepped between runs. The latter isn't /too/ bad from a user perspective, though it's confusing as a developer. We probably will only end up showing the more recent recording for a given timestamp anyway. But the former is quite annoying. It means we have to throw away part of the SourceBuffer that we may want to seek back (causing UI pauses when that happens) or keep our own spare copy of it (memory bloat). I'd like to avoid the whole mess. * Another approach is to use timestamps that are guaranteed to be in the correct order but that may have gaps. In particular, a timestamp of (recording_id * max_recording_duration) + time_within_recording. But again seeking isn't instantaneous. In my experiments, there's a visible pause between segments that drives me nuts. * Finally, the approach that led me to this schema change. Use timestamps that place each segment after the one before, possibly with an intentional gap between runs (to force a wait where we have an actual gap). This should make the browser's natural playback behavior work properly: it never goes to an incorrect place, and it only waits when/if we want it to. We have to maintain a mapping between its timestamps and segment ids but that's doable. This commit is only the schema change; the new data aren't exposed in the API yet, much less used by a UI. Note that stream.next_recording_id became stream.cum_recordings. I made a slight definition change in the process: recording ids for new streams start at 0 rather than 1. Various tests changed accordingly. The upgrade process makes a best effort to backfill these new fields, but of course it doesn't know the total duration or number of runs of previously deleted rows. That's good enough.
Closed
|
Assuming we only start loading in the mp4 segments on scrubbar release, I think progressive scrubbing (display the scrub thumbnail in the video window) would also be a great feature. |
scottlamb
added
javascript
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I've written the server-side code to support segmented
.mp4files for HTML5 Media Source Extensions. (design/api.md talks about this.) It may have some bugs but is essentially complete.The goal was that to use this to create the classic NVR UI where there's a scrub bar at the bottom showing an entire day, it grows over time as new stuff is recorded, and none of this has to wait for huge
moovsegments to transfer.I think this needs some significant work to be viable, though:
debug the current segmentedI think as part of getting the live view working I got this stuff all figured out. See 478323e..mp4files. I noticed some of them were working, some weren't. I'm not sure what was different. Maybe related to strip unnecessary non-VCL NALs from sample data #43.but this is Chrome-only last I checked and doesn't support cancelling requests, so if you seek around a lot your connection will get quite congested while the system catches up with you.fetch is well-supported by browsers now, including streams and cancellation.But it requires some additional server-side work. hyper annoyingly doesn't support sharing the HTTP port with a WebSockets implementation. Could use a separate port for now.The live view uses WebSockets now. But given that fetch works well now, I don't think we need to add a WebSockets API for existing segments.The text was updated successfully, but these errors were encountered: