Guidance on streaming and playing multiple WebM files seamlessly

[Ajaay]

I'm looking for a little guidance if possible.

I have a number of webm files stored in S3, and I'd like to stream and play these files seamlessly one after another. Is this possible using Shaka?

I'm having problems getting this working out of the box in Chrome using purely MSE, and Firefox doesn't appear to support the codecs at all (when using MSE).

Would I need to do any transcoding for this to work?

Any tips would be greatly appreciated!

Thanks!

[joeyparrish]

Since you didn't say DASH, I'm assuming your WebM content is just single WebM files, not adaptive bitrate DASH-packaged WebM. Correct me if I'm wrong.

No matter what you do, you'll need to package the WebM files to make them MSE-compatible. A DASH packager such as Shaka Packager will do this for you. You don't have to re-encode to multiple bitrates if you don't care about ABR.

I see the following options:

1. Package the WebM files and generate DASH manifests for them. Combine those manifests into one using multiple DASH Periods to stitch them together. Shaka should play this seamlessly. If you don't know in advance the sequence you want, then this might not be the solution for you.
2. Write your own MSE player. You can decide at runtime what file you want to play next. You would want to set timestampOffset in SourceBuffer to the duration so far, then append the next file. The offset will be added to the timestamps in the content. If you don't want to work with MSE directly, then this might not be the solution for you.
3. Write a custom manifest parser for Shaka Player. This is an approach somewhere in the middle. You would need to know the byte ranges for the init and index segments. From that, you could construct the segment index. Your parser would then tell Shaka that there's a period with one stream set, which contains one stream. That stream would contain callbacks to read the segment index. You could copy this from our DASH parser. You could decide at runtime in the client which clip was next and update the manifest to add a new period for that clip.
4. Write server-side code to generate a multi-period DASH manifest. If you had already generated individual manifests for each clip, it would be fairly easy to have server-side logic to generate a multi-period manifest on the fly for any given sequence. It's just parsing and combining XML documents. You could update this sequence at any time, even based on signals from the client. For example, play clip one with http://foo.bar/manifest?1, but then update the manifest URL later to http://foo.bar/manifest?1,72 to play clip 72 after 1, and so on. Shaka Player would grow the presentation as new clips were added to the manifest, and the manifest itself would signal how often to update. Client-side code (Shaka network request filter) can change the manifest URL arbitrarily to add clips to the request parameter.

I know that's a ton of info, and I'm guessing a lot of things about your purpose and content, so this may or may not be that helpful. Please let me know if any of this was on target or if I can help answer any more specific questions.

[Ajaay]

Wow thanks for taking the time to write such a detailed, informative response! Really appreciate it.

You are right that these are purely single WebM files. To give you the full context, these are essentially generated by Webcam/Mic recordings using the MediaRecorder API, and saved to S3 in chunks at set intervals (say 5 seconds), all in browser. I expect recordings of up to an hour, so I save in chunks in an effort to avoid browser memory issues.

With regards to Option 1, I would know the total number of files and the sequence I'd want to play them in (numerical order). Would that be ok?

As for Option 2, if I could get this to work it would be the most ideal because it's all done in browser which really simplifies things. I struggled to get the timestampOffset feature to work, it just returned an error for me. However came across the 'sequence' mode option which gets the browser to adjust the timestamp automatically based on the order it's passed the segments. This partially worked in the sense that it played all of the chunks, but unfortunately the audio and video are completely muddled! And then there's the issue of Firefox where according to the specs WebM playback via MSE is only supported when hardware H.264 encoders are not available....Not really sure what that means and why.

Option 3 and 4 seem a little out of my league to be completely honest.

Another option I've noticed is that AWS offer a transcoder service. Perhaps I could feed my S3 bucket as an input and have it spit out MPEG-DASH for me....

Thanks again.

[joeyparrish]

I am not aware of any restriction on WebM in Firefox. On Linux, where I have no hardware decoder, I can play both WebM and h264 on Firefox.

Note that only Chrome, Firefox, and Opera will play WebM as far as I know. (It's been a while since I tested this.)

If you feed a single WebM file from your recorder to MediaSource, can you play it? If so, you won't need to package or remux them, which is good news.

If the files are small (a few seconds), then a fake manifest parser might be the best option. You can describe each file as a single period, each period has a single stream set, each set has a single stream, each stream has a single segment (the whole file) and no init segment.

[Ajaay]

I am able to play the WebM in Firefox just by opening the file URL, but when you try and play through MSE is when it fails. Here's the link:

https://developer.mozilla.org/en-US/docs/Web/HTML/Supported_media_formats

And the comment:

[35] VP8/VP9 video codecs are only available in MSE when H.264 hardware decoders are not available. Checking for availability via MediaSource.isTypeSupported() is recommended.

I don't get any helpful console errors in vanilla Firefox but in Nightly I get the following:

Cannot play media. No decoders for requested formats: video/webm; codecs="vorbis vp8"

So it doesn't look good for FF unless I'm missing something.

As for Chrome yes I can play a single file no problem through MSE. The files are indeed small, currently 5 seconds but I can choose whatever I want the length to be. How would I go about creating the fake manifest parser?

Thanks

[sandersaares]

Combine those manifests into one using multiple DASH Periods to stitch them together. Shaka should play this seamlessly.

I know that dash.js rebuilds the media pipeline between periods, as it was found that it is not really possibly to know whether the sample stream in one period is directly compatible to the next. This causes a visible pause/glitch in playback. Are you saying that Shaka Player is able to do gapless multi-period playback? I am curious how you solved the problem of the codec parameters being different between periods, on a level that may not be obvious from the manifest.

[joeyparrish]

@Ajaay, I noticed two codecs in your type. Perhaps the issue is that your content is multiplexed. Try MediaSource.isTypeSupported('video/webm; codecs="vp8"'); That will show you whether or not WebM is supported by Firefox when it's not multiplexed. Also try MediaSource.isTypeSupported('audio/webm; codecs="vorbis"');

@sandersaares, we achieve seamless multi period playback by requiring that codecs be the same across periods. Once we choose AdaptationSets for the first period, we filter out incompatible streams from the entire manifest. We believe it would not make sense to have multicodec content where different periods use a different set of codecs, even though it is allowed by the DASH spec.

[Ajaay]

@joeyparrish , vp8 returns false and so does vp9. Whereas vorbis returns true. So is it looking like streaming any video in Firefox is a no go at the moment? Or would I need to transcode into something else first?

Presumably streaming in Firefox using Shaka is a relatively common requirement?

[joeyparrish]

That's very interesting. I see vp8 and vp9 supported on the machines available to me. What version of Firefox are you running, and on what platform?

[Ajaay]

49.0.1 on OS X

[joeyparrish]

Interesting. I'm running the same. @cpearce, any suggestions? It seems like Firefox should support vp8 and vp9 on OS X, or at least it does for me. But it does not for @Ajaay. Do I have certain flags enabled that I have forgotten about?

[cpearce]

@joeyparrish we ony enable WebM in MSE if the client machine passes our "benchmark" test, that is, if we judge the machine can adequately decode 720p VP9 in software "fast enough". I'd expect @Ajaay's machine has not passed our benchmark for some reason.

WebM can be force enabled by setting media.mediasource.webm.enabled=true in about:config.

Once we get hardware decoding for VP9, I expect we'll remove the benchmark.

[joeyparrish]

Thanks, @cpearce. @Ajaay, you can try the webm setting in about:config to unblock you on development.

I suggest using a fake manifest parser plugin for Shaka Player. Each clip can be its own 1-segment period. If that is too complicated, you can always write your own MediaSource app. I recommend strongly against using sequence mode, though.

[Ajaay]

I'm running a new Macbook Pro so a little surprised it's failed the benchmark test... However the fact it exists means I'll stick with Chrome playback for anything production for now. Good to know - thanks.

In terms of a fake manifest parser we're essentially talking about writing my own plugin as per the link below right?

https://shaka-player-demo.appspot.com/docs/api/tutorial-plugins.html

[joeyparrish]

Yes, exactly. There's an interface called ManifestParser that you will have to implement, and you'll have to register it with Shaka Player. You can either compile it into the library in a custom build, or you can leave it in the app (outside the library) and register it at runtime.

[joeyparrish]

@Ajaay, is there anything else we can do for you on this? Or is it okay to go ahead and close the issue?

[joeyparrish]

Closing due to inactivity.