Plain MP4 (not DASH/HLS) through MediaSource

[nallown]

I have a feature request, so right now the way it seems is that Shaka Player isn't made for playing MP4 files, its specifically made for playing HLS and DASH streams and not MP4.

Now, the thing is I want to use Shaka Player for playing my MP4 videos as well, that's because Shaka Player uses MSE in a great way to play its videos, MSE can be hard and complicated to work with if you haven't used it before so I don't want to risk rolling out a inefficient MSE implementation in my player because it could slow down video playback and cause issues. I have hundreds of thousands of people using my site every month and I don't want to risk slowing things down for them.

Instead, I want to use the existing MSE implementation of Shaka Player to play MP4 videos.

The reason to why I want to use MSE is because I want to have control over the buffering configuration, and after doing some searching I figured that MSE lets you do that. Also, Shaka Player has a great option for configuring the buffer options. So I want to reuse the MSE implementation from Shaka Player in my player, also I want to be able to see the playback stats to make it far easier for me debug networking issues that my users may have. I'm trying to add a "stats for nerds" option on my site, similar to the YouTube one.

Is it possible to add MP4 support to Shaka Player and if not, then is there way for me to add the YT "stats for nerds" option to my site without having to convert my videos into HLS, DASH segments and also do you have recommendations on how I can control my buffer? My resources are limited so I can't afford re-encoding my videos into HLS and DASH segments yet so I need to stick to MP4 for now to safe on storage.

Have you read the FAQ and checked for duplicate issues: Yes

What version of Shaka Player are you using: 2.3.0

Can you reproduce the issue with our latest release version: Yes

Can you reproduce the issue with the latest code from master: Yes

Are you using the demo app or your own custom app: Both

If custom app, can you reproduce the issue using our demo app: Yes

What browser and OS are you using: Chrome on Windows 10

What are the manifest and license server URIs: (any mp4 URI)

What did you do? Loaded a MP4 Video using Shaka Player

What did you expect to happen? I expected Shaka Player to load the video

What actually happened? It didn't load the video, because Shaka Player doesn't have support for playing MP4 files directly.

[joeyparrish]

You may be interested in #816, in which will we support playback of single files. However, you should note that this would not use MSE.

If I'm not mistaken, MediaSource can be used to play fragmented MP4s, but not MP4 files generally. Fragmented MP4 files are structured differently and can be fetched in independent pieces. This is what DASH and HLS use when they refer to MP4 files. General MP4 files will have to be played directly with HTML5 video, which you can already do today without Shaka Player.

As for stats, you may be interested in player.getStats(), which returns a bundle of stats for display, for aggregate QoE logging, etc. Some of those stats are only collectable, though, when we stream through MSE. Using src=, the browser becomes responsible for networking and buffering, and there is no adaptation, so most of our stats and configuration becomes meaningless in that mode. When we do offer src= playback, it will be a convenience feature only, so developers don't have to switch between Shaka Player and plain HTML5 video for different sources.

Does this answer your question?

[nallown]

Ah I see, is it not possible to add support for MP4 playback with the QoE stats and buffer control then? Because I love the Shaka Player API and a lot of things it offers. I don't mind if a lot of things become unusable but QoE monitoring and buffer control. I would love to have QoE and buffer control.

Also most tutorials and articles about MSE that I've read use MP4 files as examples, MSE only works with MP4 files if they have the "moov" atom up the front since the "moov" atom is what gives the control over seeking in the videos using the "Range" or "Bytes" header.

Most MP4 files should have the "moov" atom up the front though if they're made for web playback since otherwise you would pretty much not be able to seek the video. So majority of MP4 files should be playable using MSE.

You can play all MP4 files with MSE but not all would be seekable but I think that those MP4 files wouldn't be seekable in the HTML5 player anyway because its missing the "moov" atom that gives it seek support.

[joeyparrish]

If you use video.src= to play an MP4 without MediaSource, you give up control over networking and buffering. The browser will do those things, instead.

I am not certain, but I am under the impression that not all MP4 files are playable with MSE. I could be wrong. In any case, our plans for playing back single files do not include MSE.

[nallown]

That's really disappointing. I was hoping I could maybe use Shaka Player to play my videos in. You and the team seem to know what you're doing when it comes down to building video players that relies on custom networking so I wanted to add your guys player to my web stack, it would of made it so much better.

Maybe one day I'll have the resources to stream videos using HLS and DASH but for now I'll guess I'll just learn how to use my own implementation of MSE.

Thank you for responding to my messages, it honestly would be great if you guys could add MP4 playback using MSE but if not then thank for responding to my messages.

[joeyparrish]

Well, plain MP4 with MSE is not in our current plans, but if it's something you want to contribute, I'm not opposed to it. We just don't have time to work on it ourselves.

If you decide to pursue it, your best starting point would be some experiments using MSE directly, to figure out what will and won't be accepted by most browsers. We would be happy to advise you further after that, if it's helpful.

[nallown]

I'll give it my best then and will try to learn how to work MSE, I think with your guys advise, I can really get somewhere. I'll start by creating a basic player that relies on MSE, and once I've figured out better how MSE works, and have a working product, I'll start reading your guys code and try to implement into that.

[joeyparrish]

I just had a discussion about this with another teammate, and we think the issue with general MP4s in MediaSource is the hierarchical nature of MP4 boxes. In fragmented MP4, segments can be parsed independently, but that is not the case in general MP4s. So you may need to be careful about your content, and transmux it into fragmented MP4 if it's not in that format already.

All of this is speculation, and we haven't spent any effort to verify what browsers do or don't support when it comes to non-fragmented MP4 + MSE.

[nallown]

I see, I think the transmuxing process is adding the "moov" atom at the front of the video. At least that's what understood from skimming through articles. I think the way it works is that with non fragmented videos I just fetch the whole video rather then a partial video.

I'll give it a try and will keep you guys posted. Hopefully the process of building a basic MP4 + MSE player will be simple, once that in place the hard part of porting that over to Shaka Player will begin.

[nallown]

I've done some experimenting and this is what I found out.

MP4 files can only be played using MSE if they're fragmented (just like you said), the fragmenting process has two parts to it:

1. Moving the "moov" atom to the front of the video
2. Turning the video file into chunks by segmenting it

People can stream MP4 files using the web perfectly fine without having to segment it. All they need is the "moov" atom at the front and it should be fine. Most MP4 files that are played using the web should already have the "moov" atom at the front of the video since otherwise people wouldn't be able to seek to a specific time in the video using the web, so for most people they wouldn't have to do anything here.

Now for the segmenting part its another story. I tried segmenting some videos and figured out that they played slower then normal MP4 videos when played through the web unless if I implemented MSE, it had a much longer startup time we're talking about a 20 second difference here. This was only in Chrome though in Firefox it was playing fine, just like a normal MP4.

Since most MP4 videos play fine without segmenting, they shouldn't need to be segmented to be played through the web, if anything it could make things worse.

Another thing to note is that most encoders like "ffmpeg" and "handbrakecli" have a "optimize for web" option and when they enable that option, it only adds the "moov" atom at the front, it doesn't segment the video.

So I assume that most MP4 videos aren't segmented.

But the thing is, if a video isn't segmented then it basically isn't coded to work with MSE. MSE is made to only load video segments and not byte ranges. which is what the built in HTML5 player and any other web player that plays a MP4 file normally uses.

So the moral of the story is...

• MSE can only play MP4 files if they're fully fragmented (So if they're "moov"ed and "segmented").
• Segmented MP4 files play slower on the web unless if they're played through MSE
• Since majority of MP4 files aren't played through MSE, they're most likely not segmented
• Since majority of MP4 files aren't segmented, majority of MP4 files on the web wouldn't work with MSE

At this point, someone might as well use HLS or DASH then instead of segmented MP4 files since they're not treated as normal MP4 files. They're only better if they're played through MSE and since HLS and DASH are better options for MSE people might as well use them instead.

[joeyparrish]

Makes sense to me. Thanks for looking into this!

Since general MP4s do not work through MSE, I'm going to go ahead and close this issue as "infeasible".

Please let us know if there's anything else we can do!