Videos: Stream OGV, VP8, VP9, AV1, WebM, and HEVC if supported #440 #513 #2379

[lastzero]

As a user with a browser that supports OGV, VP8, VP9, AV1, WebM, and/or HEVC, I want to instantly watch videos in these formats without transcoding to AVC.

PR #2379 adds optional HEVC support to avoid transcoding to AVC when using Safari. Rather than just auto-detecting this particular codec, it seemed reasonable to add support for other common video formats and containers, without waiting for follow-up feature requests.

This change also introduces bitrate limiting so that high bitrate videos can still be transcoded before streaming:

• The API endpoint limits the video bitrate based on what has been configured with PHOTOPRISM_FFMPEG_BITRATE.
• If the bitrate is higher, videos in supported formats will also be transcoded, unless FFmpeg has been disabled.
• Note that videos that have already been transcoded will not be re-transcoded when the limit changes. To do this, you would have to manually delete them from the sidecar folder and then run photoprism convert in a terminal (or play them so that transcoding happens on demand).

Based on these enhancements, contributors should be able to add more formats and/or tweak browser support detection as needed. Special thanks to Andre Carrera @acarrera94 for the original pull request #2379 and pushing this forward!

[heikomat]

Transcoding the file massively decreased the file size, which is really helpful on the go.

A 30 second video taken with my smartphone is about 100MB in size. If photoprism were to directly stream that, that would eat up 100 MB of mobile data (if it can even download at constant 3.5MB/sec)

We should definitely test, if not reducing the file size through transcoding hurts user experience because of increased data usage

[heikomat]

On a desktop with a direct connection to the source, having it directly stream is awesome though, because of faster start and better quality.

[lastzero]

Guess we'll end up adding config options eventually 😅

In general, OGV, VP8, VP9, AV1, WebM, and HEVC are modern formats/containers that should not be larger than AVC unless it is 4K 60FPS material. If you know this and want to stream it over a mobile connection, you should indeed expect some issues.

Thinking pragmatically, it probably makes little difference to those who handle support requests whether a user reports a slow connection or slow transcoding....

[heikomat]

Guess we'll end up adding config options eventually 😅

In general, OGV, VP8, VP9, AV1, WebM, and HEVC are modern formats/containers that should not be larger than AVC unless it is 4K 60FPS material. If you know this and want to stream it over a mobile connection, you should indeed expect some issues.

Thinking pragmatically, it probably makes little difference to those who handle support requests whether a user reports a slow connection or slow transcoding....

the container doesn't really make a difference regarding video size. What does make a difference is how much cpu-time was spent compressing the video inside the container. The example 30-sec video i mentioned could probably be compressed down to like 20MB (without visual quality loss) if given enough cpu-time.

Smartphones usually don't compress much though, because their cpus are too slow for that.

What i'm trying to say is: even regular 1080p or 720p video could be larger than necessary, and having an option would be nice.
Maye we can at some point kinda auto-guess what would be best in a certain situation. But that might lead to questions aswell, because it then would sometimes be transcoded and sometimes not

[lastzero]

You're welcome to add a "force transcoding" follow-up issue! I didn't see this coming :)

[dror3go]

I too agree with the idea of forcing transcoding - however I feel it needs to be the default. Just like pictures are getting lower-resolution versions, showing the original version must be available but not the default.

My smartphone generates ~300MB files in ~30 seconds of video, and I'm totally OK with seeing a version but fast, rather than waiting endlessly for the original version to download.
Even old videos taken decades ago but are long enough and are ~100MB+ in size are unwatchable and deserve treatment in my opinion.

[lastzero]

• Are there mobile devices that directly create OGV, VP8, VP9, AV1 or WebM videos? Software encoding is far too slow for this use case, I would be very surprised.
• Except maybe on high-end Google devices, the only hardware encoders that are generally available are HEVC and AVC, right? 🤔
• Since AVC was not transcoded to AVC even before this change, HEVC is the only codec that could benefit from forced transcoding to AVC, which automatically limits the frame rate with our settings. Especially if a high resolution and/or frame rate was used for recording, the videos can get very large. Absolutely true.
• I'm a bit surprised this didn't come up sooner, since streaming HEVC without transcoding has been requested many times and the corresponding pull request has been open for a long time. Just because the CLA was not signed, we couldn't merge it earlier. (Edit: That's not to say I'm against it, in fact that was my thought, but no one else seemed to consider problems with video files being too large, and I don't have that many videos to worry about.)

[lastzero]

This is just an idea for now, but it seems feasible to use the existing bitrate limit to automatically force transcoding if the original bitrate (size/duration) is higher than this:

PHOTOPRISM_FFMPEG_BITRATE: "50"

Original video size and duration are generally known after indexing, although this might not be true for all containers and codecs. Let's call this Option (1).

Option (2) would be to stream large, untranscoded videos only if the HTTP request originates from a private network. The assumption here is that a local connection is always fast enough, e.g. compared to a mobile connection. This is easy to implement as there are only 3 IP ranges to check:

Class A: 10.0. 0.0 to 10.255. 255.255.
Class B: 172.16. 0.0 to 172.31. 255.255.
Class C: 192.168. 0.0 to 192.168. 255.255.

Option (3) would be to negotiate a bit rate based on the determined network bandwidth and the capabilities of the device. This option is of course the most complex and requires the most testing. So probably not for now, unless someone has expert knowledge and wants to contribute a complete solution we just need to merge.

[jahanson]

Food for thought: local IP addresses aren't always unmetered. I have an automatic wireguard VPN on my phone that connects as a 10.0.0.0/8.