Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Videos: Stream OGV, VP8, VP9, AV1, WebM, and HEVC if supported #440 #513 #2379 #2461

Closed
lastzero opened this issue Jun 24, 2022 · 17 comments
Closed
Assignees
Labels
enhancement Refactoring, improvement or maintenance task frontend Requires experience with HTML/JS/CSS released Available in the stable release ux Impacts User Experience video Video Formats, Transcoding, FFmpeg, Streaming & Co

Comments

@lastzero
Copy link
Member

lastzero commented Jun 24, 2022

As a user with a browser that supports OGV, VP8, VP9, AV1, WebM, and/or HEVC, I want to instantly watch videos in these formats without transcoding to AVC.

PR #2379 adds optional HEVC support to avoid transcoding to AVC when using Safari. Rather than just auto-detecting this particular codec, it seemed reasonable to add support for other common video formats and containers, without waiting for follow-up feature requests.

This change also introduces bitrate limiting so that high bitrate videos can still be transcoded before streaming:

  • The API endpoint limits the video bitrate based on what has been configured with PHOTOPRISM_FFMPEG_BITRATE.
  • If the bitrate is higher, videos in supported formats will also be transcoded, unless FFmpeg has been disabled.
  • Note that videos that have already been transcoded will not be re-transcoded when the limit changes. To do this, you would have to manually delete them from the sidecar folder and then run photoprism convert in a terminal (or play them so that transcoding happens on demand).

Based on these enhancements, contributors should be able to add more formats and/or tweak browser support detection as needed. Special thanks to Andre Carrera @acarrera94 for the original pull request #2379 and pushing this forward!

@lastzero lastzero self-assigned this Jun 24, 2022
@lastzero lastzero added enhancement Refactoring, improvement or maintenance task ux Impacts User Experience frontend Requires experience with HTML/JS/CSS labels Jun 24, 2022
@heikomat
Copy link
Sponsor Contributor

Transcoding the file massively decreased the file size, which is really helpful on the go.

A 30 second video taken with my smartphone is about 100MB in size. If photoprism were to directly stream that, that would eat up 100 MB of mobile data (if it can even download at constant 3.5MB/sec)

We should definitely test, if not reducing the file size through transcoding hurts user experience because of increased data usage

@heikomat
Copy link
Sponsor Contributor

On a desktop with a direct connection to the source, having it directly stream is awesome though, because of faster start and better quality.

@lastzero
Copy link
Member Author

Guess we'll end up adding config options eventually 😅

In general, OGV, VP8, VP9, AV1, WebM, and HEVC are modern formats/containers that should not be larger than AVC unless it is 4K 60FPS material. If you know this and want to stream it over a mobile connection, you should indeed expect some issues.

Thinking pragmatically, it probably makes little difference to those who handle support requests whether a user reports a slow connection or slow transcoding....

@heikomat
Copy link
Sponsor Contributor

Guess we'll end up adding config options eventually 😅

In general, OGV, VP8, VP9, AV1, WebM, and HEVC are modern formats/containers that should not be larger than AVC unless it is 4K 60FPS material. If you know this and want to stream it over a mobile connection, you should indeed expect some issues.

Thinking pragmatically, it probably makes little difference to those who handle support requests whether a user reports a slow connection or slow transcoding....

the container doesn't really make a difference regarding video size. What does make a difference is how much cpu-time was spent compressing the video inside the container. The example 30-sec video i mentioned could probably be compressed down to like 20MB (without visual quality loss) if given enough cpu-time.

Smartphones usually don't compress much though, because their cpus are too slow for that.

What i'm trying to say is: even regular 1080p or 720p video could be larger than necessary, and having an option would be nice.
Maye we can at some point kinda auto-guess what would be best in a certain situation. But that might lead to questions aswell, because it then would sometimes be transcoded and sometimes not

@lastzero
Copy link
Member Author

You're welcome to add a "force transcoding" follow-up issue! I didn't see this coming :)

@lastzero lastzero added the please-test Ready for acceptance test label Jun 24, 2022
@dror3go
Copy link

dror3go commented Jun 24, 2022

I too agree with the idea of forcing transcoding - however I feel it needs to be the default. Just like pictures are getting lower-resolution versions, showing the original version must be available but not the default.

My smartphone generates ~300MB files in ~30 seconds of video, and I'm totally OK with seeing a version but fast, rather than waiting endlessly for the original version to download.
Even old videos taken decades ago but are long enough and are ~100MB+ in size are unwatchable and deserve treatment in my opinion.

@lastzero
Copy link
Member Author

lastzero commented Jun 24, 2022

  • Are there mobile devices that directly create OGV, VP8, VP9, AV1 or WebM videos? Software encoding is far too slow for this use case, I would be very surprised.
  • Except maybe on high-end Google devices, the only hardware encoders that are generally available are HEVC and AVC, right? 🤔
  • Since AVC was not transcoded to AVC even before this change, HEVC is the only codec that could benefit from forced transcoding to AVC, which automatically limits the frame rate with our settings. Especially if a high resolution and/or frame rate was used for recording, the videos can get very large. Absolutely true.
  • I'm a bit surprised this didn't come up sooner, since streaming HEVC without transcoding has been requested many times and the corresponding pull request has been open for a long time. Just because the CLA was not signed, we couldn't merge it earlier. (Edit: That's not to say I'm against it, in fact that was my thought, but no one else seemed to consider problems with video files being too large, and I don't have that many videos to worry about.)

@lastzero
Copy link
Member Author

lastzero commented Jun 24, 2022

This is just an idea for now, but it seems feasible to use the existing bitrate limit to automatically force transcoding if the original bitrate (size/duration) is higher than this:

PHOTOPRISM_FFMPEG_BITRATE: "50"

Original video size and duration are generally known after indexing, although this might not be true for all containers and codecs. Let's call this Option (1).

Option (2) would be to stream large, untranscoded videos only if the HTTP request originates from a private network. The assumption here is that a local connection is always fast enough, e.g. compared to a mobile connection. This is easy to implement as there are only 3 IP ranges to check:

Class A: 10.0. 0.0 to 10.255. 255.255.
Class B: 172.16. 0.0 to 172.31. 255.255.
Class C: 192.168. 0.0 to 192.168. 255.255.

Option (3) would be to negotiate a bit rate based on the determined network bandwidth and the capabilities of the device. This option is of course the most complex and requires the most testing. So probably not for now, unless someone has expert knowledge and wants to contribute a complete solution we just need to merge.

@jahanson
Copy link

Food for thought: local IP addresses aren't always unmetered. I have an automatic wireguard VPN on my phone that connects as a 10.0.0.0/8.

@lastzero
Copy link
Member Author

You are right that using a VPN would interfere with Option (2). It would be a known issue for this solution.

I was hoping no one would notice (so quickly) so that the options remain easy to explain, understand and vote for xD

@dror3go
Copy link

dror3go commented Jun 24, 2022

I'm a bit surprised this didn't come up sooner

I can say for myself that it has been bugging me, however since videos are a small fraction of my library and my main focus is on photos - I I've yet to open an issue on it.

would be to negotiate a bit rate based on the determined network bandwidth and the capabilities of the device

That sounds the way to go on the long run maybe, YouTube-style.
However, I feel that as a first step - a static setting is fine.

lastzero added a commit that referenced this issue Jun 26, 2022
Implements Option (1) as described in the issue comments.
@lastzero
Copy link
Member Author

This change implements Option (1) as described above:

  • It limits the video bitrate based on what has been configured with PHOTOPRISM_FFMPEG_BITRATE.
  • If the bitrate is higher, videos in supported formats will also be transcoded, unless FFmpeg has been disabled.
  • Note that videos that have already been transcoded will not be re-transcoded when the limit changes. To do this, you would have to manually delete them from the sidecar folder and then run photoprism convert in a terminal (or play them so that transcoding happens on demand).

Unfortunately, my time to work on video transcoding is very limited, so I want to encourage everyone to help test and verify these changes. Thank you!

@lastzero
Copy link
Member Author

Special thanks to Andre Carrera @acarrera94 for the original pull request #2379 and pushing this forward! Even though the "finishing touches" took more time than expected, it helped get it done.

@lastzero
Copy link
Member Author

The preview build 220626-b0911205 is now available for testing.

@lastzero
Copy link
Member Author

Released since no problems were reported!

@lastzero lastzero added released Available in the stable release and removed please-test Ready for acceptance test labels Jun 29, 2022
@lastzero lastzero added the video Video Formats, Transcoding, FFmpeg, Streaming & Co label Jul 7, 2022
@mitohund
Copy link

Just wanted to leave a quick comment here regarding the positioning of "PHOTOPRISM_FFMPEG_BITRATE" in the example docker-compose.yml. I had skipped that variable when adjusting my docker-compose.yml since it looks like it only applies to hardware encoding, as it's grouped under "## Hardware Video Transcoding (for sponsors only due to high maintenance and support costs):"

I'm not a sponsor yet, so I guess I don't have hardware transcoding yet, but the variable works fine for me, as is probably intended.

@graciousgrey
Copy link
Member

Thank you very much for your feedback! We did not notice that this might be confusing.
The bitrate parameter is also available for non-sponsors. You find the complete list of config options including information on availability for sponsors/non-sponsosrs in our docs https://docs.photoprism.app/getting-started/config-options/ :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Refactoring, improvement or maintenance task frontend Requires experience with HTML/JS/CSS released Available in the stable release ux Impacts User Experience video Video Formats, Transcoding, FFmpeg, Streaming & Co
Projects
Status: Release 🌈
Development

No branches or pull requests

6 participants