Video

[bjornbytes]

Use libtheora most likely (other video formats don't seem to be free to use).

[bjornbytes]

Might be able to use HTMLVideoElements in the browser (texsubimage2d supports video elements as pixel data source). That way no library needs to be included and we would be able to support all formats that the browser does.

[karai17]

How would this interact with, say, a video stream player object inside of a game? Example: VRChat allows you to create video players that can take in URLs from various websites and stream (synced over the network!) YouTube videos so everyone in a room can watch together. YouTube uses H.264, as do nearly all video streaming platforms.

For local video playback, Theora is fine since you can encode videos that you want to put into your project ahead of time. But for streaming content, H.264 is the de facto standard and I think it would be really, really awesome to be able to support that.

A non-social example could be something like putting tutorial videos on YouTube that new players can access if they need help figuring something out. This would allow you to upload as many tutorials or other videos as you need without having to bundle them with the game, increasing the filesize by incredible amounts. It also allows for much higher quality videos since the files are being streamed so you don't need to worry so much about file size or compression to fit them into your game's assets.

[bjornbytes]

h264 would be awesome. Unfortunately, it is quite a beast.

• It's heavily patented. I don't know the exact details but if someone made a commercial project using LÖVR that included h264 video decoding, either they or lovr would be required to pay royalties to the MPEG LA company. That's pretty unacceptable, and goes against the open-source aspects of LÖVR.
• I don't know of any small libraries that offer h264 decoding under a permissive license. I haven't researched extensively, but options that come to mind are ffmpeg, libav, or x264. All of those are GPL/LGPL and are quite large.
• For optimized decoding, Intel, AMD, and NVIDIA each have their own proprietary SDKs for decoding video on the GPU. I think Windows has an API for it too. We'd probably want to use those so the work can be done in hardware, but that's 4 new (closed) libraries to integrate.

How does VRChat stream its youtube videos? Does youtube have an API that provides video streams? Do they embed a web browser?

Oh, since this issue was created this library was released for MPEG-1 Video decoding.

[karai17]

I don't know the exact implemnentation of VRChat's video players, but I do know that there are youtube downloader sites that you can use to expose direct access to mp4 files on the youtube servers, so i suspect they probably do something like that in the background since none of the video players i've seen have seek bars (it's difficult to seek through a massive mp4 file when streaming; youtube servers tiny chunk mp4 data streams when viewing over youtube, but store full mp4 files as well).

As for the licensing issues, I 100% understand. I think perhaps an optional ffmpeg/h.264 variant module could be created so projects that need it can use it, but it is not shipped directly. this also would (probably?) eliminate any risk of lovr being responsibe for licensing things and require anyone who includes it in their game tobe responsibe. this is one of the reasons why audacity ships LAME separately.

We were speaking in slack the other day about pulling enet out of the main distribution and making it an optional add-on. perhaps several optional add-ons could be created for lovr that are generally useful but perhaps too big or too risky to ship directly. h.264, enet, physics, ssl, etc

[bjornbytes]

Relevant article about the MPEG-1 lib mentioned above: https://phoboslab.org/log/2019/06/pl-mpeg-single-file-library

[TheKnarf]

Maybe lovr could integrate with gstreamer, gstreamer is plugin based so that you can support multiple different video, audio, and streaming formats.

The only problem being that this wouldn't work on web.

[shakesoda]

libvpx is practical and would let you deal in webm. this has the nice benefit of not needing to ship the library on web at all, since you should be able to use the native decoding in the browser (which, in all likelihood, will also be libvpx wrapped up in a different api)

[bjornbytes]

This feature won't be integrated into "core" LÖVR, instead it should use the plugin system.

[bjornbytes]

Vulkan video encode/decode extensions were released today. Might make some of this possible on some GPUs and it wouldn't require extra libraries.