Use `media_kit` to play videos

[deckerst]

(originally explored as part of #555)

The good:

• improved seek accuracy (fixing bug: Skipping in Videos is Widely Divided #660 and making it possible to implement Fast forward/backward on long press video sides #262),
• improved looping transition (but not fixing Seemless loops #356, cf [Enhancement] seamless loop media-kit/media-kit#150 loop-file: add seamless-looping option to ensure that there are no delayed frames at beginning of each loop mpv-player/mpv#10748),
• improved speed selection (works fine at least from x0.25 to x4,
• HDR support (fixing HDR videos look pale #486),
• AV1 support (fixing AV1 video codec support #555),
• some corrupted videos become playable (e.g. mp4 with inconsistent duration in their metadata).

The bad:

• often fails to load the first frame of local videos (cf [Android] First frame is sometimes a solid color media-kit/media-kit#339)

Flavor	Package	media_kit v1.1.1 + ffmpeg_kit v5.1.0-LTS (min)	media_kit v1.1.1 + ijk	ijk only
play	aab	83.6	63.7	47.4
izzy	armeabi-v7a	34.2	23.1	18.1
izzy	arm64-v8a	27.5	23.1	17.9
izzy	x86_64	29.3	24.7	18.7

[deckerst]

@alexmercerind

1. subtitles

• Using the subtitle stream works fine, but the stream only contains the unstyled text. Not a big deal, but it's a step down from the subtitle stream I was receiving from ijkplayer, where things like bold, italics, etc. was inlined with the text.

• I currently style SubtitleViewConfiguration with a transparent color to hide the embedded subtitle view, but it would be cleaner if there was a way to remove the SubtitleView from the widget tree altogether (and without using libass).

2. screenshot

It works fine!

3. codec selection

I appreciate that you reduced the library size by tightening the codec selection. However, I can no longer play some of my video/audio streams (e.g. avi, ac3). Is it possible for consuming apps to customize which codecs are bundled? Should I rebuild mpv, and then fork media_kit to point to that custom build?

4. errors

• I get the following exception when destroying video views (loaded from Android content URIs). It seems final fd = int.parse(data.pathSegments.last) is assuming pathSegments are not empty. But it may be a deeper issue.

E/flutter ( 3507): [ERROR:flutter/runtime/dart_vm_initializer.cc(41)] Unhandled Exception: Bad state: No element
E/flutter ( 3507): #0      _Array.last (dart:core-patch/array.dart:56:5)
E/flutter ( 3507): #1      Media._finalizer.<anonymous closure> (package:media_kit/src/models/media/media_native.dart:47:48)
E/flutter ( 3507): #2      _RootZone.runUnaryGuarded (dart:async/zone.dart:1594:10)
E/flutter ( 3507): #3      _RootZone.bindUnaryCallbackGuarded.<anonymous closure> (dart:async/zone.dart:1633:26)
E/flutter ( 3507): #4      _FinalizerImpl._runFinalizers (dart:core-patch/finalizer_patch.dart:63:18)
E/flutter ( 3507): #5      _FinalizerImpl._handleFinalizerMessage (dart:core-patch/finalizer_patch.dart:76:15)

• last but not least, I can still make my app crash by quickly going through my videos (after about 30 videos? maybe related to the dispose cleanup issue above). I'm willing to provide additional details, just let me know how I can help you debug this.

[alexmercerind]

Replacing mediacodec to mediacodec-copy can improve stability to some degree. The VideoControllerConfiguration allows this. An overall efficient approach will be to reuse existing Players (if possible). I'll ask for logcat; we have debug symbols now.

I'll have a look at the rest. I possibly messed up the Media GC finalizer.

EDIT: media-kit/media-kit@38d2bc9

[alexmercerind]

I did enable AVI. Something seems missing.

EDIT: An AVI sample I grabbed does play.

[deckerst]

I did enable AVI. Something seems missing.

EDIT: An AVI sample I grabbed does play.

My bad, I was imprecise: the AVI container I tried contained an MJPEG video stream. This is niche, but useful to me. Your codec selection is reasonable for most users, that's why I was asking how to customize it for my needs.

Edit: from libmpv logs:

PlayerLog(prefix: vd, level: error, text: Failed to initialize a decoder for codec 'theora'.)
PlayerLog(prefix: vd, level: error, text: Failed to initialize a decoder for codec 'msmpeg4v3'.)
PlayerLog(prefix: ad, level: error, text: Failed to initialize a decoder for codec 'wmav2'.)
PlayerLog(prefix: vd, level: error, text: Failed to initialize a decoder for codec 'mjpeg'.)
PlayerLog(prefix: ad, level: error, text: Failed to initialize a decoder for codec 'ac3'.)
PlayerLog(prefix: ad, level: error, text: Failed to initialize a decoder for codec 'eac3'.)

For me ac3, eac3 and mjpeg are important, the rest not so.

[alexmercerind]

You can create your own builds for libmpv + FFmpeg + media_kit by:

1. Fork https://github.com/media-kit/libmpv-android-video-build & enable GitHub actions.
2. Modify the FFmpeg configure step.
   It may look overwhelming (it is in-fact). Mostly you need to only look at the --enable-decoder, --enable-demuxer & --enable-parser for your codec selection; other things are configured already.
3. The GitHub actions will compile the project(s) & resulting binaries will be automatically uploaded as release & artifact.
4. Remove media_kit_libs_android_video from pubspec.yaml.
5. Bundle your *.jar(s) with your application by modifying build.gradle, something like this.

[alexmercerind]

I picked the FFmpeg configuration from fijkplayer; kind of surprising things are different. Still, you have the freedom.

[deckerst]

I picked the FFmpeg configuration from fijkplayer; kind of surprising things are different. Still, you have the freedom.

I customized the FFmpeg config for Aves, so it's not the same as the basic fijkplayer.

[alexmercerind]

Interesting.

For the note, you won't need image encoders for screenshot; we are encoding pixel-buffer with package:image.

[deckerst]

we are encoding pixel-buffer with package:image

I noticed that the screenshot function is a tad slow. Not a big deal, but maybe it's because you use package:image, which is pure dart and usually slower compared to native encoders. That package is actually more suitable for server usage, or when performance are no concern. On Android for example, you may use instead android.graphics.Bitmap and its compress(Bitmap.CompressFormat.JPEG, ...).

[alexmercerind]

It runs on Isolate, so it may be slow (but definitely non-blocking (doesn't freeze UI) unlike native interop); portable too.

The performance isn't really crucial for screenshots (not at all). When we want hardware-level performance at the time of rendering, we don't even use pixel buffers.