Skip to content
Kitty-HivensPublic

About

Video decode and playback for JVM desktop apps: FFmpeg through hand-written Java FFM (Panama) bindings, Skia frames out, Compose surface on top

Resources

Contributing

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

skinema

Video decode, encode and playback for JVM desktop apps. FFmpeg through hand-written Java FFM (Panama) bindings: frames out as raw RGBA or Skia images with a Compose Desktop surface on top, and frames back in to a muxed file. No JNI wrapper stacks, no embedded player engines. The bundled FFmpeg is built with --disable-network, so skinema performs no network I/O of its own -- it works on the file you hand it and nothing more. (The player, the decoders and the transcoder also take a MediaSource, so bytes fetched anywhere play directly, a stream that only reads forward included; see Streams.)

The same API runs on Android, on the platform's own engine: Media3 over the phone's decoders rather than FFmpeg, so shared code plays, decodes and writes through one set of calls on both. Media3 brings Guava and a few androidx libraries with it, and no native code ships for Android. Pass hardware = HwAccel.AUTO there for the hardware decoders, since the default is the desktop's software one. What differs, and why, is in the Android guide.

val player = VideoPlayer(MediaInput.File(Path.of("background.webm")), loop = true)

// Compose Desktop:
VideoSurface(player, Modifier.fillMaxSize(), scale = VideoScale.Cover)

// Or poll frames yourself from any render loop:
player.acquireFrame()?.let { frame -> /* frame.rgba, frame.width, ... */ }

positionNanos() and durationNanos carry a timeline (duration is null for an animated webp until one lap has played, since the format declares none and the length is only known once it has been read); chapters, tags and coverArt carry the rest of the container's metadata -- the cover ships as the stored png/jpeg bytes for your own image stack. A file whose only video stream is the embedded cover plays frameless, art exposed, sound intact. Phone footage's orientation surfaces as rotationDegrees; VideoSurface applies it, a consumer drawing frames itself must do the same.

Full developer documentation -- a consumer guide and the contributor internals -- lives in docs/.

Dependencies

implementation("dev.hivens:skinema-compose:0.9.0")   // brings -core and -skiko
runtimeOnly("dev.hivens:skinema-natives:9.0.1-1:decode-linux-x64")
runtimeOnly("dev.hivens:skinema-natives:9.0.1-1:decode-linux-arm64")
runtimeOnly("dev.hivens:skinema-natives:9.0.1-1:decode-linux-musl-x64")
runtimeOnly("dev.hivens:skinema-natives:9.0.1-1:decode-linux-musl-arm64")
runtimeOnly("dev.hivens:skinema-natives:9.0.1-1:decode-windows-x64")
runtimeOnly("dev.hivens:skinema-natives:9.0.1-1:decode-windows-arm64")
runtimeOnly("dev.hivens:skinema-natives:9.0.1-1:decode-macos-arm64")
runtimeOnly("dev.hivens:skinema-natives:9.0.1-1:decode-macos-x64")

On Android the same coordinate resolves to the Android build, and there are no natives to add:

implementation("dev.hivens:skinema-compose:0.9.0")   // brings -core; Media3 underneath

That one is Gradle reading the module metadata. A Maven, sbt or Bazel build reads the root POM instead, which for a multiplatform module carries no classes, so from 0.9.0 such a build names the desktop artifacts directly: skinema-core-jvm and skinema-compose-jvm.

0.9.0 compiles against Skiko 0.150.2, which is what Compose Multiplatform 1.12.2 ships. Skiko is compileOnly here, so the copy that runs is the one your Compose brings: that makes Compose 1.12 a floor rather than a preference. On an older Compose, take 0.7.0.

The two versions are independent, and that is deliberate. skinema-natives is versioned as the FFmpeg build it carries plus a repack revision (<ffmpeg>-<revision>), so it moves only when the bundles themselves change -- most library releases leave it untouched. Take the natives version the release notes name and pair it with any library version that names it.

Independent, but not unrelated: the bindings are valid for one set of FFmpeg soname majors, so a library release that moves the pin needs the natives that carry it. Gradle consumers do not have to track that by hand -- skinema-core publishes a constraint naming the natives version it was built against, so a stale pin is raised to it rather than failing at runtime. Override with version { strictly("...") } to hold a specific bundle (a local build, a repack under test); the runtime refuses a mismatched major either way.

The natives classifier is <tier>-<platform>: pick one tier for the platforms you target. The tier sets what the bundle carries -- and its license:

Tier Carries License
core the modern essentials (H.264/HEVC/VP8/VP9/AV1, mainstream audio, images), no subtitles, no GPU decode LGPL
decode core + GPU decode + subtitles + every format FFmpeg decodes natively; and writing -- AV1 video, Opus/AAC/FLAC audio, the mp4/mkv/webm containers and the audio-only .opus/.flac/.wav ones, plus GPU H.264/HEVC encode on Linux LGPL
full decode + H.264 and HEVC software encode (x264/x265) GPL

core is the lean tier -- the modern codecs only, no subtitles, no GPU decode -- for apps that just play current video. It is also the portable one: it needs nothing from the host but the C library, so it loads in a bare container and on a distribution that keeps no libraries at the usual paths. The desktop tiers carry GPU decode, which means they need libva alongside fontconfig. decode (used above) is the complete LGPL player: it adds the libass subtitle stack and the broad formats set (the legacy and broadcast containers and codecs in "What it plays" below). It is also the complete LGPL WRITER -- SVT-AV1 and libopus are BSD, so AV1 video and Opus audio come out of a bundle carrying no GPL at all. What full adds is software H.264 and HEVC specifically, and that is what makes it GPL, since x264 and x265 are; HEVC encode ships everywhere since the x265 source patch (issue #22). On first use the jars unpack to a per-user cache. Without a natives jar, skinema looks for matching system libraries -- fine for development, not what you ship.

On JDK 24+ launch with --enable-native-access=ALL-UNNAMED (or grant your named module): skinema's FFM calls -- and Skiko's, if you use Compose -- are restricted methods that otherwise warn on every run.

Platforms

Every tier ships for eight platforms -- x64 and arm64 on all three desktop operating systems, and both C libraries on Linux:

Platform Bundle built Acceptance suite
linux-x64 on metal on metal
linux-arm64 on metal on metal
linux-musl-x64 in Alpine, on metal on metal, musl JVM
linux-musl-arm64 in Alpine, on metal on metal, musl JVM
windows-x64 on metal (MSYS2 MINGW64) on metal
windows-arm64 on metal (MSYS2 CLANGARM64) on metal
macos-arm64 on metal on metal
macos-x64 cross-compiled on an arm mac none

Linux ships twice because a glibc shared object cannot load into a musl process at all -- Alpine and Void-musl take the linux-musl-* bundles, and nativesPlatform() picks between them by reading which C library the running JVM is itself linked against, not by looking at what the system has installed.

arm64 is first-class here, not an afterthought: linux-arm64 and windows-arm64 are built on real arm machines and must pass the same acceptance suite as their x64 siblings before a bundle uploads. macos-x64 is the one exception, and it runs the other way -- Intel mac runners are scarce, so it cross-compiles on an arm mac and ships without an on-metal run, because an arm JVM cannot load x86_64 dylibs. That is what community-tier support means here.

How old a system the bundles still load on. glibc 2.39 for linux-arm64 full, 2.38 for the other glibc bundles, 2.35 for core-linux-x64 -- so Ubuntu 24.04, Debian 13 and Fedora 39 upward for the desktop tiers, and not RHEL 9. core-linux-x64 alone still loads on Ubuntu 22.04. macOS bundles are built for 14.0, so Sonoma upward. The linux-musl-* bundles have no such floor: musl does not version its symbols. The decode and full Linux bundles want libfontconfig from the host -- libass's font provider links it, and without it none of their libraries load, which is worth knowing before putting them in a headless container. core needs only libc and libm and has no such requirement. VAAPI used to be a second such requirement and is not any more: those bundles carry libva, libva-drm and libdrm themselves, so a machine with no libva installed still loads them. The host's own copy is still preferred where there is one, because the driver on the machine is versioned against it rather than against ours.

What it plays

Containers mp4/mov/m4a, webm/mkv, avi, MPEG-PS/TS, flv, asf/wmv, dv, RealMedia, ogg, mp3, flac, wav, gif, apng, webp, raw ac3/eac3
Video H.264, HEVC, H.266/VVC, VP8, VP9 (incl. webm alpha), AV1; MPEG-1/2, MPEG-4 Part 2, VC-1, WMV 7-9, H.263, Theora, ProRes, DNxHD, FFV1, RealVideo, Cinepak, Indeo, VP6; MJPEG
Animated images GIF, APNG, animated WebP
Audio AAC, AC-3/E-AC-3, DTS, TrueHD, ALAC, Opus, Vorbis, MP1/MP2/MP3, FLAC, WMA (v1/v2/Pro), AMR, WavPack, Monkey's Audio, TTA, ADPCM, G.72x, RealAudio, ATRAC, GSM, WAV PCM -- the device clock masters A/V sync
Subtitles ASS/SSA, SRT, mov_text, WebVTT (libass-rendered); PGS, VobSub, DVB (bitmap); CEA-608/708 closed captions, read out of the video bitstream; external .srt/.ass
Pixels out RGBA8888, straight alpha, exact-pts pacing, BT.601/709/2020 matrix and range honored, PQ/HLG tone-mapped to SDR

The table names what is worth naming, not everything that is there. The decode and full tiers carry every decoder, demuxer and parser FFmpeg builds without an external library -- some five hundred of them -- so the legacy, broadcast and archival formats are in whether or not a row above mentions them. The lean core tier is the one with a list: the modern essentials (H.264/HEVC/VP8/VP9/AV1 and the mainstream audio), and nothing else. H.266/VVC decodes through FFmpeg's native decoder (CPU-only, no GPU path yet).

What is not there is short and stays short: anything needing a library the bundle does not carry -- JPEG XL, SVG, tracker music, DVD and Blu-ray navigation, teletext -- and TTML, which FFmpeg has no decoder for at all.

Audio: pass audio = true to VideoPlayer -- aac, ac3/eac3, alac, opus, vorbis, mp3, flac and WAV pcm (16/24/32-bit and float) decode through the same bindings, and the audio device becomes the player's clock (video follows sound, never the reverse). The sound reaches the device in the shape the file has it: six channels stay six, twenty-four bits stay twenty-four, and the rate is never converted. A device that cannot take that shape refuses, the player walks down to one it can, and activeAudioFormat reports what was actually taken rather than what was asked for. audioChannels = ChannelPreference.Stereo folds first for a consumer that knows the fold is wanted, since a device accepting six channels is not evidence that six speakers exist. Audio-only files play frameless. Files with several audio tracks expose them (audioTracks, language and title included) and switch in place (selectAudioTrack) -- the picture keeps playing and the sound re-anchors at the playhead. setRate plays at 0.5x-4x with the pitch preserved (FFmpeg's atempo). volume opens at a gain rather than at full, and is re-applied to every line the player opens -- a track switch, a device coming back after an outage -- so a quiet player stays quiet; setVolume changes it live. Opening at one is not the same as setting it straight after: the sink opens and takes its first chunk on the audio thread's own schedule, so a call racing that has already lost. HDR content (PQ/HLG over BT.2020) is tone-mapped to SDR on the decode path -- the transfer is inverted, highlights roll off against BT.2408 diffuse white, and the gamut maps to BT.709 -- so it no longer plays washed out. This is a fixed reasonable-look mapping, not a metadata-accurate one: the stream's HDR metadata (MaxCLL, mastering-display luminance, HDR10+, Dolby Vision) is not read, so the roll-off uses an assumed peak rather than the content's. Native-HDR passthrough (driving an HDR display) is out of scope.

Subtitles: subtitleTracks enumerates what the container carries (language, title, default/forced), and grows a CEA-608/708 track the moment one is seen -- closed captions are not a stream but ATSC A53 payload inside the H.264/HEVC bitstream, so no container can be asked whether a file has them and nothing is advertised until a frame answers; selectSubtitleTrack turns one on -- off by default, nothing subtitle-related runs until then. Text tracks (ASS/SSA, SRT, mov_text, WebVTT) render through libass with the full typesetting, mkv-embedded fonts included; bitmap tracks (PGS, VobSub, DVB) decode to pixels and need no libass at all. External .srt and .ass files join via addExternalSubtitles on the same timeline. VideoSurface composites the overlay and keeps glyphs crisp at window size; a consumer drawing frames itself polls acquireSubtitles -- the acquireFrame contract for text. libass is an optional capability: without it text tracks refuse selection and everything else plays on. Text rendering uses the system's fonts (fontconfig, DirectWrite, CoreText); a fontless headless box renders blank overlays.

What it writes

MediaWriter is the player inverted -- RGBA8888 frames and S16LE stereo PCM in, a muxed file out -- and Transcoder joins the two halves to convert a file. What a bundle can write is set by its tier, and there the split is about licence rather than size: everything below except the software H.264/HEVC pair is LGPL or BSD and rides decode.

Video AV1 (SVT-AV1), and on Linux H.264/HEVC on the GPU through VAAPI -- both in decode; H.264 and HEVC (x264/x265) in full, which is what makes that tier GPL
Audio AAC, Opus, FLAC, WAV PCM
Containers mp4/mov, mkv and webm; .opus, .flac and .wav for a file with sound and no picture
Pixels in RGBA8888, reverse-scaled into whatever the encoder accepts; the matrix the conversion used is tagged on the file, so a player reading the tag and one guessing from the geometry agree

Nothing is assumed about an encoder: the pixel format, sample format, channel layout and sample rate are all negotiated against what it advertises, and one it cannot take is refused by name rather than as a bare errno. Hardware encode is fail-closed -- naming a GPU encoder that cannot open throws instead of quietly falling back to software. See the encoding guide.

Behavior contract

  • Fail closed. A file the pipeline cannot handle surfaces as VideoPlayer.State.Failed -- show your fallback. No partial recovery, no garbage frames, no hangs.
  • Drop late. A slow consumer skips frames; the clock never lags.
  • It stops for a consumer that stops looking. A window nobody can see is noticed, and the player stops decoding and converting for it. VideoSurface tells by how long its window takes to answer for a frame and reports it through reportVisible. Measured under XWayland on Hyprland, that catches a window left on a workspace that is not on screen. Behind a fullscreen window it answers the same way, and a covered macOS window is expected to, going by Skiko's source. Minimising is taken from the lifecycle everywhere. A covered window on Windows most likely keeps drawing at full speed and is not seen. Outside Compose, a mailbox that was being read and then is not is noticed on its own. What stops is the policy's to say, and the default (WhenUnwatched.FollowSound, which replaced Freeze as the default) does what a browser does with a tab put behind another: a player you can hear plays on without its picture, which catches up with the sound when it is wanted again, and a silent one or one at volume zero pauses. Freeze and KeepTime fix one answer for every case, and setPresenting marks the moment exactly.
  • Seeks answer immediately. An exact seek previews its keyframe while the frame-precise landing decodes behind it; exact = false lands on the keyframe outright -- picture and sound at once, position as coarse as the file's keyframe spacing. Skip buttons want inexact; timeline scrubbing wants exact. stepForward/stepBackward move a single frame and leave the player paused on it.
  • Two threads per player, four with sound. Decode fills a small frame queue and a pacer presents from it; sound adds an audio thread and its watchdog, and a selected subtitle track a fifth. Players are independent and self-synced; play as many as your CPU affords (a desktop comfortably runs dozens of 1080p30 streams).
  • A player can open paused. startPaused = true settles State.Paused instead of Playing and stays on the first frame -- which is on screen, because a poster frame with no picture is a black rectangle. With sound, nothing reaches the line at all: the audio side is told at construction rather than paused by a command, which would arrive after the device had already started. resume is what starts the file, and this is the one pause WhenUnwatched never lifts.
  • One player, many files. setSource puts the next file on the player you already have -- the same threads, the same audio line, the same volume, rate, looping and subtitle canvas -- so a playlist is a list you own plus one call, and the queue, the order and what "next" means stay yours. A file that will not open is refused (sourceFailure) and the one playing carries on, which is what a queue needs from one broken item. loop is live in the same way: turning repeat on no longer costs the position by rebuilding the player.
  • Read-ahead is opt-in. readAheadFrames (default 1) holds that many decoded frames of inventory, so a decode stall does not stall the screen while inventory lasts. Each step of depth costs one full RGBA frame of memory -- 8.3 MB at 1080p. Backgrounds stay at 1; a player UI wants 3-5.

Modules

Module Contents Floor
skinema-core FFM bindings, demux/decode, pacing, VideoPlayer; on Android the Media3 engine JDK 22; Android 10 (API 29)
skinema-skiko VideoFrameImage/SubtitleOverlayImage: pixels as org.jetbrains.skia.Image Skiko (provided by your Compose)
skinema-compose VideoSurface, rememberPlayerState, VideoScale, VideoFrameHolder Compose 1.12, on the desktop and Android
skinema-natives trimmed FFmpeg in tiers, classifier jar per tier+platform --

ROADMAP.md is the project's working memory: every architectural decision, with its reasoning, lives there.

Compatibility policy

Decode correctness for well-formed files in the formats above is a bug when broken. Exotic, damaged or adversarial files get the fail-closed treatment by design; compatibility issues beyond that are triaged at the maintainer's discretion.

License

skinema itself (core, skiko, compose) is Apache-2.0. The natives bundle's license is set by its tier (see Dependencies): core and decode are LGPL -- decode does carry encoders, but only ones that add no GPL surface (the BSD SVT-AV1 and libopus, FFmpeg's own AAC and FLAC, and the VAAPI encoders whose codec lives in the GPU driver); the full tier adds the x264/x265 software encoders, which are GPL, so its FFmpeg build is configured --enable-gpl and its libraries are GPL. Either way the natives ride in separate per-tier, per-platform classifier jars, dynamically linked, never statically embedded into the Apache code, and every bundle ships its license texts (the GPL text on full, LGPL on the rest). An application that distributes the full natives takes on FFmpeg's GPL obligations, as anyone distributing a GPL FFmpeg build does; a consumer that needs to stay LGPL takes core or decode. skinema's own Apache code is unaffected either way.

libvpx, dav1d, SVT-AV1 and libopus are BSD-family -- SVT-AV1 additionally carries the Alliance for Open Media patent grant, whose text ships beside its license; x264 and x265 are GPL (the reason the full build is --enable-gpl). libass (ISC) ships with FreeType and HarfBuzz folded in -- portions of the bundled software are copyright The FreeType Project (freetype.org), licensed under the FreeType License -- while FriBidi stays a separate shared library precisely because it is LGPL. On Windows, where libtool cannot fold a static archive into a DLL, FreeType and HarfBuzz ship as their own DLLs instead, and the bundle also carries the MinGW runtime they link -- zlib, bzip2, and lzma (permissive), winpthread, and libiconv (LGPL, dynamically linked) -- each with its license text.

About

Video decode and playback for JVM desktop apps: FFmpeg through hand-written Java FFM (Panama) bindings, Skia frames out, Compose surface on top

Resources

Contributing

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages