Add transcoding on FFmpeg, GStreamer and mpv

MediaToolkit.NET 0.3.0

The transcoding release. A file can now be probed, converted and read frame by
frame, and the conversion is described once and run by whichever backend can do
it: FFmpeg, GStreamer or mpv.

TranscodeRequest says what the output holds — which streams of which inputs,
copied or re-encoded, subtitles converted or burned in, a trim, tags and
chapters. Every backend resolves it into the same TranscodePlan, then says for
itself what it cannot do before anything is written. The facade asks FFmpeg
first, then GStreamer, then mpv, and when all three refuse the exception carries
each refusal.

Added

A common transcoding API (MediaToolkitNet.Abstractions)

await MediaToolkitNetBackends.TranscodeAsync(TranscodeRequest.Remux("in.mkv", "out.mp4"));

await MediaToolkitNetBackends.TranscodeAsync(new TranscodeRequest("out.mkv")
{
    Inputs = ["film.mkv", "film.en.srt"],
    Start = TimeSpan.FromMinutes(1),
    End = TimeSpan.FromMinutes(2),
    Streams =
    [
        OutputStream.Video(StreamSource.First(MediaStreamKind.Video),
            new VideoOutputSettings(MediaCodec.H264) { Width = 1280, Quality = 23, Speed = EncoderSpeed.Fast }),
        OutputStream.Audio(StreamSource.All(MediaStreamKind.Audio), new AudioOutputSettings(MediaCodec.Opus)),
        OutputStream.Subtitles(StreamSource.First(MediaStreamKind.Subtitle, input: 1), MediaCodec.SubRip),
    ],
    Metadata = new Dictionary<string, string> { ["title"] = "Film" },
});

• TranscodeRequest, OutputStream (copy, drop, re-encode video or audio,
  convert subtitles, burn them in), StreamSource selectors by index, by kind,
  by language and by position, VideoOutputSettings and AudioOutputSettings,
  and typed VideoFilter and AudioFilter steps with a Custom escape hatch
  per backend.
• TranscodePlan.Resolve, the rules every backend shares: which stream a
  selector means, that cover art is not "the first video", that a dropped stream
  is taken out of "all", that burning in needs the video re-encoded, that the
  output is not one of the inputs.
• IMediaTranscoder with Validate and RunAsync, progress as
  TranscodeProgress, cancellation that deletes the partial file, and
  TranscodeRejectedException listing every issue.
• IMediaProber and MediaInfo: container, duration, streams with codec,
  language, title, default and forced flags, tags and chapters. Streams are
  numbered in container order on every backend, so an index read with one works
  with another.
• MediaCodec gained the subtitle codecs SubRip, Ass, WebVtt, MovText,
  DvdSubtitle and Pgs, and ISubtitleRecorder lets a recorder write
  SubtitleCues.
• BackendCapabilities.Probing and Transcoding, and
  MediaToolkitNetBackends.Probe, CreateTranscoder and TranscodeAsync.

FFmpeg

• H.264 falls back to libopenh264 and h264_mf, and HEVC to hevc_amf and
  hevc_mf, after the encoders tried before, so an LGPL build without x264
  still encodes H.264. A trimmed video is encoded with the input's frame rate
  as its time base, as the ffmpeg program does, rather than the container's.
• FFmpegTranscoder does everything the request can say: stream copy into any
  container libavformat writes, re-encoding through libavfilter, several inputs,
  subtitle conversion between SubRip, ASS, WebVTT and MP4 timed text, burn-in
  through libass, trims, tags and chapters. Encoders are chosen by what they
  accept (avcodec_get_supported_config on 7.1 and later), so Opus is fed
  48 kHz without being asked.
• FFmpegProber.
• FFmpegMediaReader, in the manner of FFMediaToolkit: open a file and read
  decoded frames in order or at a time, converted to the pixel format, size,
  sample format and rate asked for, and subtitle cues as text.
• FFmpegMuxer, the packet-level layer the transcoder is built on, public for
  whoever wants to mux packets directly.
• FFmpegRecorder implements ISubtitleRecorder.

GStreamer

• GStreamerProber, on GstDiscoverer.
• GStreamerTranscoder builds one pipeline from the plan: parsebin per input,
  a branch per output stream, one muxer. It copies and re-encodes video and
  audio from any number of inputs, burns in text subtitles with textoverlay,
  trims, and writes tags. GStreamerTranscoder.Describe prints the equivalent
  gst-launch-1.0 line.

mpv

• MpvProber, and MpvTranscoder on mpv's encoding mode: one video and one
  audio track re-encoded, subtitles burned in, external audio and subtitle
  files, trims and tags. MpvTranscoder.CommandLine prints the equivalent mpv
  command. A file of subtitles alone, which mpv refuses to play, is probed as
  an external subtitle file beside a blank picture.

Sample commands

probe (with a backend name), transcode (with --vencoder and --aencoder
to name an encoder outright), read, mpv-args and gst-args. encoders now
lists the video encoder MFTs as well.

dotnet run --project samples/MediaToolkitNet.Sample.Cli -- transcode in.mkv out.mp4 --vcodec H264 --size 1280x-1 --start 60 --end 120
dotnet run --project samples/MediaToolkitNet.Sample.Cli -- transcode in.mkv out.mp4 --backend gstreamer --vcodec H264 --sub in.en.srt --scodec burn
dotnet run --project samples/MediaToolkitNet.Sample.Cli -- read in.mkv 12.5 --format Bgra32 --size 640x360

Changed

• GStreamerBackend.Capabilities is Probing | Transcoding instead of None.
• MediaToolkitNetBackends.Probe and CreateTranscoder try FFmpeg, then
  GStreamer, then mpv, whatever the general registry order: FFmpeg reports and
  does the most.
• MediaToolkitNet.Mpv references MediaToolkitNet.FFmpeg, for the encoder,
  muxer and filter names mpv's encoding mode shares with it. No FFmpeg library
  is loaded unless the FFmpeg backend is used.

Fixed

These are defects of earlier releases that running the code found.

Windows

• Media Foundation could not record video at all. Two GUIDs had been written
  from memory and were wrong: MF_MT_INTERLACE_MODE, so every video media type
  lacked its interlace mode and IMFSinkWriter::SetInputMediaType failed with
  MF_E_INVALIDMEDIATYPE, and MFT_CATEGORY_VIDEO_ENCODER, so the check meant
  to explain that failure found no video encoder and blamed the machine. Every
  GUID in the source has now been checked against the Windows SDK headers, and
  the camera records to H.264 and HEVC.
• A bit rate of 0 overwrote the encoder's own default instead of leaving it
  to the encoder, as BitrateBitsPerSecond documents. Most encoders did not
  mind; Media Foundation's H.264 encoder refused to start. Encoders now keep
  their default, which for libx264 is still constant quality.

FFmpeg

• libavfilter audio graphs failed on every series. The abuffer source
  takes channel_layout, not ch_layout.
• FFmpegDecoder reported zero channels for audio, read from an option that
  is no longer set; the channel count now comes from ch_layout.

GStreamer

• Three GstMessageType values were wrong. The enum had been written from
  memory; the probe for GstMessage::type caught it the first time it ran on
  Linux. The values now come from compiling against gstmessage.h.
• GStreamerSink.NegotiatedFormat was always empty, and a pull asked for the
  wrong kind of frame dropped the sample silently. The format now comes from the
  sink pad's current caps, a mismatch throws, and planar audio is delivered.
• gst_plugin_feature_get_name is a macro in current GStreamer and was bound as
  a function; element names now come from gst_object_get_name.

Verified by running

On Debian 13 under WSL, against FFmpeg 7.1.5, GStreamer 1.26.2 and libmpv 2.5:

• All 206 unit tests and all 85 integration tests pass, three runs in a row.
  The integration tests write their inputs with this library's recorder and
  read every result back with the FFmpeg prober and reader, so no backend checks
  its own output.
• FFmpeg: a remux keeps every stream and language; SubRip converted to MP4
  timed text reads back with the same text and timing as ffmpeg's own
  conversion; Matroska to MPEG-TS and back to MP4; trims whose cues straddle the
  start and the end come out cut at both; scaling, frame rate, CRF, Opus from a
  44.1 kHz source, volume; burn-in measured by the luma of the bottom of the
  frame; VP9 with WebVTT in WebM; file names with colons, commas and quotes in a
  burn-in path.
• GStreamer: the prober numbers streams and reads their codecs, languages,
  titles and chapters as FFmpeg does; H.264, VP9 and MJPEG with Opus and FLAC
  encoded into MP4, Matroska and WebM; copies into Matroska, with and without
  a trim, and into MPEG-TS; burn-in from the same file and from a separate SRT,
  trimmed and not, checked by luma.
• mpv, through libmpv: the prober agrees with FFmpeg's, flags included, and
  reads a bare SRT or ASS file; H.264, VP9 and MJPEG with AAC, Opus and FLAC,
  scaled and trimmed; burn-in from the same file and from a separate SRT,
  trimmed and not, checked by luma; copy and several audio tracks refused with
  the reason. Running it through libmpv found that mpv_set_option_string
  does not take the sub-files-append the mpv command line does, so external
  files are now added with change-list. It also found that mpv fails quietly:
  an encoder it lacks leaves that stream out of a file it reports as written,
  and one that cannot open stalls it for ever. The transcoder now picks
  encoders from mpv's encoder-list, gives up on a stall with mpv's own error
  in the message, and checks its output for every stream it asked for.

On Windows 11, against FFmpeg 7.1.5 (an LGPL build) and libmpv 2.5:

• The unit tests, and the FFmpeg and mpv integration tests; the GStreamer ones
  skip, being Linux-only.
• FFmpeg: remux with chapters, SubRip to MP4 timed text, Matroska to MPEG-TS,
  H.264 through h264_nvenc, libopenh264 and h264_mf, trimmed and not, VP9
  with WebVTT in WebM, MJPEG with FLAC resampled to 22 050 Hz mono, an external
  ASS file added as a stream, the frame reader, and burn-in through libass both
  from the same file and from an absolute Windows path holding a comma, quotes,
  brackets and a semicolon, checked by luma.
• mpv: the prober, H.264 (through h264_mf among others), VP9 and MJPEG
  encodes, burn-in from the same file and from that awkward path. MJPEG from a limited-range source failed until the
  encoder was allowed it (strict=-1); mpv's own error now reaches the
  exception message.
• Media Foundation: camera recording to H.264 and HEVC, once the two GUIDs
  above were corrected, and audio recording to AAC as before.
• The new FFmpeg struct offsets — tags, chapters, dispositions, subtitles and
  the codec descriptor — are identical in FFmpeg 7, 8 and 9, checked against the
  headers of 7.1.5, 8.1.2 and 9.0.1, and each group is confirmed at runtime by a
  probe that disables the feature when it fails.

Known limitations

• GStreamer cannot write subtitles as a stream: its muxers take plain UTF-8 and
  its subtitle parsers produce Pango markup. The transcoder refuses and names
  FFmpeg.
• GStreamer 1.26's matroskamux writes tag values that contain spaces
  serialised ("a\ b"), and writes the input's chapters whatever the request
  says and with their original times. The transcoder warns about both. The
  discoverer does not report default or forced flags.
• mpv re-encodes what it plays, so it copies nothing, writes one video and one
  audio track, and writes no chapters. It writes its streams in the order its
  encoders start, so in MP4 the audio comes first.
• Burn-in is text only on FFmpeg and GStreamer. PGS has no FFmpeg encoder, so
  it can be copied but not produced.
• The GStreamer backend binds the Linux library names, so it reports itself
  unavailable on Windows and macOS even where GStreamer is installed.
• Media Foundation video recording needs a video encoder MFT, which Windows N
  without the Media Feature Pack does not have; the recorder says so.
• The Sample Grabber is registered for SampleCB only; a filter graph is not a
  player and renders nothing on its own.
• MacOSBackend cannot write files, FFmpegPlayer only decodes, and only the
  mpv client API is wrapped, not its render API.
• 64-bit processes only.

Packages

Package	Contents
MediaToolkitNet.All	Facade: pulls in every backend and picks one at runtime
MediaToolkitNet.Abstractions	Abstractions only; no native code, no dependencies
MediaToolkitNet.Interop	Native loading, UTF-8 marshalling, COM vtable access
MediaToolkitNet.FFmpeg	FFmpeg 7.x, 8.x or 9.x: demux, decode, encode, mux, filter graphs, probing, transcoding, frame reader
MediaToolkitNet.Mpv	libmpv: playback, filter chains, probing, transcoding
MediaToolkitNet.Windows	Media Foundation, WASAPI, DirectShow
MediaToolkitNet.Linux	V4L2, ALSA, PulseAudio
MediaToolkitNet.GStreamer	GStreamer 1.x on Linux: pipelines, probing, transcoding
MediaToolkitNet.MacOS	AVFoundation, CoreAudio

dotnet add package MediaToolkitNet.All