A fix release for the GStreamer and mpv transcoders

MediaToolkit.NET 0.3.1

A fix release for the GStreamer and mpv transcoders. Both could accept an audio
request, report success, and write something other than what was asked: a
different channel count, rate or duration. Everything a request says now either
reaches the file or is reported, as an error before anything is written or as a
warning in TranscodeResult.Warnings.

The defects were found by generating the MP-MediaInfo test corpus — 220 Matroska
audio files over eleven formats — through each backend in turn and reading every
file back with the FFmpeg prober. The FFmpeg transcoder made all 220 as asked
before and after. The others:

Backend	0.3.0	0.3.1	What is still not made
GStreamer	121	193	19 TrueHD, refused; 8 AAC VBR, warned
mpv	86	175	19 TrueHD, refused; 26 PCM, warned

No public API changed.

Fixed

GStreamer

• Channel layouts past 5.1 were refused or wrong. ChannelMask holds
  FFmpeg's speaker bits and was passed to GStreamer's channel-mask unchanged,
  but the two number the speakers differently from the side pair on (FFmpeg's
  side left is bit 9, GStreamer's is 10). 7.1 and the side layouts DTS needs
  failed to link. The mask is now translated, and a speaker GStreamer has no
  position for is an error.
• DTS and RealAudio had no encoder. They now use avenc_dca (with
  strict=experimental, as libavcodec requires) and avenc_real_144.
• 24-bit PCM had no path. PcmS24 is now written raw as S24LE.
• 24-bit FLAC was refused. SampleFormat.S32 is how libavcodec carries
  24-bit FLAC; for flacenc it now becomes S24_32LE, the format that element
  takes.
• A planar sample format was refused by every encoder. libav's elements and
  matroskamux take interleaved audio only, so a planar request now pins the
  sample width alone.
• MP3 came out at half the rate asked for. At low bit rates lamemp3enc
  resamples unless the rate downstream is fixed (48 kHz at 64 kbit/s became
  24 kHz); the rate is now pinned after the encoder.
• MP2 failed at valid bit rates. twolamemp2enc refuses combinations such
  as 320 kbit/s mono that libavcodec accepts; avenc_mp2 is now preferred.
• TrueHD is refused with the reason. avenc_truehd exists, but no GStreamer
  muxer takes its output; Validate now says so instead of the pipeline
  failing to link.

mpv

• Channels, rate and sample format were ignored. mpv's encoder follows its
  audio output, which keeps the input's layout and rate; asked for six channels
  it wrote the input's two and reported success. The request now reaches
  --audio-channels (as a speaker list), --audio-samplerate and
  --audio-format.
• TrueHD and MLP in Matroska are refused. mpv stamps their frames a
  millisecond apart where each lasts 1/1200 s, so the file ran 20% long.
• PCM output carries a warning. mpv writes uncompressed audio in blocks of
  16384 samples and pads the last one with silence, so the file runs up to that
  much past the end of the input or the trim.
• The output check could not read RealAudio. mpv's Matroska demuxer does not
  know A_REAL/14_4; the check now uses the FFmpeg prober when FFmpeg is
  available.

Known limitations added

• GStreamer: AAC at a variable quality is not mapped. gst-libav does not expose
  libavcodec's qscale flag, so the setting is ignored with a warning; use a
  bit rate or the FFmpeg backend.
• GStreamer cannot write TrueHD; mpv cannot write it in Matroska. Use FFmpeg.

Add transcoding on FFmpeg, GStreamer and mpv

MediaToolkit.NET 0.3.0

The transcoding release. A file can now be probed, converted and read frame by
frame, and the conversion is described once and run by whichever backend can do
it: FFmpeg, GStreamer or mpv.

TranscodeRequest says what the output holds — which streams of which inputs,
copied or re-encoded, subtitles converted or burned in, a trim, tags and
chapters. Every backend resolves it into the same TranscodePlan, then says for
itself what it cannot do before anything is written. The facade asks FFmpeg
first, then GStreamer, then mpv, and when all three refuse the exception carries
each refusal.

Added

A common transcoding API (MediaToolkitNet.Abstractions)

await MediaToolkitNetBackends.TranscodeAsync(TranscodeRequest.Remux("in.mkv", "out.mp4"));

await MediaToolkitNetBackends.TranscodeAsync(new TranscodeRequest("out.mkv")
{
    Inputs = ["film.mkv", "film.en.srt"],
    Start = TimeSpan.FromMinutes(1),
    End = TimeSpan.FromMinutes(2),
    Streams =
    [
        OutputStream.Video(StreamSource.First(MediaStreamKind.Video),
            new VideoOutputSettings(MediaCodec.H264) { Width = 1280, Quality = 23, Speed = EncoderSpeed.Fast }),
        OutputStream.Audio(StreamSource.All(MediaStreamKind.Audio), new AudioOutputSettings(MediaCodec.Opus)),
        OutputStream.Subtitles(StreamSource.First(MediaStreamKind.Subtitle, input: 1), MediaCodec.SubRip),
    ],
    Metadata = new Dictionary<string, string> { ["title"] = "Film" },
});

• TranscodeRequest, OutputStream (copy, drop, re-encode video or audio,
  convert subtitles, burn them in), StreamSource selectors by index, by kind,
  by language and by position, VideoOutputSettings and AudioOutputSettings,
  and typed VideoFilter and AudioFilter steps with a Custom escape hatch
  per backend.
• TranscodePlan.Resolve, the rules every backend shares: which stream a
  selector means, that cover art is not "the first video", that a dropped stream
  is taken out of "all", that burning in needs the video re-encoded, that the
  output is not one of the inputs.
• IMediaTranscoder with Validate and RunAsync, progress as
  TranscodeProgress, cancellation that deletes the partial file, and
  TranscodeRejectedException listing every issue.
• IMediaProber and MediaInfo: container, duration, streams with codec,
  language, title, default and forced flags, tags and chapters. Streams are
  numbered in container order on every backend, so an index read with one works
  with another.
• MediaCodec gained the subtitle codecs SubRip, Ass, WebVtt, MovText,
  DvdSubtitle and Pgs, and ISubtitleRecorder lets a recorder write
  SubtitleCues.
• BackendCapabilities.Probing and Transcoding, and
  MediaToolkitNetBackends.Probe, CreateTranscoder and TranscodeAsync.

FFmpeg

• H.264 falls back to libopenh264 and h264_mf, and HEVC to hevc_amf and
  hevc_mf, after the encoders tried before, so an LGPL build without x264
  still encodes H.264. A trimmed video is encoded with the input's frame rate
  as its time base, as the ffmpeg program does, rather than the container's.
• FFmpegTranscoder does everything the request can say: stream copy into any
  container libavformat writes, re-encoding through libavfilter, several inputs,
  subtitle conversion between SubRip, ASS, WebVTT and MP4 timed text, burn-in
  through libass, trims, tags and chapters. Encoders are chosen by what they
  accept (avcodec_get_supported_config on 7.1 and later), so Opus is fed
  48 kHz without being asked.
• FFmpegProber.
• FFmpegMediaReader, in the manner of FFMediaToolkit: open a file and read
  decoded frames in order or at a time, converted to the pixel format, size,
  sample format and rate asked for, and subtitle cues as text.
• FFmpegMuxer, the packet-level layer the transcoder is built on, public for
  whoever wants to mux packets directly.
• FFmpegRecorder implements ISubtitleRecorder.

GStreamer

• GStreamerProber, on GstDiscoverer.
• GStreamerTranscoder builds one pipeline from the plan: parsebin per input,
  a branch per output stream, one muxer. It copies and re-encodes video and
  audio from any number of inputs, burns in text subtitles with textoverlay,
  trims, and writes tags. GStreamerTranscoder.Describe prints the equivalent
  gst-launch-1.0 line.

mpv

• MpvProber, and MpvTranscoder on mpv's encoding mode: one video and one
  audio track re-encoded, subtitles burned in, external audio and subtitle
  files, trims and tags. MpvTranscoder.CommandLine prints the equivalent mpv
  command. A file of subtitles alone, which mpv refuses to play, is probed as
  an external subtitle file beside a blank picture.

Sample commands

probe (with a backend name), transcode (with --vencoder and --aencoder
to name an encoder outright), read, mpv-args and gst-args. encoders now
lists the video encoder MFTs as well.

dotnet run --project samples/MediaToolkitNet.Sample.Cli -- transcode in.mkv out.mp4 --vcodec H264 --size 1280x-1 --start 60 --end 120
dotnet run --project samples/MediaToolkitNet.Sample.Cli -- transcode in.mkv out.mp4 --backend gstreamer --vcodec H264 --sub in.en.srt --scodec burn
dotnet run --project samples/MediaToolkitNet.Sample.Cli -- read in.mkv 12.5 --format Bgra32 --size 640x360

Changed

• GStreamerBackend.Capabilities is Probing | Transcoding instead of None.
• MediaToolkitNetBackends.Probe and CreateTranscoder try FFmpeg, then
  GStreamer, then mpv, whatever the general registry order: FFmpeg reports and
  does the most.
• MediaToolkitNet.Mpv references MediaToolkitNet.FFmpeg, for the encoder,
  muxer and filter names mpv's encoding mode shares with it. No FFmpeg library
  is loaded unless the FFmpeg backend is used.

Fixed

These are defects of earlier releases that running the code found.

Windows

• Media Foundation could not record video at all. Two GUIDs had been written
  from memory and were wrong: MF_MT_INTERLACE_MODE, so every video media type
  lacked its interlace mode and IMFSinkWriter::SetInputMediaType failed with
  MF_E_INVALIDMEDIATYPE, and MFT_CATEGORY_VIDEO_ENCODER, so the check meant
  to explain that failure found no video encoder and blamed the machine. Every
  GUID in the source has now been checked against the Windows SDK headers, and
  the camera records to H.264 and HEVC.
• A bit rate of 0 overwrote the encoder's own default instead of leaving it
  to the encoder, as BitrateBitsPerSecond documents. Most encoders did not
  mind; Media Foundation's H.264 encoder refused to start. Encoders now keep
  their default, which for libx264 is still constant quality.

FFmpeg

• libavfilter audio graphs failed on every series. The abuffer source
  takes channel_layout, not ch_layout.
• FFmpegDecoder reported zero channels for audio, read from an option that
  is no longer set; the channel count now comes from ch_layout.

GStreamer

• Three GstMessageType values were wrong. The enum had been written from
  memory; the probe for GstMessage::type caught it the first time it ran on
  Linux. The values now come from compiling against gstmessage.h.
• GStreamerSink.NegotiatedFormat was always empty, and a pull asked for the
  wrong kind of frame dropped the sample silently. The format now comes from the
  sink pad's current caps, a mismatch throws, and planar audio is delivered.
• gst_plugin_feature_get_name is a macro in current GStreamer and was bound as
  a function; element names now come from gst_object_get_name.

Verified by running

On Debian 13 under WSL, against FFmpeg 7.1.5, GStreamer 1.26.2 and libmpv 2.5:

• All 206 unit tests and all 85 integration tests pass, three runs in a row.
  The integration tests write their inputs with this library's recorder and
  read every result back with the FFmpeg prober and reader, so no backend checks
  its own output.
• FFmpeg: a remux keeps every stream and language; SubRip converted to MP4
  timed text reads back with the same text and timing as ffmpeg's own
  conversion; Matroska to MPEG-TS and back to MP4; trims whose cues straddle the
  start and the end come out cut at both; scaling, frame rate, CRF, Opus from a
  44.1 kHz source, volume; burn-in measured by the luma of the bottom of the
  frame; VP9 with WebVTT in WebM; file names with colons, commas and quotes in a
  burn-in path.
• GStreamer: the prober numbers streams and reads their codecs, languages,
  titles and chapters as FFmpeg does; H.264, VP9 and MJPEG with Opus and FLAC
  encoded into MP4, Matroska and WebM; copies into Matroska, with and without
  a trim, and into MPEG-TS; burn-in from the same file and from a separate SRT,
  trimmed and not, checked by luma.
• mpv, through libmpv: the prober agrees with FFmpeg's, flags included, and
  reads a bare SRT or ASS file; H.264, VP9 and MJPEG with AAC, Opus and FLAC,
  scaled and trimmed; burn-in from the same file and from a separate SRT,
  trimmed and not, checked by luma; copy and several audio tracks refused with
  the reason. Running it through libmpv found that mpv_set_option_string
  does not take the sub-files-append the mpv command line does, so external
  files are now added with change-list. It also found that mpv fails quietly:
  an encoder it lacks leaves that stream out of a file it reports as written,
  and one that cannot open stalls it for ever. The transcoder now picks
  encoders from mpv's encoder-list, gives up on a stall with mpv's own error
  in the message, and checks its output for every stream it asked for.

On Windows 11, against FFmpeg 7.1.5 (an LGPL build) and libmpv 2.5:

• The unit tests, and the FFmpeg and mpv integr...

Filter graphs

MediaToolkit.NET 0.2.0

The filter graph release. Four native stacks can assemble a processing graph, and
all four are wrapped here: the DirectShow filter graph on Windows, libavfilter
through FFmpeg, mpv's vf and af chains, and GStreamer pipelines on Linux.

Each is wrapped in its own terms rather than behind one common abstraction. The
shapes are genuinely different — DirectShow links pins and pushes through a
callback, libavfilter takes a chain string and is pumped, mpv builds the graph
itself and only takes a property, GStreamer parses a whole pipeline and is
pulled — and flattening that would have cost more than it saved.

New package

MediaToolkitNet.GStreamer, for GStreamer 1.x on Linux.

dotnet add package MediaToolkitNet.GStreamer

A pipeline built from a gst-launch description, its state, bus and seeking, the
element-factory registry, and video and audio frames taken out of an appsink:

using var pipeline = GStreamerPipeline.Parse(
    "filesrc location=clip.mp4 ! decodebin ! videoconvert ! " +
    "video/x-raw,format=BGR ! appsink name=sink");

var sink = pipeline.GetSink("sink");
sink.DropWhenFull(true);

pipeline.Pause();                                // preroll, so the duration is known
Console.WriteLine($"{pipeline.Duration}, {sink.NegotiatedFormat}");

pipeline.Play();
while (!sink.IsEndOfStream)
{
    sink.TryPullVideo(TimeSpan.FromMilliseconds(500), (in VideoFrame frame) => Save(frame));
}

It implements no capture or playback interface of its own, because the pipeline
covers those roles in GStreamer's own terms. MediaToolkitNet.All pulls it in.

Added

DirectShow filter graph and Sample Grabber

MediaToolkitNet.Windows wrapped DirectShow for device enumeration only. It now
builds graphs: filters, pins, media types, intelligent connect, RenderFile,
transport control, seeking, bus events, and a text dump of what was built.

Thirty built-in filters are addressed by name through the DsFilter enum rather
than by class id — the splitters and parsers, the ACM and MPEG decoders, the DVD
Navigator, the tees, the renderers and the muxers.

The Sample Grabber is frame access from inside the graph:

using var graph = DirectShowGraph.Create();

var source = graph.AddSourceFilter(@"C:\clip.mp4");
var grabber = DirectShowSampleGrabber.AddTo(graph);
grabber.AcceptVideo(PixelFormat.Bgr24);          // forces a decoder and a converter in
var sink = graph.AddFilter(DsFilter.NullRenderer);

graph.Connect(source.FirstFreeOutput!, grabber.Filter.InputPins[0]);
graph.Connect(grabber.Filter.FirstFreeOutput!, sink.InputPins[0]);

grabber.VideoGrabbed += (in VideoFrame frame) =>
{
    // Borrowed, and bottom-up: Stride(0) is negative for a DIB.
    ReadOnlySpan<byte> pixels = frame.GetPlane(0);
};

graph.Run();

AcceptVideo and AcceptAudio pin the format the graph has to negotiate to, so
the builder inserts a decoder and a colour converter as needed. Bottom-up DIBs
arrive with a negative stride, which VideoFrame already understood. For a
thumbnail rather than a stream there is the buffered path: SetOneShot with
BufferSamples and CopyCurrentBuffer.

This is also the one place in the repository that implements a COM interface
instead of calling one. ISampleGrabberCB is a hand-built COM object: a struct
whose first field points at a vtable of [UnmanagedCallersOnly] function
pointers, with a GCHandle back to managed code. No [ComImport], nothing that
would cost AOT-safety.

libavfilter

FFmpegFilterGraph takes the chain in FFmpeg's own syntax — the same string the
command line takes after -vf or -af. The buffer source and sink at either end
are added for you, so the string holds only the filters in between:

var input = new VideoFormat(1920, 1080, PixelFormat.Bgr24, 30);
using var chain = FFmpegFilterGraph.ForVideo(input, "scale=640:-1,hflip");

chain.Push(decodedFrame);                        // an AVFrame, or a borrowed VideoFrame
while (chain.TryReceive((in VideoFrame frame) => Save(frame)))
{
}

chain.Flush();
Console.WriteLine(chain.OutputVideoFormat);      // 640x360 Bgr24, as negotiated

Timestamps travel in TimeSpan ticks: the source is configured with a time base
of 1/10000000 so a tick is one unit, and the sink's own time base converts the
result back. A mid-stream format change is honoured, because a sample that
carries a media type is the only notice libavfilter gives.

FFmpegFilters lists what the loaded library was built with. Unlike the
DirectShow catalogue this cannot be written down in advance: libavfilter carries
its own registry and which filters exist depends on how it was configured.

mpv filter chains

mpv builds and rebuilds the graph itself, so what is wrapped is the vf and af
properties it is set through:

player.VideoFilters.Add(MpvFilterChain.Lavfi("hqdn3d,eq=contrast=1.1"));
player.AudioFilters.Set("lavfi=[volume=0.5]");

Set, Add, Remove, Toggle and Clear, plus MpvFilterChain.Lavfi to hand
a chain written for libavfilter to mpv.

Sample commands

ds-filters, ds-graph, ds-grab, av-filters, av-filter, gst-elements
and gst-run.

dotnet run --project samples/MediaToolkitNet.Sample.Cli -- ds-filters
dotnet run --project samples/MediaToolkitNet.Sample.Cli -- ds-grab clip.mp4 5
dotnet run --project samples/MediaToolkitNet.Sample.Cli -- av-filter "scale=640:-1,hflip"
dotnet run --project samples/MediaToolkitNet.Sample.Cli -- gst-run "videotestsrc ! videoconvert ! appsink name=sink" 3

Changed

• FFmpegGeneration takes an AvFilter major version, as its seventh positional
  parameter. Code that constructs one has to pass it; nothing else moves.

Series	avutil	avcodec	avformat	swscale	swresample	avfilter
7.x	59	61	61	8	5	10
8.x	60	62	62	9	6	11
9.x	61	63	63	10	7	12

• libavfilter is optional, like libavdevice, and is looked for only in the
  major version belonging to the series that was already loaded. A graph built
  by one major against the structs of another is exactly the mismatch the version
  gate exists to prevent. When it is missing, FFmpegFilterGraph.IsAvailable is
  false and nothing else changes.

Verified by running

On Windows 11:

• All thirty catalogued DirectShow class ids were instantiated. Twenty-eight
  resolve. Line21Decoder and WavDest are not registered on current Windows —
  the operating system's doing, not a bad GUID. This matters more than it looks:
  a wrong class id is indistinguishable from an absent filter until you try it.
• A WAVE file through the Sample Grabber: 23 buffers, 122 868 samples at
  22 050 Hz. That is 5.5722 seconds, exactly the duration the graph reported, so
  the sample counts and the timestamps agree with the container.
• An H.264 file through decoder, colour converter and Sample Grabber to BGR24
  at 1728×1080: thirty frames per second sustained, a negative stride as a
  bottom-up DIB should have, and pixels read end to end of every plane rather
  than at the corners — so a wrong pointer or a wrong stride could not pass as a
  plausible frame count.

Everything 0.1.2 verified still stands: the FFmpeg backend was run on Debian 13
against FFmpeg 7.1.5 and on Windows 11 against FFmpeg 9.0.1.

Not exercised yet

Please read this before depending on it.

• libavfilter has not been run. The development machine has no FFmpeg
  installed. The two struct layouts it needs — AVFilterInOut and the head of
  AVFilter — are found at startup by parsing [in]null[out] and reading the
  labels back, and the graph refuses to build when that probe does not pass, so a
  mismatch should produce a clear error rather than memory corruption. That path
  is untested too.
• The mpv filter chains have not been run; libmpv was not installed either.
• Nothing in MediaToolkitNet.GStreamer has run at all. It binds the Linux
  library names and the development machine is Windows.

GStreamer exposes four things only as macros, with no accessor at all:
GST_MESSAGE_TYPE, GST_BUFFER_PTS, the fields of GstMapInfo and the links of
a GList. None of them is hard-coded. The message type is found by posting a
GST_MESSAGE_APPLICATION, whose value (1 << 22) does not occur in the
reference counts and pointers around it; the buffer timestamp by the run of five
GST_CLOCK_TIME_NONE qwords a fresh buffer starts with; GstMapInfo by mapping
a buffer of a size chosen here and reading that size back; GList against a
one-element list built here. A probe that fails leaves the backend disabled.

Those probes and the floating-reference handling in Parse and
GStreamerElements.Exists are the parts most worth checking first on real
hardware.

That is why this is 0.2.0 rather than 1.0. The public API may still change.

Known limitations

• The GStreamer backend reports itself unavailable on Windows and macOS even
  where GStreamer is installed, because it binds the Linux library names. Lifting
  that is a matter of the file names and nothing else.
• The Sample Grabber is registered for SampleCB only. BufferCB returns
  E_NOTIMPL: SampleCB carries the sample itself, and therefore its timestamps
  and any mid-stream format change.
• A filter graph is not a player. None of the four renders audio or video on its
  own — they hand frames back, and what happens next is the caller's.
• Media Foundation video recording still needs a video encoder MFT registered on
  the machine; editions such as Windows N without the Media Feature Pack have
  none.
• MacOSBackend cannot write files, because AVAssetWriter is not wrapped.
• `FFmpe...

Advanced support FFMPEG

MediaToolkit.NET 0.1.2

The FFmpeg backend release. Audio encoding did not work at all in 0.1.1 — on any
FFmpeg version — and the fix for that turned up a second defect underneath it, in
the struct offsets themselves. Both are fixed here, FFmpeg 8.x and 9.x are now
supported alongside 7.x, and the whole of it is covered by a test suite that
checks every offset against the library that is actually installed.

Fixed

Audio encoding never worked. The channel count was set on the encoder through
the ac AVOption, which has not existed on AVCodecContext since FFmpeg 7
removed the old channel API. The call returned "option not found", the return
value was not checked, and avcodec_open2 then refused the encoder with
Encoder requires channel layout to be set. The same mistake sat in the routine
that probes the AVFrame audio fields at startup, so AudioFrameLayoutVerified
stayed false and every audio stream was rejected with
audio encoding is unavailable. The layout now travels through ch_layout.

The AVStream offsets were wrong — in every FFmpeg from 6 to 9. codecpar
was read at 208 where it has always lived at 16, and time_base, start_time,
duration and avg_frame_rate were each eight bytes short. This was not a
reading fault only: the muxer writes codec parameters through codecpar, so
adding any stream to an output wrote through whatever the rubbish at +208 pointed
at. A new stream is now checked before anything is written through it.

Channel layouts were guessed arithmetically. (1 << channels) - 1 gives 3.1
for four channels rather than quad, and an invalid mask for eight rather than
7.1. Both were accepted by the AC-3 encoder, which then wrote a file whose
speakers were in the wrong places. The layout of a bare channel count now comes
from av_channel_layout_default.

Encoder flags were assigned rather than accumulated, so setting any second
flag dropped the global-header bit.

Matroska audio did not ask for a global header. .mka and .mks are the
same muxer as .mkv and are now treated as such.

Added

FFmpeg 8.x and 9.x, beside 7.x. The loader reads the versions it actually
got, matches them against a table of release series, and refuses a mixture that
belongs to none of them; the two struct offsets that differ between series come
from the matched series.

Series	avutil	avcodec	avformat	swscale	swresample
7.x	59	61	61	8	5
8.x	60	62	62	9	6
9.x	61	63	63	10	7

AC-3 and DTS, as MediaCodec.Ac3 and MediaCodec.Dts. The DCA encoder is
marked experimental in FFmpeg and is opened as such without the caller having to
say so.

PCM at every width: MediaCodec.PcmU8, PcmS16, PcmS24 and PcmS32.
MediaCodec.Pcm remains a synonym for 16-bit.

Variable bitrate: AudioEncodingSettings.Quality, on the scale -q:a uses.

Per-stream encoder settings: EncoderName names an encoder outright and
overrides the codec, and Options passes AVOptions through to that stream alone.
FFmpegRecorder.EncoderOptions still reaches every stream, and a stream's own
options are applied afterwards and win.

Stated channel layouts: the new ChannelLayout constants and
AudioFormat.ChannelMask. Six channels are 5.1 twice over — the back
arrangement and the side one — and only a mask tells an encoder which.

Tests: 134 unit tests, plus 43 integration tests that check every struct
field this binding reads by offset against the installed library. Each check
establishes its expected value by a route that does not use the offset it is
checking, so a mismatch fails by name instead of corrupting memory. They skip
themselves where FFmpeg is not loadable.

Changed

• Six channels with no stated layout now default to 5.1 side, which is what
  av_channel_layout_default gives, rather than 5.1 back. Ask for
  ChannelLayout.FivePoint1Back where the back arrangement is meant.
• AudioFormat takes a fourth, optional ChannelMask. Source-compatible;
  recompile against it.
• AbiLayout.CodecContextTimeBase and CodecContextPixFmt are properties rather
  than constants, because the series decides them.
• FFmpegLibraries.AvUtilMajor, AvCodecMajor and AvFormatMajor are gone.
  FFmpegLibraries.Generation and FFmpegGeneration.Known replace them.
• AbiLayout.PrepareAudioFrame takes the channel mask.

Requirements

• .NET 10
• A 64-bit process. Every mapped struct layout assumes a 64-bit pointer.
• MediaToolkitNet.FFmpeg needs FFmpeg 7.x, 8.x or 9.x at runtime. No native
  binary ships in any package.
• MediaToolkitNet.Mpv needs libmpv at runtime.
• The platform backends only call libraries that are already part of the
  operating system.

Verified by running

On Windows 11 against FFmpeg 9.0.1 (avutil 61 / avcodec 63 / avformat 63):

• AC-3 at one, four and six channels; DTS; AAC at a fixed rate and at a quality;
  PCM at 8, 16, 24 and 32 bits — each encoded into Matroska and read back through
  this library's own demuxer as the codec, rate and channel count it was given;
• MJPEG video encoded and muxed, which is the path that writes through
  AVStream::codecpar;
• every hard-coded offset checked against the running library.

The offset tests were themselves checked by breaking three offsets on purpose.
AVStream::codecpar back at 208 failed twenty tests and corrupted nothing;
AVCodecContext::time_base moved to FFmpeg 6's value failed eight; and
pix_fmt moved to FFmpeg 7's value failed exactly one, the test named for that
field — the startup self-check missed that one, because the neighbouring field in
FFmpeg 9 happens to hold the same sentinel.

Not exercised yet

• FFmpeg 7.x and 8.x have not been run. Their entries in the series table
  were read with offsetof against the headers of n7.1.5 and n8.1.2, and the
  fields they name are the same ones 9.x uses, but no library of those series was
  loaded. Running dotnet test on a machine that has one is the quickest way to
  confirm them, and a report either way is welcome.
• The Linux and macOS backends have still never been run, nor has the mpv
  backend.
• Media Foundation video recording is still untested: the development machine
  has no video encoder MFT registered. Audio recording on that path does work.

This is why the release is still 0.1.x. The public API may still change.

Known limitations

• An encoder's own constraints are the caller's to meet: the DCA encoder takes
  only the side arrangement of 5.1 and refuses stereo below 384 kbps, and AC-3
  does not do 7.1 at all. The failure names the encoder, the sample rate and the
  layout that was asked for. The FFmpeg command line hides this behind an
  automatic conversion filter; this API does not.
• Matroska stores no channel mask, so a PCM stream read back from it reports an
  unspecified layout, and AC-3 or DTS report whatever their own bitstream says.
  The mask matters for encoding, not for what the container records.
• Media Foundation video recording needs a video encoder MFT registered on the
  machine.
• MacOSBackend cannot write files — AVAssetWriter is not wrapped.
• FFmpegPlayer only decodes: no renderer, no audio output.
• Only the mpv client API is wrapped; the render API is not.
• DirectShow is wrapped for device enumeration only; no filter graph.

Trying it

dotnet add package MediaToolkitNet.All

// Five point one Dolby Digital into a Matroska audio file. The layout is stated
// because six channels alone do not say where the speakers are.
using var recorder = MediaToolkitNetBackends.CreateRecorder("surround.mka");
var format = new AudioFormat(48000, 6, SampleFormat.S16, ChannelLayout.FivePoint1Back);
var stream = recorder.AddAudioStream(new AudioEncodingSettings(format, MediaCodec.Ac3, 448_000));
recorder.Start();
// … recorder.WriteAudio(stream, frame) …
recorder.Stop();

# The ABI tests, against whichever FFmpeg is installed.
dotnet test

# Or against a particular build.
MEDIATOOLKITNET_FFMPEG_DIR=/opt/ffmpeg-9/lib dotnet test

Full documentation: README

First public release

MediaToolkit.NET 0.1.1

First usable release: low-level wrappers over the native media stacks for
.NET 10, plus one cross-platform API on top of them.

There is no 0.1.0. Two packages of that version reached nuget.org before
the rest of the set was rejected. nuget.org refuses an id that matches an
existing one once . and - are stripped, so MediaToolkitNet.Core collided
with MediaToolkit.NetCore and the facade MediaToolkitNet collided with
MediaToolkit.NET. They are now MediaToolkitNet.Abstractions and
MediaToolkitNet.All. MediaToolkitNet.Interop 0.1.0 and
MediaToolkitNet.FFmpeg 0.1.0 depend on an id that never shipped, so they are
unlisted and must not be used.

All interop goes through [LibraryImport] and unmanaged function pointers. COM
on Windows is called directly through vtable slots, without [ComImport] or the
built-in COM marshaller, which keeps the assemblies AOT- and trim-safe and keeps
the exact native signature visible at every call site.

Install

dotnet add package MediaToolkitNet.All

That pulls in every backend and picks one at runtime. Take a single backend
instead when you do not want the rest:

dotnet add package MediaToolkitNet.Windows

Package	Contents
MediaToolkitNet.All	Facade: backend registry and per-platform selection
MediaToolkitNet.Abstractions	Abstractions only; no native code, no dependencies
MediaToolkitNet.Interop	Native loading, UTF-8 marshalling, COM vtable access
MediaToolkitNet.FFmpeg	FFmpeg 7.x: demux, decode, encode, mux
MediaToolkitNet.Mpv	libmpv client API
MediaToolkitNet.Windows	Media Foundation, WASAPI, DirectShow
MediaToolkitNet.Linux	V4L2, ALSA, PulseAudio
MediaToolkitNet.MacOS	AVFoundation, CoreAudio

What is in it

Capability	Windows	Linux	macOS	FFmpeg	mpv
Device enumeration	MF + WASAPI + DShow	ALSA + V4L2	CoreAudio + AVF	avdevice	—
Video capture	IMFSourceReader	V4L2 mmap	AVCaptureSession	—	—
Audio capture	WASAPI (+ loopback)	ALSA / PulseAudio	AudioQueue	—	—
Audio render	WASAPI	ALSA / PulseAudio	AudioQueue	—	—
Playback	—	—	AVPlayer	decode to frames	full
Recording to file	IMFSinkWriter	—	—	encode + mux	—

A dash means the backend does not take that role: FFmpeg owns no audio output,
mpv never hands frames back, and on Linux and macOS writing files is the FFmpeg
backend's job.

Requirements

• .NET 10
• A 64-bit process. Every mapped struct layout assumes a 64-bit pointer.
• MediaToolkitNet.FFmpeg needs FFmpeg 7.x present at runtime
  (avutil 59 / avcodec 61 / avformat 61). No native binary ships in any package.
• MediaToolkitNet.Mpv needs libmpv present at runtime.
• The platform backends only call libraries that are already part of the
  operating system.

Verified by running

Exercised against real hardware on Windows 11:

• device enumeration through Media Foundation, WASAPI and DirectShow, including
  default-endpoint flags;
• video capture from a USB camera, 640x480 NV12 at 30 fps, sustained ~23 fps
  delivered to the frame handler;
• audio capture through WASAPI at 48 kHz stereo float;
• recording live audio to AAC/MP4 through IMFSinkWriter, producing a
  structurally complete MP4 — ftyp / mdat / moov, brand mp42 — which
  confirms the writer was finalised rather than merely fed.

The Windows COM vtable slot tables were corrected twice during development by
running them, not by reading them: a wrong slot number compiles, raises no
exception, and silently calls the neighbouring method.

Not exercised yet

Please read this before depending on it.

• The Linux and macOS backends have never been run. They compile and are
  guarded at runtime, but the V4L2 struct sizes and the Objective-C runtime
  layout have not been confirmed on real hardware.
• The FFmpeg and mpv backends have never been run, because those native
  libraries were not installed on the development machine. The FFmpeg ABI
  self-check refuses to proceed on a layout it cannot confirm, so a mismatch
  should produce a clear error rather than memory corruption, but that path is
  untested too.
• Media Foundation video recording is untested: the development machine has
  no video encoder MFT registered at all. MediaFoundationRecorder detects that
  case and reports it plainly. Audio recording on that path does work.

This is why the release is 0.1.1. The public API may still change.

Known limitations

• Media Foundation video recording needs a video encoder MFT registered on the
  machine. Editions such as Windows N without the Media Feature Pack have none;
  the encoders sample command reports what a machine actually has.
• MacOSBackend cannot write files — AVAssetWriter is not wrapped. Use FFmpeg.
• FFmpegPlayer only decodes: no renderer, no audio output.
• Only the mpv client API is wrapped; the render API is not.
• DirectShow is wrapped for device enumeration only; no filter graph.

Design notes

Frames are borrowed. VideoFrame and AudioFrame are ref structs over
memory the backend owns, valid only inside the handler that received them. The
compiler refuses to let you store one. Zero copies and zero allocations per
frame; copy the data out if you need to keep it.

FFmpeg is pinned to 7.x, and checks itself. FFmpeg keeps no stable ABI for
its public structs across majors, so every direct field access lives in a single
file, the loader gates on the major versions, and the layout is validated at
startup against documented post-allocation defaults. The AVCodecParameters
offsets and the AVFrame audio fields are discovered by runtime probing rather
than hard-coded, and codecs are resolved by name instead of by the unstable
AVCodecID values.

Trying it

The sample CLI covers every capability and is how the library was validated:

dotnet run --project samples/MediaToolkitNet.Sample.Cli -- backends
dotnet run --project samples/MediaToolkitNet.Sample.Cli -- devices
dotnet run --project samples/MediaToolkitNet.Sample.Cli -- rec-video 2
dotnet run --project samples/MediaToolkitNet.Sample.Cli -- rec-audio-file 3 out.m4a

Reports from Linux and macOS are especially welcome.

Full documentation: README