Filter graphs

MediaToolkit.NET 0.2.0

The filter graph release. Four native stacks can assemble a processing graph, and
all four are wrapped here: the DirectShow filter graph on Windows, libavfilter
through FFmpeg, mpv's vf and af chains, and GStreamer pipelines on Linux.

Each is wrapped in its own terms rather than behind one common abstraction. The
shapes are genuinely different — DirectShow links pins and pushes through a
callback, libavfilter takes a chain string and is pumped, mpv builds the graph
itself and only takes a property, GStreamer parses a whole pipeline and is
pulled — and flattening that would have cost more than it saved.

New package

MediaToolkitNet.GStreamer, for GStreamer 1.x on Linux.

dotnet add package MediaToolkitNet.GStreamer

A pipeline built from a gst-launch description, its state, bus and seeking, the
element-factory registry, and video and audio frames taken out of an appsink:

using var pipeline = GStreamerPipeline.Parse(
    "filesrc location=clip.mp4 ! decodebin ! videoconvert ! " +
    "video/x-raw,format=BGR ! appsink name=sink");

var sink = pipeline.GetSink("sink");
sink.DropWhenFull(true);

pipeline.Pause();                                // preroll, so the duration is known
Console.WriteLine($"{pipeline.Duration}, {sink.NegotiatedFormat}");

pipeline.Play();
while (!sink.IsEndOfStream)
{
    sink.TryPullVideo(TimeSpan.FromMilliseconds(500), (in VideoFrame frame) => Save(frame));
}

It implements no capture or playback interface of its own, because the pipeline
covers those roles in GStreamer's own terms. MediaToolkitNet.All pulls it in.

Added

DirectShow filter graph and Sample Grabber

MediaToolkitNet.Windows wrapped DirectShow for device enumeration only. It now
builds graphs: filters, pins, media types, intelligent connect, RenderFile,
transport control, seeking, bus events, and a text dump of what was built.

Thirty built-in filters are addressed by name through the DsFilter enum rather
than by class id — the splitters and parsers, the ACM and MPEG decoders, the DVD
Navigator, the tees, the renderers and the muxers.

The Sample Grabber is frame access from inside the graph:

using var graph = DirectShowGraph.Create();

var source = graph.AddSourceFilter(@"C:\clip.mp4");
var grabber = DirectShowSampleGrabber.AddTo(graph);
grabber.AcceptVideo(PixelFormat.Bgr24);          // forces a decoder and a converter in
var sink = graph.AddFilter(DsFilter.NullRenderer);

graph.Connect(source.FirstFreeOutput!, grabber.Filter.InputPins[0]);
graph.Connect(grabber.Filter.FirstFreeOutput!, sink.InputPins[0]);

grabber.VideoGrabbed += (in VideoFrame frame) =>
{
    // Borrowed, and bottom-up: Stride(0) is negative for a DIB.
    ReadOnlySpan<byte> pixels = frame.GetPlane(0);
};

graph.Run();

AcceptVideo and AcceptAudio pin the format the graph has to negotiate to, so
the builder inserts a decoder and a colour converter as needed. Bottom-up DIBs
arrive with a negative stride, which VideoFrame already understood. For a
thumbnail rather than a stream there is the buffered path: SetOneShot with
BufferSamples and CopyCurrentBuffer.

This is also the one place in the repository that implements a COM interface
instead of calling one. ISampleGrabberCB is a hand-built COM object: a struct
whose first field points at a vtable of [UnmanagedCallersOnly] function
pointers, with a GCHandle back to managed code. No [ComImport], nothing that
would cost AOT-safety.

libavfilter

FFmpegFilterGraph takes the chain in FFmpeg's own syntax — the same string the
command line takes after -vf or -af. The buffer source and sink at either end
are added for you, so the string holds only the filters in between:

var input = new VideoFormat(1920, 1080, PixelFormat.Bgr24, 30);
using var chain = FFmpegFilterGraph.ForVideo(input, "scale=640:-1,hflip");

chain.Push(decodedFrame);                        // an AVFrame, or a borrowed VideoFrame
while (chain.TryReceive((in VideoFrame frame) => Save(frame)))
{
}

chain.Flush();
Console.WriteLine(chain.OutputVideoFormat);      // 640x360 Bgr24, as negotiated

Timestamps travel in TimeSpan ticks: the source is configured with a time base
of 1/10000000 so a tick is one unit, and the sink's own time base converts the
result back. A mid-stream format change is honoured, because a sample that
carries a media type is the only notice libavfilter gives.

FFmpegFilters lists what the loaded library was built with. Unlike the
DirectShow catalogue this cannot be written down in advance: libavfilter carries
its own registry and which filters exist depends on how it was configured.

mpv filter chains

mpv builds and rebuilds the graph itself, so what is wrapped is the vf and af
properties it is set through:

player.VideoFilters.Add(MpvFilterChain.Lavfi("hqdn3d,eq=contrast=1.1"));
player.AudioFilters.Set("lavfi=[volume=0.5]");

Set, Add, Remove, Toggle and Clear, plus MpvFilterChain.Lavfi to hand
a chain written for libavfilter to mpv.

Sample commands

ds-filters, ds-graph, ds-grab, av-filters, av-filter, gst-elements
and gst-run.

dotnet run --project samples/MediaToolkitNet.Sample.Cli -- ds-filters
dotnet run --project samples/MediaToolkitNet.Sample.Cli -- ds-grab clip.mp4 5
dotnet run --project samples/MediaToolkitNet.Sample.Cli -- av-filter "scale=640:-1,hflip"
dotnet run --project samples/MediaToolkitNet.Sample.Cli -- gst-run "videotestsrc ! videoconvert ! appsink name=sink" 3

Changed

• FFmpegGeneration takes an AvFilter major version, as its seventh positional
  parameter. Code that constructs one has to pass it; nothing else moves.

Series	avutil	avcodec	avformat	swscale	swresample	avfilter
7.x	59	61	61	8	5	10
8.x	60	62	62	9	6	11
9.x	61	63	63	10	7	12

• libavfilter is optional, like libavdevice, and is looked for only in the
  major version belonging to the series that was already loaded. A graph built
  by one major against the structs of another is exactly the mismatch the version
  gate exists to prevent. When it is missing, FFmpegFilterGraph.IsAvailable is
  false and nothing else changes.

Verified by running

On Windows 11:

• All thirty catalogued DirectShow class ids were instantiated. Twenty-eight
  resolve. Line21Decoder and WavDest are not registered on current Windows —
  the operating system's doing, not a bad GUID. This matters more than it looks:
  a wrong class id is indistinguishable from an absent filter until you try it.
• A WAVE file through the Sample Grabber: 23 buffers, 122 868 samples at
  22 050 Hz. That is 5.5722 seconds, exactly the duration the graph reported, so
  the sample counts and the timestamps agree with the container.
• An H.264 file through decoder, colour converter and Sample Grabber to BGR24
  at 1728×1080: thirty frames per second sustained, a negative stride as a
  bottom-up DIB should have, and pixels read end to end of every plane rather
  than at the corners — so a wrong pointer or a wrong stride could not pass as a
  plausible frame count.

Everything 0.1.2 verified still stands: the FFmpeg backend was run on Debian 13
against FFmpeg 7.1.5 and on Windows 11 against FFmpeg 9.0.1.

Not exercised yet

Please read this before depending on it.

• libavfilter has not been run. The development machine has no FFmpeg
  installed. The two struct layouts it needs — AVFilterInOut and the head of
  AVFilter — are found at startup by parsing [in]null[out] and reading the
  labels back, and the graph refuses to build when that probe does not pass, so a
  mismatch should produce a clear error rather than memory corruption. That path
  is untested too.
• The mpv filter chains have not been run; libmpv was not installed either.
• Nothing in MediaToolkitNet.GStreamer has run at all. It binds the Linux
  library names and the development machine is Windows.

GStreamer exposes four things only as macros, with no accessor at all:
GST_MESSAGE_TYPE, GST_BUFFER_PTS, the fields of GstMapInfo and the links of
a GList. None of them is hard-coded. The message type is found by posting a
GST_MESSAGE_APPLICATION, whose value (1 << 22) does not occur in the
reference counts and pointers around it; the buffer timestamp by the run of five
GST_CLOCK_TIME_NONE qwords a fresh buffer starts with; GstMapInfo by mapping
a buffer of a size chosen here and reading that size back; GList against a
one-element list built here. A probe that fails leaves the backend disabled.

Those probes and the floating-reference handling in Parse and
GStreamerElements.Exists are the parts most worth checking first on real
hardware.

That is why this is 0.2.0 rather than 1.0. The public API may still change.

Known limitations

• The GStreamer backend reports itself unavailable on Windows and macOS even
  where GStreamer is installed, because it binds the Linux library names. Lifting
  that is a matter of the file names and nothing else.
• The Sample Grabber is registered for SampleCB only. BufferCB returns
  E_NOTIMPL: SampleCB carries the sample itself, and therefore its timestamps
  and any mid-stream format change.
• A filter graph is not a player. None of the four renders audio or video on its
  own — they hand frames back, and what happens next is the caller's.
• Media Foundation video recording still needs a video encoder MFT registered on
  the machine; editions such as Windows N without the Media Feature Pack have
  none.
• MacOSBackend cannot write files, because AVAssetWriter is not wrapped.
• FFmpegPlayer only decodes: no renderer and no audio output.
• Only the mpv client API is wrapped; the render API is not.
• 64-bit processes only.

Packages

Package	Contents
MediaToolkitNet.All	Facade: pulls in every backend and picks one at runtime
MediaToolkitNet.Abstractions	Abstractions only; no native code, no dependencies
MediaToolkitNet.Interop	Native loading, UTF-8 marshalling, COM vtable access
MediaToolkitNet.FFmpeg	FFmpeg 7.x, 8.x or 9.x backend
MediaToolkitNet.Mpv	libmpv backend
MediaToolkitNet.Windows	Media Foundation, WASAPI, DirectShow
MediaToolkitNet.Linux	V4L2, ALSA, PulseAudio
MediaToolkitNet.GStreamer	GStreamer 1.x pipelines on Linux
MediaToolkitNet.MacOS	AVFoundation, CoreAudio

No native binary ships in any package. Reports from Linux and macOS are
especially welcome.