Skip to content

Advanced support FFMPEG

Choose a tag to compare

View all tags
@yartat yartat released this 17 Sep 20:31
· 7 commits to main since this release
0.1.2
f00a80a

MediaToolkit.NET 0.1.2

The FFmpeg backend release. Audio encoding did not work at all in 0.1.1 — on any
FFmpeg version — and the fix for that turned up a second defect underneath it, in
the struct offsets themselves. Both are fixed here, FFmpeg 8.x and 9.x are now
supported alongside 7.x, and the whole of it is covered by a test suite that
checks every offset against the library that is actually installed.

Fixed

Audio encoding never worked. The channel count was set on the encoder through
the ac AVOption, which has not existed on AVCodecContext since FFmpeg 7
removed the old channel API. The call returned "option not found", the return
value was not checked, and avcodec_open2 then refused the encoder with
Encoder requires channel layout to be set. The same mistake sat in the routine
that probes the AVFrame audio fields at startup, so AudioFrameLayoutVerified
stayed false and every audio stream was rejected with
audio encoding is unavailable. The layout now travels through ch_layout.

The AVStream offsets were wrong — in every FFmpeg from 6 to 9. codecpar
was read at 208 where it has always lived at 16, and time_base, start_time,
duration and avg_frame_rate were each eight bytes short. This was not a
reading fault only: the muxer writes codec parameters through codecpar, so
adding any stream to an output wrote through whatever the rubbish at +208 pointed
at. A new stream is now checked before anything is written through it.

Channel layouts were guessed arithmetically. (1 << channels) - 1 gives 3.1
for four channels rather than quad, and an invalid mask for eight rather than
7.1. Both were accepted by the AC-3 encoder, which then wrote a file whose
speakers were in the wrong places. The layout of a bare channel count now comes
from av_channel_layout_default.

Encoder flags were assigned rather than accumulated, so setting any second
flag dropped the global-header bit.

Matroska audio did not ask for a global header. .mka and .mks are the
same muxer as .mkv and are now treated as such.

Added

FFmpeg 8.x and 9.x, beside 7.x. The loader reads the versions it actually
got, matches them against a table of release series, and refuses a mixture that
belongs to none of them; the two struct offsets that differ between series come
from the matched series.

Series avutil avcodec avformat swscale swresample
7.x 59 61 61 8 5
8.x 60 62 62 9 6
9.x 61 63 63 10 7

AC-3 and DTS, as MediaCodec.Ac3 and MediaCodec.Dts. The DCA encoder is
marked experimental in FFmpeg and is opened as such without the caller having to
say so.

PCM at every width: MediaCodec.PcmU8, PcmS16, PcmS24 and PcmS32.
MediaCodec.Pcm remains a synonym for 16-bit.

Variable bitrate: AudioEncodingSettings.Quality, on the scale -q:a uses.

Per-stream encoder settings: EncoderName names an encoder outright and
overrides the codec, and Options passes AVOptions through to that stream alone.
FFmpegRecorder.EncoderOptions still reaches every stream, and a stream's own
options are applied afterwards and win.

Stated channel layouts: the new ChannelLayout constants and
AudioFormat.ChannelMask. Six channels are 5.1 twice over — the back
arrangement and the side one — and only a mask tells an encoder which.

Tests: 134 unit tests, plus 43 integration tests that check every struct
field this binding reads by offset against the installed library. Each check
establishes its expected value by a route that does not use the offset it is
checking, so a mismatch fails by name instead of corrupting memory. They skip
themselves where FFmpeg is not loadable.

Changed

  • Six channels with no stated layout now default to 5.1 side, which is what
    av_channel_layout_default gives, rather than 5.1 back. Ask for
    ChannelLayout.FivePoint1Back where the back arrangement is meant.
  • AudioFormat takes a fourth, optional ChannelMask. Source-compatible;
    recompile against it.
  • AbiLayout.CodecContextTimeBase and CodecContextPixFmt are properties rather
    than constants, because the series decides them.
  • FFmpegLibraries.AvUtilMajor, AvCodecMajor and AvFormatMajor are gone.
    FFmpegLibraries.Generation and FFmpegGeneration.Known replace them.
  • AbiLayout.PrepareAudioFrame takes the channel mask.

Requirements

  • .NET 10
  • A 64-bit process. Every mapped struct layout assumes a 64-bit pointer.
  • MediaToolkitNet.FFmpeg needs FFmpeg 7.x, 8.x or 9.x at runtime. No native
    binary ships in any package.
  • MediaToolkitNet.Mpv needs libmpv at runtime.
  • The platform backends only call libraries that are already part of the
    operating system.

Verified by running

On Windows 11 against FFmpeg 9.0.1 (avutil 61 / avcodec 63 / avformat 63):

  • AC-3 at one, four and six channels; DTS; AAC at a fixed rate and at a quality;
    PCM at 8, 16, 24 and 32 bits — each encoded into Matroska and read back through
    this library's own demuxer as the codec, rate and channel count it was given;
  • MJPEG video encoded and muxed, which is the path that writes through
    AVStream::codecpar;
  • every hard-coded offset checked against the running library.

The offset tests were themselves checked by breaking three offsets on purpose.
AVStream::codecpar back at 208 failed twenty tests and corrupted nothing;
AVCodecContext::time_base moved to FFmpeg 6's value failed eight; and
pix_fmt moved to FFmpeg 7's value failed exactly one, the test named for that
field — the startup self-check missed that one, because the neighbouring field in
FFmpeg 9 happens to hold the same sentinel.

Not exercised yet

  • FFmpeg 7.x and 8.x have not been run. Their entries in the series table
    were read with offsetof against the headers of n7.1.5 and n8.1.2, and the
    fields they name are the same ones 9.x uses, but no library of those series was
    loaded. Running dotnet test on a machine that has one is the quickest way to
    confirm them, and a report either way is welcome.
  • The Linux and macOS backends have still never been run, nor has the mpv
    backend.
  • Media Foundation video recording is still untested: the development machine
    has no video encoder MFT registered. Audio recording on that path does work.

This is why the release is still 0.1.x. The public API may still change.

Known limitations

  • An encoder's own constraints are the caller's to meet: the DCA encoder takes
    only the side arrangement of 5.1 and refuses stereo below 384 kbps, and AC-3
    does not do 7.1 at all. The failure names the encoder, the sample rate and the
    layout that was asked for. The FFmpeg command line hides this behind an
    automatic conversion filter; this API does not.
  • Matroska stores no channel mask, so a PCM stream read back from it reports an
    unspecified layout, and AC-3 or DTS report whatever their own bitstream says.
    The mask matters for encoding, not for what the container records.
  • Media Foundation video recording needs a video encoder MFT registered on the
    machine.
  • MacOSBackend cannot write files — AVAssetWriter is not wrapped.
  • FFmpegPlayer only decodes: no renderer, no audio output.
  • Only the mpv client API is wrapped; the render API is not.
  • DirectShow is wrapped for device enumeration only; no filter graph.

Trying it

dotnet add package MediaToolkitNet.All
// Five point one Dolby Digital into a Matroska audio file. The layout is stated
// because six channels alone do not say where the speakers are.
using var recorder = MediaToolkitNetBackends.CreateRecorder("surround.mka");
var format = new AudioFormat(48000, 6, SampleFormat.S16, ChannelLayout.FivePoint1Back);
var stream = recorder.AddAudioStream(new AudioEncodingSettings(format, MediaCodec.Ac3, 448_000));
recorder.Start();
// … recorder.WriteAudio(stream, frame) …
recorder.Stop();
# The ABI tests, against whichever FFmpeg is installed.
dotnet test

# Or against a particular build.
MEDIATOOLKITNET_FFMPEG_DIR=/opt/ffmpeg-9/lib dotnet test

Full documentation: README

Assets 2
Loading