contrib/avcodec: whole-track audio decode to PCM — load_pcm's producer - #1483
Merged
Conversation
From the aether-ui video line. contrib/avcodec removed the intermediate file for VIDEO -- frames go straight from FFmpeg into a vg raster region -- but audio still cannot make the same trip, so video_frame requires a hand-extracted sidecar WAV: 20 MB for a 21 MB source, a manual pre-step before playback, and no workaround at all for a live source. Measured the gap rather than assuming it, and it is not where the API name suggests. load_wav is ma_decoder_init_memory, which sniffs the format, so it ALREADY accepts MP3 (duration_ms=6013 on a test file). It rejects MP4 and raw AAC. So the missing thing is not "formats beyond WAV" -- it is that every entry point takes an encoded container miniaudio can demux itself, with no way in for samples a DIFFERENT decoder produced. Noted separately that the load_wav name understates what it does. Asks for load_pcm(data, length, rate, channels, format) as the simple shape, and sketches a streaming push variant for live sources, flagging that position_ms must then report the DEVICE position since that is the clock video chases. Deliberately marked not-urgent: A/V sync is proven and correct today (video tracks audio.position_ms within 3 ms on a real 720p/5.1 clip), and nothing about the clock relationship changes with where samples come from. This is packaging, not architecture -- the sidecar is just the last hand-cranked step in an otherwise in-process pipeline. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The other half of the pcm-please ask, now that std.audio's load_pcm (e0738b4) landed. avc_audio_decode_raw demuxes and decodes a file's audio stream to interleaved s16 STEREO at the source rate in one shot -- libswresample converts whatever the source is (Big Buck Bunny's 5.1 float-planar AAC included) in the same pass. Exposed as avcodec.audio_pcm(url) -> (pcm, n, rate, ch, err), the whole-buffer shape matching load_pcm's. Proven C-only first (the video noise bug taught that order): 117.3s clip decodes to 22,523,904 bytes at 48 kHz stereo = 117.3s exactly, real samples mid-track. Then end-to-end in Aether: load_pcm reports duration_ms=117312 and position_ms advances in real time. A transient SIGKILL during first end-to-end testing was chased and did not reproduce -- co-resident make -j8 memory pressure, not a leak; the identical test passes with 18 GB free. Probe and contrib-check gain libswresample (all five FFmpeg libs required together; partial install stays a SKIP). Test gains the no-audio case (the video-only clip must report an error, not garbage) and a decoded-size range assertion whose LOWER bound is the real guard. Falsified via the downmix contract: reporting 6 channels instead of 2 gives "FAIL: channels 6 want 2"; restored, PASS. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The avcodec audio-decode commit landed with no CHANGELOG entry and there was no [current] section, so per this file's own workflow the next release would have tagged a version with whole-track audio decode unmentioned — the same way 0.506.0 through 0.509.0 shipped empty (backfilled in #1474, root cause in #1477). Written from the shipped code rather than the commit subject, and pairs the entry with load_pcm (0.512.0) since the two only make sense together: this is the producer, that is the consumer, and asks/pcm-please.md wanted both. Verified end-to-end here before writing it, rather than transcribing the commit's numbers: a 2s AAC clip decodes to 356352 bytes at 44100 Hz stereo and audio.load_pcm reports duration_ms=2020. (The ~1% over 352800 is AAC encoder padding, not a defect.) [skip actions] deliberately NOT used here, despite this being the docs commit: it is the branch HEAD, so the token would skip CI for the whole branch — including 244 lines of new C in contrib/avcodec that wants exercising on the Windows and macOS runners. The token is for branches that are docs-only end to end, not for the docs commit of a branch that ships code. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
paul-hammant
force-pushed
the
feat/avcodec-audio-pcm
branch
from
August 9, 2026 23:06
cd3ef4c to
fabc485
Compare
The CHANGELOG commit carried the skip token in its subject and was the branch HEAD, so GitHub suppressed the workflow for the whole PR — including 244 lines of new C in contrib/avcodec that wants exercising on Windows and macOS. Amending the token away did not retrigger, because the push event had already been skipped; an empty commit produces the fresh event that does. The subject here deliberately avoids the token text itself — an earlier attempt explained it in the subject line and GitHub matched THAT, skipping again. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
paul-hammant
force-pushed
the
feat/avcodec-audio-pcm
branch
from
August 9, 2026 23:09
34e9b45 to
c528671
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The producer half of
asks/pcm-please.md, completing the loop with theaudio.load_pcmconsumer that landed in 0.512.0 (#1481). Together they mean an MP4's audio reaches the speakers with no hand-extracted sidecar WAV — the last hand-cranked step in an otherwise in-process pipeline.Three commits: the ask itself, the implementation (Paul's), and a CHANGELOG entry.
What it adds
avc_audio_decode_rawdemuxes and decodes a file's audio stream to interleaved s16 stereo at the source rate in one shot. libswresample does the conversion in the same pass, so a 5.1 float-planar AAC track comes back as plain stereo s16 without the caller arranging anything. The whole-buffer shape deliberately mirrorsload_pcm's, so the two compose directly.Verified end to end, not just asserted
I rebuilt the pipeline here before writing the CHANGELOG rather than transcribing the commit's numbers — a 2s AAC clip, all the way to a playable source:
FFmpeg demux → s16 stereo →
std.audio→ playable. The ~1% over 352800 is AAC encoder padding, not a defect. (The original commit's own evidence is a 117.3s clip → 22,523,904 bytes at 48 kHz stereo = 117.3s exactly, withposition_msadvancing in real time.)make contrib-checkpasses all seven contrib tests including the extended avcodec one.Dependency change
contrib/avcodecnow requires libswresample alongside the other four FFmpeg libraries. Both the contrib-check table and the build probe were updated together; all five are required as a set, and a partial install stays a clean SKIP rather than a build failure.Shaping up
Three things the commits needed before this could go out:
main, which had diverged fromorigin/main. Moved to a branch and rebased.Claude-Session:trailer; stripped, so each ends atCo-Authored-By.[current]section — the exact gap that let 0.506.0–0.509.0 ship empty (backfilled in docs(CHANGELOG): backfill 0.506.0-0.509.0, which shipped unrecorded #1474, root cause in Release pipeline silently ships releases with no CHANGELOG section (4 so far), and races the merge window #1477). Added one, pairingaudio_pcmwithload_pcmsince producer and consumer only make sense together.Verification
make ci— C suite 230/230;.ae978/979. The one failure isintegration_http_server_h2, pre-existing and diagnosed inpesky_bug.md.gcc -Werror -Wall -Wextraon the new C with the FFmpeg cflags, checked locally before pushing.One ergonomic note
Building this by hand hit the
ae build --extra shim.clink-flag limitation — it compiles a C shim but cannot pass-lflags, so anaether.tomlworkspace is required. That is the same gap the avcodec contrib-check fix worked around, and it is what theTODO.md"contrib runtime coverage is split across two mechanisms" entry is about. Not addressed here.🤖 Generated with Claude Code