Read encoding, tags, waveforms, and MIDI from audio files - #5714
Merged
Conversation
Contributor
Preview deploymentsHost Test Results 1 files ± 0 1 suites ±0 3h 9m 42s ⏱️ + 3m 12s Results for commit 240db05. ± Comparison against earlier commit 518a89d. Realm Server Test Results 1 files ± 0 1 suites ±0 14m 15s ⏱️ -28s Results for commit 240db05. ± Comparison against earlier commit 518a89d. |
lukemelia
force-pushed
the
cs-12230-audio-midi-metadata
branch
from
August 9, 2026 03:21
ff6256c to
eb55d6d
Compare
jurgenwerk
approved these changes
Aug 10, 2026
Base automatically changed from
cs-12230-shared-metadata-fields-image-exif
to
main
August 10, 2026 10:27
lukemelia
force-pushed
the
cs-12230-audio-midi-metadata
branch
2 times, most recently
from
August 11, 2026 02:15
83a7baf to
518a89d
Compare
The audio FileDefs recorded a duration and nothing else, so a tagged,
mastered track indexed as anonymously as a voice memo — no artist, no
sample rate, no way to tell a 24-bit master from a 128 kbps stream.
Three shared shapes join the metadata module: `MediaEncodingField` for
how a stream is encoded, `MediaTagsField` for what a track says about
itself, and `WaveformMetadataField` for the decoded envelope. Encoding
and tags are shared per metadata family rather than per extension, so a
sample rate reads the same from an MP3 frame header as it will from a
video's audio track.
Every encoding fact is read from a header each format's duration reader
already walks, so it costs no extra I/O. Two tag conventions are new
modules because more than one format speaks them: `vorbis-comment-parser`
is shared by FLAC and both Ogg codecs, and `id3v2-parser` reads what
`mp3-meta-extractor` already skips past to find the first frame.
`audio-waveform.ts` is the browser-only half. It reduces decoded PCM to a
fixed 96-bar envelope, resampled across the *whole* signal rather than
downsampled from the opening, so a waveform drawn from it is the shape of
the track and not the shape of its intro. Bars are RMS, not peak: RMS
tracks perceived loudness, whereas a peak-per-bar envelope saturates to a
solid block on anything mastered loud. Every channel contributes, so a
hard-panned track isn't reported as silent.
Three judgment calls worth flagging:
A waveform needs the whole file, unlike everything else here. Ogg and M4A
stream their duration walk specifically to avoid buffering long
recordings, and asking for a second stream makes the extract runner
re-fetch. So the size check comes first and uses the `contentSize` the
indexer already supplies: a file over the 16 MB ceiling never triggers the
extra read at all. That ceiling is set on encoded size because that's what
is known before committing to a decode, and it is deliberately well under
what looks generous — float PCM costs roughly fifteen times the encoded
size, so 16 MB of MP3 is already a quarter-gigabyte decoded inside the
shared prerender pool.
Failure is recorded rather than thrown. `decodeStatus` distinguishes
`skipped` (too large), `unsupported` (no Web Audio), and `failed`, so a
renderer can tell "no waveform yet" from "this file cannot produce one",
and a file whose audio won't decode still indexes with every
header-derived fact intact.
Facts a container never stated stay unset. MP3 has no bit-depth concept,
so `bitDepth` is absent rather than filled with a plausible 16; FLAC never
states a bitrate, so none is invented; a lossy AAC sample entry carries a
sample-size field that describes nothing, so it's read only for lossless.
A track number keeps its total as authored ("4/12") rather than being
coerced to a number that discards it.
MIDI is deliberately not here. It is a separate family in the taxonomy —
a note sequence with no amplitude until something synthesizes it — so it
does not inherit `waveform`, and it needs its own parser and subclass.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
MIDI had no FileDef at all: a `.mid` in a realm indexed as a generic binary, with no indication it was music. `midi-meta-extractor.ts` walks a Standard MIDI File and reports its structure and content — format, division, notes, tempo map, time and key signatures, General MIDI programs, channels, and pitch range. `MidiDef` extends `FileDef` rather than `AudioDef`. A Standard MIDI File is symbolic performance data: a list of which notes to play and when, with no sound until a synthesizer renders it. It has no sample rate, no bit depth, no channel layout, and no amplitude envelope, so inheriting `AudioDef` would make every MIDI file advertise fields it can never fill. The taxonomy registry already models this as its own `music` family distinct from `audio`; this is the class that realizes that distinction. Details the format makes easy to get wrong, each pinned by a test: A note-on with zero velocity is the conventional note-off. Counting it as a note would double the total in most real files. Running status — omitting a repeated status byte — is the compression every sequencer emits, so a walk that doesn't follow it loses all but the first note of each run. Duration comes from walking the tempo map to the last event rather than averaging, so a piece that changes tempo reports the time it actually takes. A file timed in SMPTE timecode reports no ppq and no duration, because ticks don't convert to musical time there — rather than reporting a number that means nothing. Channel 10 is percussion by General MIDI convention, where a note number names a drum and a program names a kit. Both are kept out of the pitch range and the instrument list, and flagged separately. Sounding tracks are counted apart from the tracks the header declares: a format 1 file conventionally opens with a conductor track carrying only tempo and meter, which is not a part anyone plays. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The waveform was the one audio fact needing a full decode, and MP3 was by far its worst case: float PCM costs `duration x sampleRate x channels x 4`, which for a 128 kbps stream is roughly twenty times the encoded size — against four to six for FLAC, Ogg, and M4A. A 16 MB MP3 is around a thousand seconds, decoding to ~350 MB resident inside the shared prerender pool. That ratio is what forced the size ceiling in the first place. MP3 doesn't need a decoder for this. Every Layer III frame carries a side-info block whose granules each hold an 8-bit `global_gain` — the quantizer step exponent used to requantize that granule: sample = sign(is) . |is|^(4/3) . 2^((global_gain - 210) / 4) So `2^((global_gain - 210)/4)` is that granule's amplitude scale, readable by walking frame headers and picking eight bits out of side info. No Huffman decode, no IMDCT, no synthesis filterbank, and no decoded audio ever resident. Resolution is generous either way: MPEG-1 frames hold two granules of 576 samples, so ~76 amplitude points per second against the 96 bars an entire track reduces to. `streaming-envelope.ts` is what makes it flat in memory. A streaming producer has to place a value into a bar before knowing how many values there will be, and guessing the total from a header estimate would dump the whole tail of a longer-than-expected file into the last bar. So it accumulates into far more buckets than it needs and halves resolution whenever it runs out — the standard doubling histogram, where each fold merges adjacent pairs so a bucket always spans an equal stretch of signal. Two honest limits recorded rather than papered over: A quantizer scale is not calibrated amplitude. It tracks loudness well enough to draw but its absolute values aren't comparable with a decoded RMS, so bars are normalized to the track's own peak and `peakAmplitude`/`rmsAmplitude` are left unset rather than reported on a scale that means something different. The envelope records `algorithm: 'mp3-side-info-v1'` so a consumer can tell the two apart — which is what that field was for. There is no decode fallback when the side-info walk finds no frames. Adding one would reintroduce exactly the cost this avoids, so a file that yields nothing reports the failure. The ceiling stays for FLAC, Ogg, and M4A, which still need a real decoder, and its rationale is rewritten around them: a 16 MB FLAC is about three minutes and ~60 MB decoded, which the pool can absorb. MP3 is now exempt from it entirely. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
WAV needs no decoder for its envelope — the `data` chunk already holds the samples. The previous path read a header window, took a second stream, buffered the entire file, and handed it to Web Audio, which allocated a float copy roughly twice the file's size to recover samples the file already contained. `extractWavFromStream` walks the container once instead, keeping a bounded header buffer and one chunk of payload at a time. Duration, encoding, tags, and the envelope all come out of that single pass, so memory is flat in duration and it costs one fetch rather than two. Because these are real samples rather than MP3's quantizer proxy, the envelope is true RMS on the same scale a decoded one would produce — so it needs no normalization and `peakAmplitude`/`rmsAmplitude` mean what they say. WAV and the decoded formats now report directly comparable figures. Two details the buffered path never had to handle: A PCM frame can straddle a stream-chunk boundary. Leftover bytes are carried forward rather than dropped, and the test drives the reader with 7-byte chunks against 4-byte stereo frames so nearly every boundary falls mid-frame. Payload folding stops at the size the `data` chunk declares. Some encoders write LIST-INFO after the audio, and reading past the declared size would fold a tag block into the envelope as a burst of noise. Tags are parsed from before or after the payload, so either layout is read. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Working through the formats after WAV, each had its own waste. MP3's envelope no longer needs the encoded file resident. The side-info walk removed the decode, which was the large cost, but it still called `byteStreamToUint8Array` first — so the previous claim that it was "flat in memory regardless of duration" was only true of the decoded audio, not the file. Frames are self-delimiting, so the scan now streams: it keeps a rolling buffer of a few frames, discards what it has passed, and skips an ID3v2 tag by count rather than accumulating it, which matters when artwork makes that tag megabytes. A test drives it at 13, 417, and 1000-byte chunks and requires the same envelope each time. FLAC was silently losing tags. Its read window was 256 bytes, sized for STREAMINFO at offset 42 — but VORBIS_COMMENT sits behind whatever other metadata blocks the encoder wrote, and a SEEKTABLE (which most rippers emit, at 18 bytes per point) puts it tens of kilobytes in. Even without one, a realistic tag set with a vendor string, MusicBrainz ids, and ReplayGain values runs past 500 bytes. A short window yields no tags rather than an error, which is exactly why this went unnoticed. The regression test asserts both directions: the new window finds tags behind a 1000-point seek table, and the old one demonstrably did not. Ogg and M4A each made three fetches, one of which I introduced when adding encoding and tags. Both already stream a walk that retains precisely what the metadata readers need — Ogg keeps a head buffer, M4A reassembles the `moov` box — so both now hand that back and the separate header read is gone. Ogg's head is widened from 4 KB to 64 KB to reach the comment block, which is a far better trade than re-fetching the file. M4A needed no widening at all: `extractM4aEncoding` and `extractM4aTags` scan for `moov` from offset zero, and a lone moov box is one at offset zero. Fetches per file are now one for WAV, and two for the rest — the second being the decode the remaining formats still need. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The waveform decode was capped at 16 MB of encoded audio, which turns out to be the wrong quantity to measure. Float PCM costs `duration x sampleRate x channels x 4`, so the expansion factor is `(sampleRate x channels x 32) / bitrate` — about 4x for FLAC but 22x for AAC and up to 48x for low-bitrate Opus. One encoded ceiling therefore means wildly different decodes. At 16 MB encoded, FLAC decodes to ~64 MB, which is what I sized the ceiling against — but 64 kbps Opus at the same 16 MB is 35 minutes of audio and ~768 MB decoded, and it passed the check. The ceiling was admitting an order of magnitude more than intended for exactly the lossy formats it was supposed to bound. Predicting the decoded size costs nothing, because every caller has already read duration, sample rate, and channel count from the container's header before deciding whether to decode. So the budget now applies to that figure and each format lands wherever its own codec puts it — one rule, per-codec outcomes. 128 MB admits roughly five minutes of 44.1 kHz stereo, comfortably every ordinary song, while refusing the long recordings that would dominate a prerender page's memory alongside every other render sharing the pool. Mono costs half as much and is allowed twice the duration, which falls out of measuring the right thing rather than needing its own rule. The encoded ceiling survives only as a fallback for a container that stated too little to predict from, and is relabelled to say so. The skip reason now names the predicted size, so an operator can see how far over a file was rather than only that it was over. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
lukemelia
force-pushed
the
cs-12230-audio-midi-metadata
branch
from
August 11, 2026 02:51
518a89d to
240db05
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Split out of #5702, which had grown to cover three families. That PR is now image/EXIF only — the part jurgenwerk reviewed. This is everything after it, stacked on it; retarget to
mainonce it lands.Second slice of CS-12230, covering CS-12238's metadata half.
The gap
Audio FileDefs recorded a
durationand nothing else, so a tagged, mastered track indexed as anonymously as a voice memo. MIDI had no FileDef at all.What's here
Three shared shapes join the metadata module —
MediaEncodingField,MediaTagsField,WaveformMetadataField— plusMidiMetadataField. Encoding facts come from headers each duration reader already walked, so they cost no extra I/O. Two tag conventions became shared modules because more than one format speaks them:vorbis-comment-parser(FLAC + both Ogg codecs) andid3v2-parser(whatmp3-meta-extractoralready skipped past).MIDI extends
FileDef, notAudioDef. A Standard MIDI File is symbolic performance data with no sound until a synthesizer renders it — no sample rate, no bit depth, no amplitude envelope. InheritingAudioDefwould make every MIDI file advertise fields it can never fill. The taxonomy registry already models it as its ownmusicfamily.Waveforms without decoders
The envelope is the one audio fact a header can't give you, and it started as a full Web Audio decode. Two formats no longer need one:
2^((global_gain − 210)/4)is the granule's amplitude scale — with no Huffman decode, no IMDCT, no filterbank. Frames are self-delimiting, so it streams on a rolling few-frame buffer. This mattered most: float PCM is ~22× the encoded size for a 128 kbps stream, so a 16 MB MP3 was ~350 MB decoded.streaming-envelope.tsis what makes both flat in memory: a producer must place a value into a bar before knowing how many are coming, so it over-buckets and halves resolution on overflow — a doubling histogram where each fold merges adjacent pairs.FLAC, Ogg, and M4A still decode; none exposes an amplitude proxy.
Efficiency pass, per format
FLAC was silently losing every tag. Its window was sized for STREAMINFO at offset 42, but
VORBIS_COMMENTsits behind whatever blocks the encoder wrote — a SEEKTABLE (most rippers emit one, 18 bytes per point) puts it ~18 KB in, and a realistic tag set alone runs past 500 bytes. A short window returns no tags rather than an error, which is why it went unnoticed. The regression test asserts both directions.Ogg and M4A each made three fetches; both already streamed a walk retaining exactly what the metadata readers need (Ogg's head buffer, M4A's
moov), so they now hand it back.The decode budget was measuring the wrong thing
The ceiling was 16 MB of encoded audio. But the expansion factor is
(sampleRate × channels × 32) / bitrate— ~4× for FLAC, 22× for AAC, up to 48× for low-bitrate Opus. So one encoded ceiling meant wildly different decodes: 16 MB of FLAC is ~64 MB, but 16 MB of 64 kbps Opus is 35 minutes and ~768 MB, and it passed the same check.The budget now applies to the predicted decoded size, which costs nothing to compute — every caller has already read duration, sample rate, and channels before deciding. One rule, per-codec outcomes. 128 MB admits ~5 minutes of 44.1 kHz stereo; mono gets twice the duration, which falls out of measuring the right thing.
Honest limits, recorded not hidden
MP3's quantizer scale isn't calibrated amplitude, so its bars are normalized to the track's own peak and
peakAmplitude/rmsAmplitudeare left unset rather than reported on an incomparable scale.algorithmdistinguishesmp3-side-info-v1fromrms-peak-v1.decodeStatusseparatesskipped,unsupported, andfailed, so a renderer can tell "too large" from "cannot".Facts a container never stated stay unset: MP3 has no bit-depth concept, FLAC states no bitrate, a lossy AAC sample entry's size field describes nothing. A track number keeps its total as authored (
4/12).Testing
118 tests / 310 assertions over hand-built byte fixtures. Includes the cases that motivated each fix: the 64 kbps Opus file asserted to be within the old encoded ceiling and refused by the new one; FLAC tags found behind a 1000-point seek table and demonstrably not found by the old window; the MP3 streaming scan required to agree with the buffered one at 13, 417, and 1000-byte chunks; WAV driven at 7-byte chunks against 4-byte stereo frames so nearly every boundary falls mid-frame.
realm indexing140/140,ember-tscacross host/ai-bot/bot-runner/billing, and base's own lint all clean.🤖 Generated with Claude Code