feat(video): support muxing an audio track in make_mp4_writer - #138
Open
lstein wants to merge 1 commit into
Open
feat(video): support muxing an audio track in make_mp4_writer#138lstein wants to merge 1 commit into
lstein wants to merge 1 commit into
Conversation
Adds optional audio_path/audio_codec to make_mp4_writer (imageio-ffmpeg forwards them to ffmpeg as a second input) and a write_stereo_wav helper that produces the 16-bit stereo WAV the writer consumes. Groundwork for video models that generate audio alongside frames (e.g. MiniMax H3). Callers must pre-trim audio to the video duration: imageio-ffmpeg passes no -shortest, so the container duration is the max of the streams. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Groundwork for MiniMax H3 (stacked PR 1 of 5): teaches the shared MP4 writer to mux an audio track, so video models that jointly generate audio (H3 produces 32 kHz stereo alongside frames) can save it inside the video container — no new asset type, service, or dependency needed.
make_mp4_writergains optionalaudio_path/audio_codec="aac"kwargs. imageio 2.37 / imageio-ffmpeg 0.6 already forward these to ffmpeg (-i <audio> -acodec <codec>); AAC-LC is the browser-safe choice for MP4.write_stereo_wavhelper: float PCM(2, n)in [-1, 1] → 16-bit stereo WAV. Conversion runs in float64 (fp16 input would round 1.0·32767 up to 32770 and wrap peaks to -32768); NaN becomes silence; out-of-range clips.audio_pathis existence-checked at construction: ffmpeg spawns lazily on the first frame append, so a missing file otherwise surfaces as an unexplained BrokenPipeError mid-encode — or, for tiny clips, silently produces no output.Two caveats documented in the module docstring for future callers:
-shortest, so container duration = max(stream durations). Callers must pre-trim audio to the video duration.audio_codec="copy"for a PCM WAV — PCM-in-MP4 doesn't play in browsers.Existing callers (
wan_latents_to_video,video_concat,video_frame_extract_range) are unaffected: the no-audio path builds the identical writer call as before.Testing
uv run pytest tests/app/util/test_video_encoding.py— 8 passed. New cases: aac stream present + container duration not stretched; codec forwarding pinned with a non-default codec (mp3) since ffmpeg's MP4 default is already aac; fast-fail on missing audio file; fp16/NaN safety; WAV format/clipping; no-audio output has no audio stream.h264+aac (LC), 32000 Hz, stereo, duration exactly matches frames/fps.🤖 Generated with Claude Code