feat(dub): audio-only dubbing mode (#119) - #150
Merged
Conversation
Add an audio→audio dubbing path: upload an audio file, get dubbed audio
out, with no video processing. The transcribe → translate → TTS core is
unchanged; only the video-coupled stages are skipped.
Backend:
- dub_core /dub/upload: new `input_type` form field ("video"|"audio").
Audio mode validates the upload is a known audio container (else 400)
and threads input_type into the ingest source dict.
- dub_pipeline ingest: for audio input, skip scene detection + thumbnail
ffmpeg passes (still emits scene_done count=0 so the prep SSE contract
the frontend waits on is unchanged); stores input_type on the job.
- dub_export /dub/download: for audio jobs, branch to an audio-only export
(_build_audio_export_cmd) — no video input/map/codec/subtitle pass.
Outputs dubbed_audio_{lang}_{stamp}.{wav|m4a|mp3|flac} via `out_format`
(default m4a), optionally mixed with the separated background. Unknown
formats fall back to AAC.
Frontend:
- dubSlice: dubInputType state + setter (default 'video').
- DubTab: auto-select audio-only mode when an audio file is dropped/picked.
- dub.ts/useDubWorkflow: pass input_type on upload.
Tests (11): _build_audio_export_cmd format/mix matrix; end-to-end audio-only
export produces an audio file (no video mux); unknown-format fallback;
upload rejects a video extension in audio mode.
Closes #119.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
💤 Files with no reviewable changes (1)
📝 WalkthroughWalkthroughAdds audio-only dubbing: frontend tracks input modality and sends ChangesAudio-Only Dubbing Feature
Sequence DiagramsequenceDiagram
participant User as User / Browser
participant DubTab as DubTab (UI)
participant Store as AppStore
participant dubUpload as frontend dubUpload
participant BackendUpload as /dub/upload
participant Pipeline as IngestPipeline
participant Export as /dub/download
User->>DubTab: Select/drag audio file
DubTab->>Store: setDubInputType("audio")
DubTab->>dubUpload: Call upload (includes input_type="audio")
dubUpload->>BackendUpload: POST /dub/upload with input_type
BackendUpload->>BackendUpload: validate input_type and file extension
BackendUpload->>Pipeline: Enqueue prep task with source.input_type="audio"
Pipeline->>Pipeline: derive/persist input_type
Pipeline->>Pipeline: skip scene/thumb for audio
User->>Export: Request /dub/download?out_format=m4a
Export->>Export: detect job.input_type == "audio"
Export->>Export: build ffmpeg command (codec mapping, optional amix)
Export->>Export: run ffmpeg and validate output
Export->>User: return audio file (m4a/mp3/wav/flac)
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Contributor
|
| Filename | Overview |
|---|---|
| backend/api/routers/dub_core.py | Adds input_type form field with validation: unknown values → 400, audio mode + non-audio extension → 400. Extension check uses an allowset (_AUDIO_EXTS). Looks correct. |
| backend/api/routers/dub_export.py | New _build_audio_export_cmd and audio-only branch in dub_download. Path sanitization for lang_code and out_format is correct; unknown format falls back to m4a. amix weights use space-separated floats which ffmpeg parses correctly within the filtergraph string. |
| backend/services/dub_pipeline.py | Audio jobs skip scene-detect and thumbnail ffmpeg passes; now correctly emits scene_start+scene_done(count=0) pair to keep SSE stage sequence consistent. input_type is persisted in both the full and partial job dicts. |
| frontend/src/hooks/useDubWorkflow.js | Reads dubInputType from store and conditionally shows "Preparing audio…" vs "Preparing video…". The follow-up "Extracting audio & scenes…" pill is not updated for audio jobs. |
| frontend/src/pages/DubTab.jsx | Both drag-drop and file-picker paths now call setDubInputType based on file type/extension. The accept attribute now includes all backend-supported audio extensions. |
| frontend/src/store/dubSlice.ts | Adds dubInputType state (default 'video') and setDubInputType setter. Included in INITIAL so resetDubState correctly resets it. |
| frontend/src/api/dub.ts | Adds inputType option to dubUpload, appended to the FormData as input_type. Typed as `'video' |
| tests/test_dub_audio_export.py | Unit tests for _build_audio_export_cmd covering all four formats, background mixing, and unknown-format fallback. All assertions are clear and meaningful. |
| tests/test_dub_export_unique.py | End-to-end audio-only export test asserts audio file produced and no video file created; upload rejection test for mismatched extension. Good coverage of the new path. |
Sequence Diagram
sequenceDiagram
participant U as User
participant FE as DubTab / useDubWorkflow
participant API as /dub/upload
participant PP as dub_pipeline (ingest)
participant EX as /dub/download
U->>FE: Drop / pick audio file
FE->>FE: setDubInputType('audio')
FE->>API: "POST /dub/upload (input_type=audio)"
API->>API: Validate ext in _AUDIO_EXTS
API->>PP: "ingest_pipeline(source={kind:file, input_type:audio})"
PP->>PP: extract audio (-vn)
PP->>PP: Demucs separation
PP-->>FE: SSE demucs_start / demucs_done
PP->>PP: Skip scene detect and thumbnail
PP-->>FE: "SSE scene_start / scene_done(count=0)"
PP-->>FE: SSE ready
Note over FE,EX: transcribe → translate → TTS (unchanged)
FE->>EX: "GET /dub/download/{job_id}?out_format=m4a"
EX->>EX: "job.input_type == 'audio' → audio branch"
EX->>EX: _build_audio_export_cmd(track, [bg], out, fmt)
EX-->>U: "FileResponse dubbed_audio_es_*.m4a"
Reviews (2): Last reviewed commit: "fix(#119): drop unused dubInputType read..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
The track id is already constrained to an existing track key, but allowlist-sanitize it before it reaches the output path (same pattern as the existing safe_name) so a path component can never carry separators — clears the CodeQL path-injection flag on the new audio-export branch. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 3
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
backend/services/dub_pipeline.py (1)
748-790:⚠️ Potential issue | 🟠 Major | ⚡ Quick winDon't reuse cached scene/thumbnail artifacts for audio-only jobs.
This branch still copies
scene_cutsandthumb_pathfrom a prior cached job even wheninput_type == "audio". If the same soundtrack was previously ingested from a video, the new audio-only job will report cached scene metadata and segment against video cuts, which breaks the audio-only contract.Suggested fix
- if cached["thumb_path"] and os.path.isfile(cached["thumb_path"]): + if input_type != "audio" and cached["thumb_path"] and os.path.isfile(cached["thumb_path"]): shutil.copy2(cached["thumb_path"], thumb_path) else: thumb_path = None - scene_cuts = cached["scene_cuts"] or [] + scene_cuts = [] if input_type == "audio" else (cached["scene_cuts"] or []) @@ yield prep_event("cached", has_bg=bool(no_vocals_path and os.path.exists(no_vocals_path)), scene_count=len(scene_cuts))🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@backend/services/dub_pipeline.py` around lines 748 - 790, The cached-branch is reusing video-specific artifacts for audio-only jobs; update the cached handling so when input_type == "audio" you do NOT copy or reuse cached["thumb_path"] or cached["scene_cuts"] (and ensure thumb_path ends up None and scene_cuts is an empty list for audio jobs). Concretely, in the cached block around the existing vocals/no_vocals/thumb logic, guard the thumbnail copy and the assignment scene_cuts = cached["scene_cuts"] with a check input_type != "audio" (or explicitly set thumb_path = None and scene_cuts = [] when input_type == "audio"); keep the vocals/no_vocals logic unchanged, and ensure the full_job dict uses the adjusted thumb_path and scene_cuts before calling put_job/save_job and yielding events.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@frontend/src/pages/DubTab.jsx`:
- Around line 416-417: The drop/picker gate that filters selected files is
out-of-sync with the regex used in setDubInputType (which now accepts .mp3 .wav
.flac .m4a .aac .ogg .opus .wma), so audio files with empty/generic file.type
get misclassified; update the file extension whitelist used in the drop/picker
logic to match the same regex/extension list used in the setDubInputType call
(and any other checks around the file input gate referenced near the same area
and at the other occurrence around lines 474-475), ensuring both file.type
checks and the fallback extension test use the identical pattern for audio
extensions (.mp3, .wav, .flac, .m4a, .aac, .ogg, .opus, .wma).
In `@tests/test_dub_audio_export.py`:
- Line 22: The current assertion uses OR and may pass incorrectly; update the
assertion that references the string variable s to ensure video mapping isn't
present by either asserting "-map" not in s (strict audio-only) or asserting the
specific combination "-map 0:v" not in s (allowing other -map entries like
audio), e.g., replace the line containing "assert \"-map\" not in s or \"0:v\"
not in s" with one of those stronger checks so the test fails if a video mapping
is actually present.
- Around line 17-52: Tests in test_dub_audio_export.py use hard-coded Unix-style
paths which break on Windows; update the test fixtures to construct
platform-agnostic paths (e.g., via pathlib.Path or os.path.join) when calling
_build_audio_export_cmd in test_wav_uses_pcm16le, test_m4a_uses_aac,
test_mp3_uses_lame, test_background_mix_adds_amix_and_second_input, and
test_unknown_format_falls_back_to_aac_m4a so the input/output paths are created
with Path("j") / "dubbed_de.wav" (or equivalent) rather than "/j/dubbed_de.wav",
ensuring the string passed into _build_audio_export_cmd is platform-correct
across macOS/Windows/Linux.
---
Outside diff comments:
In `@backend/services/dub_pipeline.py`:
- Around line 748-790: The cached-branch is reusing video-specific artifacts for
audio-only jobs; update the cached handling so when input_type == "audio" you do
NOT copy or reuse cached["thumb_path"] or cached["scene_cuts"] (and ensure
thumb_path ends up None and scene_cuts is an empty list for audio jobs).
Concretely, in the cached block around the existing vocals/no_vocals/thumb
logic, guard the thumbnail copy and the assignment scene_cuts =
cached["scene_cuts"] with a check input_type != "audio" (or explicitly set
thumb_path = None and scene_cuts = [] when input_type == "audio"); keep the
vocals/no_vocals logic unchanged, and ensure the full_job dict uses the adjusted
thumb_path and scene_cuts before calling put_job/save_job and yielding events.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: c0825f90-39e3-48cd-b2f2-73c9543432b7
📒 Files selected for processing (9)
backend/api/routers/dub_core.pybackend/api/routers/dub_export.pybackend/services/dub_pipeline.pyfrontend/src/api/dub.tsfrontend/src/hooks/useDubWorkflow.jsfrontend/src/pages/DubTab.jsxfrontend/src/store/dubSlice.tstests/test_dub_audio_export.pytests/test_dub_export_unique.py
Comment on lines
+17
to
+52
| cmd = _build_audio_export_cmd("ffmpeg", "/j/dubbed_de.wav", None, "/j/out.wav", "wav") | ||
| s = _flat(cmd) | ||
| # the dubbed track is the only input; never the source media / a video map | ||
| assert cmd.count("-i") == 1 | ||
| assert "/j/dubbed_de.wav" in s | ||
| assert "-map" not in s or "0:v" not in s # no video stream mapping | ||
| assert "-c:v" not in s # no video codec | ||
| assert "pcm_s16le" in s # wav → PCM | ||
| assert cmd[-1] == "/j/out.wav" | ||
|
|
||
|
|
||
| def test_m4a_uses_aac(): | ||
| cmd = _build_audio_export_cmd("ffmpeg", "/j/dubbed_de.wav", None, "/j/out.m4a", "m4a") | ||
| s = _flat(cmd) | ||
| assert "aac" in s | ||
| assert "-c:v" not in s | ||
|
|
||
|
|
||
| def test_mp3_uses_lame(): | ||
| cmd = _build_audio_export_cmd("ffmpeg", "/j/dubbed_de.wav", None, "/j/out.mp3", "mp3") | ||
| assert "libmp3lame" in _flat(cmd) | ||
|
|
||
|
|
||
| def test_background_mix_adds_amix_and_second_input(): | ||
| cmd = _build_audio_export_cmd("ffmpeg", "/j/dubbed_de.wav", "/j/no_vocals.wav", "/j/out.m4a", "m4a") | ||
| s = _flat(cmd) | ||
| assert cmd.count("-i") == 2 # track + background | ||
| assert "/j/no_vocals.wav" in s | ||
| assert "amix" in s # mixed, not just concatenated | ||
| assert "-filter_complex" in s | ||
|
|
||
|
|
||
| def test_unknown_format_falls_back_to_aac_m4a(): | ||
| # Defensive: an unexpected format string must not produce a broken command. | ||
| cmd = _build_audio_export_cmd("ffmpeg", "/j/dubbed_de.wav", None, "/j/out.bin", "weird") | ||
| assert "aac" in _flat(cmd) |
Contributor
There was a problem hiding this comment.
Ensure cross-platform path compatibility in test fixtures.
All test functions use Unix-style forward-slash paths (/j/dubbed_de.wav), which may cause failures on Windows where paths typically use backslashes or drive letters. Since the coding guidelines require features to behave identically on macOS, Windows, and Linux, the test paths should use pathlib.Path or os.path.join to ensure cross-platform compatibility.
🔧 Proposed fix using pathlib
+from pathlib import Path
+
def _flat(cmd):
return " ".join(cmd)
def test_wav_track_only_no_video():
- cmd = _build_audio_export_cmd("ffmpeg", "/j/dubbed_de.wav", None, "/j/out.wav", "wav")
+ cmd = _build_audio_export_cmd("ffmpeg", str(Path("/j/dubbed_de.wav")), None, str(Path("/j/out.wav")), "wav")
s = _flat(cmd)
# the dubbed track is the only input; never the source media / a video map
assert cmd.count("-i") == 1
- assert "/j/dubbed_de.wav" in s
+ assert str(Path("/j/dubbed_de.wav")) in sAlternatively, use platform-agnostic test fixtures:
+from pathlib import Path
+
+# Platform-agnostic test paths
+TEST_DUBBED = str(Path("j") / "dubbed_de.wav")
+TEST_OUT_WAV = str(Path("j") / "out.wav")
+TEST_OUT_M4A = str(Path("j") / "out.m4a")
+TEST_OUT_MP3 = str(Path("j") / "out.mp3")
+TEST_BG = str(Path("j") / "no_vocals.wav")
+TEST_OUT_BIN = str(Path("j") / "out.bin")
+
def test_wav_track_only_no_video():
- cmd = _build_audio_export_cmd("ffmpeg", "/j/dubbed_de.wav", None, "/j/out.wav", "wav")
+ cmd = _build_audio_export_cmd("ffmpeg", TEST_DUBBED, None, TEST_OUT_WAV, "wav")
s = _flat(cmd)
assert cmd.count("-i") == 1
- assert "/j/dubbed_de.wav" in s
+ assert TEST_DUBBED in sAs per coding guidelines: "Default features must behave identically on macOS, Windows and Linux. Platform-specific implementation is allowed, but a divergent user-visible default is a P0 bug — flag it and suggest an opt-in (Settings/env/flag)."
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/test_dub_audio_export.py` around lines 17 - 52, Tests in
test_dub_audio_export.py use hard-coded Unix-style paths which break on Windows;
update the test fixtures to construct platform-agnostic paths (e.g., via
pathlib.Path or os.path.join) when calling _build_audio_export_cmd in
test_wav_uses_pcm16le, test_m4a_uses_aac, test_mp3_uses_lame,
test_background_mix_adds_amix_and_second_input, and
test_unknown_format_falls_back_to_aac_m4a so the input/output paths are created
with Path("j") / "dubbed_de.wav" (or equivalent) rather than "/j/dubbed_de.wav",
ensuring the string passed into _build_audio_export_cmd is platform-correct
across macOS/Windows/Linux.
| # the dubbed track is the only input; never the source media / a video map | ||
| assert cmd.count("-i") == 1 | ||
| assert "/j/dubbed_de.wav" in s | ||
| assert "-map" not in s or "0:v" not in s # no video stream mapping |
Contributor
There was a problem hiding this comment.
🛠️ Refactor suggestion | 🟠 Major | ⚡ Quick win
Strengthen the video mapping assertion.
The assertion assert "-map" not in s or "0:v" not in s uses OR logic that passes if either condition is true. This could allow a command containing -map 0:a (audio mapping, valid) plus an unrelated 0:v string elsewhere to pass incorrectly. For audio-only commands, either verify that -map is completely absent or that the specific pattern -map 0:v doesn't appear together.
♻️ Proposed assertion improvements
Option 1: Assert no -map at all (strictest for audio-only):
- assert "-map" not in s or "0:v" not in s # no video stream mapping
+ assert "-map" not in s # audio-only commands should not map streamsOption 2: Check that video mapping specifically doesn't appear:
- assert "-map" not in s or "0:v" not in s # no video stream mapping
+ assert "-map 0:v" not in s and "0:v" not in s # no video stream mapping📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| assert "-map" not in s or "0:v" not in s # no video stream mapping | |
| assert "-map" not in s # audio-only commands should not map streams |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/test_dub_audio_export.py` at line 22, The current assertion uses OR and
may pass incorrectly; update the assertion that references the string variable s
to ensure video mapping isn't present by either asserting "-map" not in s
(strict audio-only) or asserting the specific combination "-map 0:v" not in s
(allowing other -map entries like audio), e.g., replace the line containing
"assert \"-map\" not in s or \"0:v\" not in s" with one of those stronger checks
so the test fails if a video mapping is actually present.
- dub_pipeline: emit scene_start before scene_done(count=0) for audio so the prep SSE stage sequence is symmetric with the video path. - useDubWorkflow: 'Preparing audio…' pill for audio jobs (was always 'Preparing video…'). - DubTab: widen the drop-accept regex + file-input accept to the full supported audio set (aac/opus/wma) so it matches the input-type detection and the backend allowlist. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Owner
Author
|
Addressed review in
11 tests green; frontend typecheck+build clean. |
Only setDubInputType is used; the value read was dead. Clears the CodeQL unused-variable alert. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #119.
What
Adds an audio→audio dubbing path: upload an audio file, pick source/target language, get dubbed audio out — no video processing, no mandatory video output. The transcribe → translate → TTS core is identical; only the video-coupled stages are skipped.
How
Backend
dub_core/dub/upload: newinput_typeform field ("video"default |"audio"). Audio mode validates the upload is a known audio container (.wav/.mp3/.m4a/.aac/.flac/.ogg/.opus/.wma, else400) and threadsinput_typeinto the ingest source dict.dub_pipelineingest: for audio input, skip the scene-detect + thumbnail ffmpeg passes (still emitsscene_donecount=0so the prep SSE contract the frontend waits on is unchanged).input_typeis persisted on the job. Audio extraction (-vn) and Demucs are already audio-generic — reused as-is.dub_export/dub/download: for audio jobs, branch to a focused audio-only export (_build_audio_export_cmd) — no video input/stream-map/codec/subtitle/stretch. Outputsdubbed_audio_{lang}_{stamp}.{wav|m4a|mp3|flac}via a newout_formatquery (defaultm4a), optionally mixed with the separated background (no_vocals). Unknown formats fall back to AAC.Frontend
dubSlice:dubInputTypestate + setter (default'video').DubTab: auto-selects audio-only mode when an audio file is dropped/picked (the drop zone already accepted audio).dub.ts/useDubWorkflow: passinput_typeon upload.Constraints
input_typedefaults tovideo; existing video jobs and on-disk job state are untouched (absent field → video). ✅Tests (11)
test_dub_audio_export.py:_build_audio_export_cmdformat/mix matrix (wav→pcm, m4a→aac, mp3→lame, bg→amix, unknown→aac).test_dub_export_unique.py::TestAudioOnlyDubbing: end-to-end audio-only export produces an audio file (asserts nodubbed_video_*.mp4is created), unknown-format→m4a fallback, and upload rejects a video extension in audio mode.All green (80 dub-suite tests pass, 0 failures).
🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Improvements
Tests