fix(dub): auto-assign per-speaker cloned voices to segments (#486)

[debpalash]

What & why

#486 (reported on Discord with screenshots): multi-speaker dubbing diarizes the speakers and clones each from the video (the Voice dropdown shows From Video → Speaker 1 / Speaker 2), but every segment was left on "Default" — a row correctly labelled Speaker 1 had its Voice column reading Default, so the user had to set the voice on every segment by hand.

Root cause: the clone→segment binding never happened. The transcribe final handler (useDubWorkflow.js) stored speaker_clones but set the segments without filling their profile_id.

The fix

New pure helper applySpeakerCloneDefaults(segments, speakerClones) binds each segment to its detected speaker's clone up front:

• Sets profile_id = autoProfileId(speaker_id) (auto:<safe>) when a clone exists for that speaker.
• autoProfileId() mirrors both the backend clone-resolution key (speaker_id.lower().replace(" ","_"), dub_generate.py) and the DubTab dropdown option value (DubTab.jsx:889) — so all three agree and the auto-selected option renders correctly.
• Only fills an empty profile_id — an explicit per-speaker/per-segment choice is never clobbered.

Tests

segments.speakerClone.test.js (4 cases): assign-when-cloned, never-clobber-explicit, no-clone-stays-Default, no-op-without-clones. Full vitest suite + typecheck:ci green.

Scope note

The issue's second symptom — different speakers' turns merged onto one line — is a separate diarization/segment-grouping concern (speaker-turn re-split). It's tracked as a follow-up; this PR fixes the per-speaker voice assignment (the primary complaint).

🤖 Generated with Claude Code

Multi-Speaker Dubbing Auto-Voice Assignment Fix

Overview

This PR fixes a critical issue in multi-speaker dubbing workflows where auto-cloned speaker voices were not being assigned to their corresponding segments. When users dubbed multi-speaker videos, the system would diarize speakers and clone each one, but all segments remained set to "Default," requiring manual reassignment.

Root Cause

The transcribe final handler in useDubWorkflow.js stored the speaker_clones metadata but created segments without populating their profile_id field.

Solution

New Helper Functions (frontend/src/utils/segments.js)

Two new exported utilities manage auto-assigned voice profiles for diarized speakers:

• autoProfileId(speakerId): Normalizes a speaker ID into the backend-compatible auto:<safe> form (lowercased, whitespace collapsed to underscores). This format aligns with the backend clone-resolution key and DubTab dropdown option values.

• applySpeakerCloneDefaults(segments, speakerClones): Binds each segment to its detected speaker's cloned voice at initialization by:

  • Setting profile_id = autoProfileId(speaker_id) when a clone exists for that speaker
  • Only filling empty profile_id values, preserving explicit per-speaker or per-segment voice selections
  • Returning an empty array when no valid clones are provided

Integration (frontend/src/hooks/useDubWorkflow.js)

The transcription SSE "final" event handler now:

1. Normalizes the segment list (id, text_original, speaker_id)
2. Passes normalized segments through applySpeakerCloneDefaults(normalized, m.speaker_clones)
3. Stores segments with speaker-clone bindings applied upfront

Behavior Flow

graph TD
    A["SSE: final Event<br/>(Transcription Complete)"] --> B["Normalize Segments<br/>(id, text_original, speaker_id)"]
    B --> C["applySpeakerCloneDefaults<br/>(segments, speaker_clones)"]
    C --> D{Clone exists for<br/>this speaker?}
    D -->|Yes & profile_id empty| E["Set profile_id =<br/>auto:speaker_id"]
    D -->|No| F["Leave profile_id empty<br/>(Default)"]
    D -->|User already chose| G["Preserve user's<br/>profile_id"]
    E --> H["Store Segments<br/>with Auto-Bindings"]
    F --> H
    G --> H
    H --> I["UI Displays Voices<br/>Pre-assigned to Speakers"]
    I --> J{User satisfied?}
    J -->|Yes| K["Ready to Generate"]
    J -->|No| L["User can override<br/>per-speaker/segment"]
    L --> K

Loading

Testing

• 4 unit test cases (frontend/src/test/segments.speakerClone.test.js):
  • Auto-assignment when speaker clones exist
  • Preservation of explicit user voice selections
  • Default behavior when no clone is available
  • Correct no-op operation without clones (including null clones)
• Full vitest suite and typecheck:ci pass successfully

Changes Summary

File	Changes
frontend/src/utils/segments.js	+24 LOC: New autoProfileId() and applySpeakerCloneDefaults() functions
frontend/src/hooks/useDubWorkflow.js	+6/-3 LOC: Import and apply applySpeakerCloneDefaults() in SSE "final" handler
frontend/src/test/segments.speakerClone.test.js	+35 LOC: Four test cases validating auto-assignment behavior

Related Issue

A secondary concern—different speakers' turns merging onto one line—is identified as a separate diarization/segment-grouping concern and tracked as a follow-up item.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 7b636651-77a3-4e1c-98e7-863efbaeb291

📥 Commits

Reviewing files that changed from the base of the PR and between ff16804 and 55f2e70.

📒 Files selected for processing (3)

• frontend/src/hooks/useDubWorkflow.js
• frontend/src/test/segments.speakerClone.test.js
• frontend/src/utils/segments.js

---

📝 Walkthrough

Walkthrough

Adds autoProfileId and applySpeakerCloneDefaults to frontend/src/utils/segments.js. The workflow hook useDubWorkflow.js is updated to pass normalized segments through applySpeakerCloneDefaults during the transcription SSE "final" event before storing state. A new Vitest suite covers the function's core branches.

Changes

Speaker Clone Default Binding

Layer / File(s)	Summary
autoProfileId and applySpeakerCloneDefaults utilities frontend/src/utils/segments.js	autoProfileId lowercases and collapses whitespace to produce auto:<safe> IDs. applySpeakerCloneDefaults builds a clone lookup keyed by speaker_id, then maps segments to fill empty profile_id from that lookup, returning an empty array on invalid inputs.
SSE final handler wiring frontend/src/hooks/useDubWorkflow.js	Imports applySpeakerCloneDefaults; the transcription SSE "final" event now builds a normalized array and pipes it through applySpeakerCloneDefaults(normalized, m.speaker_clones) before setDubSegments.
Vitest suite frontend/src/test/segments.speakerClone.test.js	Tests: auto profile IDs assigned to matching cloned speakers, no overwrite of existing profile_id, empty/Default left alone when clone absent, and no-op for {} or null clones.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Possibly related issues

• [Bug] Multi-speaker dubbing: per-speaker voices not auto-assigned (fall back to Default) + speaker turns merged onto one line #486 — This PR directly implements the fix: applySpeakerCloneDefaults auto-assigns profile_id values derived from speaker_clones to diarized segments at transcription completion, resolving the per-speaker voice auto-assignment bug.

Possibly related PRs

• debpalash/OmniVoice-Studio#50: Overlaps at the same SSE "final" handler in useDubWorkflow.js where normalized segment state is set.
• debpalash/OmniVoice-Studio#369: Both PRs govern how profile_id is populated for cloned speakers, with the backend generating auto-seg:{id} refs that must coexist with the frontend's auto:{speaker} defaults introduced here.
• debpalash/OmniVoice-Studio#490: Backend counterpart that prioritizes auto:{speaker} for multi-speaker voice assignment, matching exactly what this PR's autoProfileId produces on the frontend.

---

Panel notes (ML inference, audio DSP, desktop systems, product polish):

segments.js line 37–43 — autoProfileId normalization is lossy and not reversible
speakerId.toLowerCase().replace(/\s+/g, '_') will silently conflate "Speaker 1" and "speaker_1" into the same auto:speaker_1 key. If the upstream diarizer can emit either form, two distinct speakers collapse to one profile. The safe fix is to normalize on input ingestion once (canonical form stored in state), not at mapping time, or at minimum assert in tests that "Speaker 1" and "speaker_1" produce the same key and that the backend is guaranteed to match that form.

segments.js line 45–60 — clone lookup is keyed by speakerId post-normalization but segments match on raw speaker_id
clonesLookup is built as { [clone.speakerId]: clone } (after normalization) but the segment match is clonesLookup[seg.speaker_id] with raw seg.speaker_id. If seg.speaker_id arrives as "Speaker 1" (with capital and space) and the clone key is "speaker_1", the lookup misses. Either normalize both sides at lookup time or store the raw key and normalize only at autoProfileId call.

useDubWorkflow.js line 103–110 — m.speaker_clones shape is not validated before being passed
applySpeakerCloneDefaults guards against null clones, but if m.speaker_clones arrives as an array (a reasonable wire format) instead of an object, Object.entries in applySpeakerCloneDefaults will iterate numeric indices and produce no useful matches. Add a shape check or ensure the SSE payload contract is tested end-to-end.

segments.speakerClone.test.js — no test for mixed-case or whitespace-containing speaker_id
Given the normalization concern above, a test case with speaker_id: "Speaker 1" and a clone keyed to "speaker_1" (or vice versa) would catch the mismatch before it ships. The current suite only exercises already-normalized IDs ("speaker_1", "speaker_2"), so the normalization path in autoProfileId is not exercised in a realistic scenario.

🚥 Pre-merge checks | ✅ 8 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 75.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (8 passed)

Check name	Status	Explanation
Title check	✅ Passed	Title follows conventional-commit style with scope and issue reference, clearly identifying the fix for speaker clone voice assignment.
Description check	✅ Passed	Description covers root cause, implementation approach, and testing strategy, though the formal template sections are not explicitly filled.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Cross-Platform Default Parity	✅ Passed	PR introduces platform-agnostic default behavior change: auto-assigns cloned speaker voices via pure JavaScript string/array operations. No filesystem I/O, OS detection, or conditional logic; runs...
I18n Completeness (21 Locales)	✅ Passed	No new t() calls or hardcoded user-facing strings introduced. All PR changes (two pure utility functions in segments.js, import in useDubWorkflow.js, and test suite) contain zero translatable conte...
Local-First Guarantee	✅ Passed	PR adds only pure local functions: autoProfileId() performs string normalization (lower/replace), applySpeakerCloneDefaults() maps segments with local data, no new dependencies, no API/cloud ca...
Backward Compatibility	✅ Passed	PR contains only frontend utility code additions with no database schema changes, alembic migrations, model modifications, or breaking changes to existing omnivoice_data/ structures. Existing segme...

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/dub-speaker-voice-autoassign

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[greptile-apps]

Greptile Summary

Fixes the root cause of #486: when multi-speaker transcription completes, the SSE final handler now passes segments through applySpeakerCloneDefaults() before storing them, so each segment's profile_id is pre-filled to auto:<speaker_slug> whenever the backend cloned that speaker's voice — eliminating the need for users to reassign voices manually.

• New pure helpers autoProfileId() and applySpeakerCloneDefaults() in segments.js implement the binding; the slug formula matches the backend clone-resolution key and the existing dropdown option values, so all three agree out of the box.
• useDubWorkflow.js wires the helper into the final SSE event handler in three lines; the segments streaming chunks are intentionally left unbound since final is the authoritative replacement.
• Four unit tests cover the assign, no-clobber, no-clone, and null-clones cases; the one pre-existing inline auto: slug formula in DubSegmentRow, DubTab, and CastingView could now be replaced with autoProfileId() to prevent future drift.

Confidence Score: 4/5

Safe to merge — the fix is a small, focused pure-function addition wired into one event handler, and the existing no-clobber guard preserves any explicit user choice.

The three changed files are tightly scoped, the helpers are pure and side-effect-free, and the binding happens on the authoritative final SSE event. The only rough edge is that several components (DubSegmentRow, DubTab, CastingView) still inline the same slug computation instead of calling autoProfileId(), leaving them one divergent edit away from producing mismatched IDs.

DubSegmentRow.jsx, DubTab.jsx, and CastingView.jsx each contain inline auto: slug expressions that duplicate the new helper and were not updated in this PR.

Important Files Changed

Filename	Overview
frontend/src/utils/segments.js	Adds autoProfileId() and applySpeakerCloneDefaults(); logic is correct and defensive (null-safe, non-clobbering). The slug formula matches all existing inline copies.
frontend/src/hooks/useDubWorkflow.js	Correctly wires applySpeakerCloneDefaults into the SSE "final" handler; reads speaker_clones directly from the message payload, not from stale store state.
frontend/src/test/segments.speakerClone.test.js	Four targeted unit tests cover the happy path, no-clobber invariant, missing-clone fallback, and null/empty clones no-op. Adequate for the surface area changed.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant BE as Backend (SSE)
    participant WF as useDubWorkflow
    participant SEG as segments.js
    participant ST as Store

    BE->>WF: SSE "segments" (streaming chunks)
    WF->>ST: setDubSegments(prev + incoming) [no profile_id binding]

    BE->>WF: "SSE "final" {segments, speaker_clones}"
    WF->>SEG: applySpeakerCloneDefaults(normalized, m.speaker_clones)
    SEG-->>WF: "segments with profile_id = auto:speaker_n (where clone exists)"
    WF->>ST: setDubSegments(bound segments)
    WF->>ST: setSpeakerClones(m.speaker_clones)
    ST-->>WF: UI re-renders: Voice column shows Speaker 1/2 instead of Default

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant BE as Backend (SSE)
    participant WF as useDubWorkflow
    participant SEG as segments.js
    participant ST as Store

    BE->>WF: SSE "segments" (streaming chunks)
    WF->>ST: setDubSegments(prev + incoming) [no profile_id binding]

    BE->>WF: "SSE "final" {segments, speaker_clones}"
    WF->>SEG: applySpeakerCloneDefaults(normalized, m.speaker_clones)
    SEG-->>WF: "segments with profile_id = auto:speaker_n (where clone exists)"
    WF->>ST: setDubSegments(bound segments)
    WF->>ST: setSpeakerClones(m.speaker_clones)
    ST-->>WF: UI re-renders: Voice column shows Speaker 1/2 instead of Default

Loading

_{Reviews (1): Last reviewed commit: "fix(dub): auto-assign per-speaker cloned..." | Re-trigger Greptile}