Baatein

Multi-ASR → LLM fusion for candid, noisy real-world speech.

Baatein asks a simple research question: when Whisper, Qwen, Grok, and MAI disagree on the same messy clip, can a frontier LLM fuse them into a cleaner, more faithful transcript — without inventing content?

This repo is a fast notebook pipeline for building that dataset. It is not a product ASR stack.

What we're validating

Target audio: vlogs, street / driving footage, office day-in-the-life — ambient noise, overlapping talk, casual register. Not studio podcasts.

Working hypotheses (no automated WER; human scores still empty on v1):

ID	Claim	v1 status
H1	On noisy clips, ASR models disagree often enough that fusion can beat any single system	Precondition supported — 0/36 all-ASR verbatim agree; fusion marked needed on 61% of chunks
H2	LLM cleanup improves readability/faithfulness without hallucinating unsupported content	Inconclusive — notes show real merges / name fixes; faithfulness unaudited until human_score is filled
H3	Self-reported confidence loosely tracks human judgment	Unevaluable — no human labels; model looks overconfident (mean ~0.91, never flags low intelligibility)
H4	Domain / noise type changes which ASR wins	Early support — English → whisper/qwen; Hindi/Hinglish → mai/qwen (whisper never best)

Full run metrics: docs/learning-notes.md. Backlog (quality gate → scale): docs/todos.md. Design notes: docs/baatein-plan.md.

Pipeline

YouTube URLs
    → metadata (yt-dlp)
    → audio (16 kHz mono WAV, first 5 min)
    → ~75s chunks
    → N ASR models via OpenRouter (parallel, resumable)
    → optional official captions for the same window
    → LLM fusion → cleaned text + confidence / disagreement / intelligibility flags
    → CSV + JSONL (one row per chunk)

Hard prompt constraints: only use words supported by at least one ASR candidate (or a light correction of one); drop clear filler; keep speaker register; treat title/description as name/jargon disambiguation only — never as a content source.

Notebooks

Notebook	Role
baatein_v0_pipeline.ipynb	First cut: 3 ASR slots + LLM cleanup
baatein_v1_pipeline.ipynb	Current: more ASR families, official CC, concurrent + mid-run flush (resumable)

v1 ASR slots (OpenRouter; Chirp + Parakeet commented out in the notebook for this run):

• openai/whisper-large-v3
• qwen/qwen3-asr-flash-2026-02-10
• x-ai/grok-stt-1.0
• microsoft/mai-transcribe-1.5

Cleanup LLM is configured in the notebook (currently deepseek/deepseek-v4-flash-0731 via OpenRouter).

Setup

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Needs ffmpeg / ffprobe on PATH, and an OpenRouter key:

cp .env.example .env
# set OPENROUTER_API_KEY=sk-or-...

jupyter notebook baatein_v1_pipeline.ipynb

Fill VIDEO_URLS, run top → bottom. v1 writes under output/v1/ and skips already-filled ASR/cleanup cells on re-run.

Output

One row per audio chunk. Key fields:

• Raw texts: asr_whisper_text, asr_qwen_text, asr_grok_text, asr_mai_text (+ optional official CC cols)
• Fusion: llm_cleaned_text, llm_confidence, llm_disagreement_flag, llm_low_intelligibility, llm_fusion_needed, llm_best_single_model, llm_notes
• Review blanks: human_score, human_notes

Latest v1 sheet: 36 chunks / 9 videos under output/v1/baatein_dataset_v1.csv (4 EN + 5 Hindi/Hinglish). ASR reliability high; cleanup filled after retries (sticky parse_error on some rows). Next gate is human scoring of those 36 rows — that closes H2/H3.

After a run: listen to the chunk WAVs, score 1–5, note over-cleaning / hallucination / wrong names / wrong language. That labeled sheet is the eval for this round.

Explicitly out of scope (for now)

Diariization, local Whisper/GPU stacks, non-OpenRouter providers (Sarvam, Deepgram, …), automated judge/WER, SFT/distillation. Those land after the fusion idea holds up under human review.

Repo layout

baatein_v0_pipeline.ipynb
baatein_v1_pipeline.ipynb
requirements.txt
docs/baatein-plan.md      # detailed v0 design + prompt constraints
docs/learning-notes.md    # run-time findings
docs/todos.md             # quality gate → scale backlog
output/v1/                # datasets + audio from v1 runs