HH Goa 2026 · Shortlisting Task 2. Speak a question in Hindi, Tamil, Bengali, or English → Sarvam transcribes it → hybrid retrieval over an Indic MS MARCO index → a grounded, cited answer streams back — with the post-transcript pipeline engineered for a sub-200ms budget and every stage measured.
flowchart LR
A["🎤 voice<br/>(browser, 16k WAV)"] --> B["Sarvam STT<br/>saaras:v3"]
B --> C["input guards ∥ e5-small embed<br/>(concurrent, ~3ms)"]
C --> D["Qdrant hybrid search<br/>dense + BM25, weighted RRF<br/>language-filtered (~8ms)"]
D --> E{tier gate}
E -->|"score ≥ τ_cache"| F["Tier 0 · gold answer<br/>from index (~30ms)"]
E -->|"τ_topic ≤ score"| G["Tier 1 · sarvam-105b<br/>streamed, budgeted"]
E -->|"score < τ_topic"| H["Tier 3 · refusal"]
G -->|"grounded ✓"| I["answer + citations<br/>+ stage timings"]
G -->|"timeout / ungrounded"| J["Tier 2 · extractive<br/>sentence (<5ms)"]
F --> I
J --> I
Stage timings in the diagram are budget targets; measured percentiles land in docs/LATENCY.md once the deploy-time bench runs.
1. Chunking is a measured decision, not a guess. Nine chunking strategies are implemented behind one interface, indexed separately, and raced on 500 held-out labeled queries. passage_in_packed180 won and is the production index. The honest headline is that the top seven are statistically tied (2.0pp spread inside a 2.3pp standard error) — chunking barely moves the needle on a pre-segmented corpus — while sentence-granularity strategies lose by 15–22pp and union-style multi-chunking fails on 14 of 15 pairs. Two earlier runs are retracted in place because their metric never checked span overlap; cite Run 5. Protocol, results, and retractions: docs/CHUNKING_EVAL.md.
2. Latency is defined honestly and measured at percentiles. The 200ms budget covers transcript-in → final-byte-out, server-side (the brief's "chunking + retrieval + everything to final output"; STT is upstream vendor latency, reported separately — never hidden). The bench harness produces P50/P70/P90/P95/P100 across the full bench set, split by answer tier, plus first-token and full-audio tables (pending the deploy-time bench run): docs/LATENCY.md.
3. The model runs inside a real harness and knows when not to answer. Tiered fallback chain (cache → LLM → extractive → refusal), pre-first-token retries, circuit breaker, per-stage deadlines, Pydantic-typed boundaries — and guardrails on both sides: unsafe/injection screening in, groundedness + numeric-containment checks out, off-topic refusal via retrieval confidence. Details: docs/ARCHITECTURE.md.
| Piece | Choice | Why |
|---|---|---|
| STT | Sarvam saaras:v3 (REST, <30s clips) | 22 Indian languages, auto language detection |
| Embeddings | multilingual-e5-small, official ONNX, onnxruntime CPU | 2–4ms query encode on a dev laptop (to be re-benched on deploy); no embeddings API dependency in the hot path |
| Vector DB | Qdrant server (same container) | HNSW + payload filters + sparse vectors; localhost round trip |
| Retrieval | dense + BM25, weighted RRF | query-type routed weights (keyword → sparse, descriptive → dense) exercised in the chunking eval; live queries run balanced weights until a query-type classifier lands (future work) |
| Generation | sarvam-105b (default) or gemini-3.5-flash-lite, streamed, thinking off, ≤90 tokens | swap with LLM_PROVIDER; first-token deadline from the budget manager. sarvam-30b is deprecated. self-hosting was evaluated and rejected — docs/SELF_HOSTED_LLM.md |
| Answer delivery | sentence-gated streaming | each sentence is groundedness-checked then emitted, so text appears progressively without ever showing something the guard would reject — docs/STREAMING.md |
| Serving | FastAPI (async) + vanilla-JS SSE frontend | zero framework overhead in the measured path |
# 0. prerequisites: uv (https://docs.astral.sh/uv/), a Sarvam API key, qdrant binary
uv sync
cp .env.example .env # paste SARVAM_API_KEY
# 1. data: ~3k labeled queries per language -> synthesized documents
uv run python -m ingest.download # streams 3 per-language parquet subsets
uv run python -m ingest.corpus # dedupe + document synthesis + splits
# 2. qdrant (local dev; in Docker it's baked in)
docker run -p 6333:6333 qdrant/qdrant:v1.15.1 # or run a native qdrant binary
# 3. index every chunking strategy, race them, crown the winner
uv run python -m ingest.build_index
uv run python -m ingest.eval_chunking # writes docs/CHUNKING_EVAL.md
uv run python -m ingest.build_index --production <winner> --snapshot
# 4. serve
uv run uvicorn app.main:app --reload # http://localhost:8000
# 5. benches (against a RUNNING instance; deployed URL for honest numbers)
uv run python -m bench.bench_text --host http://localhost:8000 # -> docs/LATENCY.md
uv run python -m bench.make_audio # TTS bench clips
uv run python -m bench.bench_audio --host http://localhost:8000scripts/latency_spike.py is the day-1 de-risking probe: measures sarvam-105b TTFT/completion percentiles and prints which latency framing the numbers support.
One container: Qdrant + app. The index snapshot is fetched at first boot.
docker build -t vaani .
docker run -p 8000:8000 \
-e SARVAM_API_KEY=... \
-e INDEX_SNAPSHOT_URL='https://<account>.blob.core.windows.net/snapshots/chunks_passage_in_packed180.snapshot?<SAS>' \
vaaniRegion matters: the LLM round trip dominates the budget, so deploy near Sarvam's India-hosted API (Fly.io bom / anything Mumbai; scripts/rtt_check.sh compares candidates).
app/ the serving path: api/ (SSE endpoints) · harness/ (orchestrator, budget,
retries, breaker, timing) · retrieval/ (hybrid search + weighted RRF) ·
guardrails/ (input + output) · embed/ · llm/ · stt/ · analytics/ · static/ (UI)
ingest/ download -> corpus (document synthesis) -> chunkers/ (6 strategies) ->
build_index -> eval_chunking (the shoot-out)
bench/ bench_text (headline percentiles) · make_audio + bench_audio (STT/E2E tables)
scripts/ latency_spike (day-1 gate) · rtt_check
docs/ ARCHITECTURE · LATENCY · CHUNKING_EVAL · AZURE_STATE · AZURE_DEPLOY ·
SELF_HOSTED_LLM · PRODUCTION_PLAN
Start with CLAUDE.md — the orientation file for anyone (human or AI) picking this up: what was decided, why, and which questions are already settled.
- CLAUDE.md — settled decisions, conventions, known drift
- docs/AZURE_STATE.md — what is actually running in the cloud, and how to work on this without Azure access
- docs/SELF_HOSTED_LLM.md — the self-hosted Gemma 4 evaluation, the bandwidth arithmetic behind it, and why it was removed
- docs/STREAMING.md — sentence-gated streaming: why the unit is a sentence, and what it changes about retry and fallback
- docs/ARCHITECTURE.md — every module, every design decision, and why
- docs/LATENCY.md — the latency doctrine, budget, and measured percentile tables
- docs/CHUNKING_EVAL.md — chunking methodology and the strategy race results