From be422e81c662c8dbad6b9c97b141f3c0c127c4bb Mon Sep 17 00:00:00 2001 From: Anish Kumar Date: Sat, 11 Jul 2026 10:49:21 -0700 Subject: [PATCH 1/2] Add 'Fast LLM + Human-in-the-Loop' post; refresh featured post and series New post 'How a Three-Person Team Used Cheap, Fast LLMs + Human-in-the-Loop to Classify Hundreds of Millions of Documents in ~2 Months', tagged both 'featured' (becomes the featured post) and 'content-trust-series' (joins the featured series). Removes 'content-trust-series' from the PhotoDNA post so the series now shows Dual-Embedding Trust Scoring + this new post. Co-Authored-By: Claude Opus 4.8 (1M context) --- _posts/2026-01-20-photodna-csam-detection.md | 1 - ...st-llm-human-in-the-loop-classification.md | 297 ++++++++++++++++++ 2 files changed, 297 insertions(+), 1 deletion(-) create mode 100644 _posts/2026-07-11-fast-llm-human-in-the-loop-classification.md diff --git a/_posts/2026-01-20-photodna-csam-detection.md b/_posts/2026-01-20-photodna-csam-detection.md index bcca299..5b8a412 100644 --- a/_posts/2026-01-20-photodna-csam-detection.md +++ b/_posts/2026-01-20-photodna-csam-detection.md @@ -6,7 +6,6 @@ tags: - aws - lambda - databricks -- content-trust-series team: ML Data Engineering author: anishk123 --- diff --git a/_posts/2026-07-11-fast-llm-human-in-the-loop-classification.md b/_posts/2026-07-11-fast-llm-human-in-the-loop-classification.md new file mode 100644 index 0000000..31f6c95 --- /dev/null +++ b/_posts/2026-07-11-fast-llm-human-in-the-loop-classification.md @@ -0,0 +1,297 @@ +--- +layout: post +title: "How a Three-Person Team Used Cheap, Fast LLMs + Human-in-the-Loop to Classify Hundreds of Millions of Documents in ~2 Months" +tags: +- featured +- content-trust-series +- machinelearning +- llm +- databricks +- scribd +team: ML Data Engineering +author: anishk123 +--- + +*A repeatable "Fast Model → Judge → SME → Iterate" workflow using familiar tools.* + +## TL;DR — Key Learnings + +LLM models change. Prompt quality changes. Cost changes. We assumed that from day one. + +So instead of building a one-off system tied to a specific model or a single "perfect prompt," we focused on a **repeatable process** that we successfully applied to two large-scale projects (Slideshare and Scribd). That repeatable process is what we're sharing. + +**Key learnings:** + +1. **Fast models give you breadth.** A simple prompt on a fast, low-cost model can label tens of thousands of documents quickly. The results won't be perfect — and that's okay. It turns Subject Matter Expert (SME) work from "label everything from scratch" into "review, correct, and curate." +2. **Batch inference makes scaling simpler.** For large backfills and corpus-scale labeling, asynchronous batch inference is usually both operationally simpler than building realtime request pipelines and meaningfully cheaper. +3. **"Fast model → Judge model" creates a scalable quality loop.** Use a higher-capability model as a judge to agree/disagree with the fast model's output. Use disagreements (and a small sample of agreements) as the highest-leverage items for SME review and prompt iteration. +4. **Validate beyond your small golden set.** Engineer prompts on a small golden dataset, then check generalization using 1% of the corpus, and run judge review on a 0.1% random sample (plus targeted slices when needed). +5. **Use familiar tools to keep a small team moving fast.** Databricks for data sourcing and orchestration → batch LLM calls → Google Sheets for SME validation → ingest back into Databricks → repeat. + +Repeatable mantra (we used it constantly): + +> **Fast Model → Judge Model → SME Review → Ingest → Iterate** + +## What We Achieved (With a Small Team) + +This process enabled projects that normally require far more people and time: + +- **Slideshare:** scanned tens of millions of slideshows, classifying each against adult-related content categories (e.g., sexually explicit content, drugs, alcohol, tobacco) and flagging the small fraction that matched. +- **Scribd:** scanned hundreds of millions of documents with a small cross-functional team — 1 engineer, 1 SME (annotations), 1 data scientist — in about two months. + +The specific labels and thresholds vary by project and policy requirements. The important part is that the **process** was portable across both efforts. + +## The Core Point + +LLM models, prompt quality, and cost will change over time. The durable asset is not a single prompt or model — it's a **repeatable human-in-the-loop process**. + +That process is: + +1. Use a fast model to generate candidate labels and short rationales at scale +2. Use a judge model to flag disagreements and ambiguous cases +3. Focus SME review only on the highest-value slices +4. Ingest SME corrections as a growing golden dataset +5. Iterate prompts and re-run only what you need (slices / disagreements) + +> **Fast Model → Judge Model → SME Review → Ingest → Iterate** + +## Workflow Overview + +Here's the end-to-end loop we repeated: + +1. **Source documents in Databricks** +2. **Standardize a document PDF** and metadata +3. **Build JSONL batches** for LLM inference +4. **Run batch inference** with a fast model (candidate label + short rationale + context) +5. **Store outputs** with prompt/model versioning +6. **Validate** on 1% of the corpus; run judge on 0.1% random sample +7. **Export disagreements** (and a small sample of agreements) to Google Sheets +8. **SMEs review and correct labels** quickly +9. **Ingest corrections back into Databricks** as the next golden dataset version +10. **Improve prompts** and repeat the loop + +> **Fast Model → Judge Model → SME Review → Ingest → Iterate** + +## Step-by-Step + +### Step 1 — Standardize Inputs (Before You Touch Prompts) + +The easiest way to make prompt work repeatable is to standardize what you feed the model. We used a consistent document "snippet" concept so every workflow starts the same way. + +**Example fields:** + +- `doc_id` +- `title` (if available) +- `pdf` +- `language` (if known) +- optional metadata + +**Why this matters:** + +- Prompts are simpler and more stable +- Batch sizing is predictable +- You avoid rewriting "how do we feed docs to the model?" for every project + +### Step 2 — Fast-Model Bulk Pass (Breadth First) + +**Goal:** create candidate labels at scale, cheaply and quickly. + +We asked the fast model for: + +- **label** +- **short rationale** + +Illustrative prompt (example only; actual production prompts were tuned and different): + +``` +Classify this document as ADULT, NOT_ADULT, or UNKNOWN. +Provide a 1–2 sentence rationale + +Document: + +``` + +**Why ask for rationales:** + +- Makes SME review faster +- Makes judge evaluation more reliable +- Makes debugging prompt failures dramatically easier + +### Step 3 — Batch Inference (Keep It Boring) + +For very large datasets, asynchronous batch inference is the "boring" choice that unlocks speed. You submit a JSONL file of requests, let it run, then download results later. + +**Why it works well:** + +- Generally 50% cheaper than realtime inference +- Fewer moving parts than realtime queueing systems +- Fewer timeout/retry headaches +- Easier to reproduce runs +- Easier to parallelize and monitor at the batch level + +### Step 4 — Store Everything with Versioning (So You Can Iterate) + +Store both raw and parsed outputs with run metadata: + +- prompt version +- model name / model version +- run timestamp +- `doc_id` +- label / rationale + +This turns iteration into something you can measure and regress-test, not a "vibes-based" exercise. + +### Step 5 — Judge Model Pass (Quality Signal at Scale) + +The judge model is a higher-capability model used to review a sample and decide whether it agrees with the fast model. It should explicitly: + +- **agree or disagree** +- provide a short rationale +- provide a corrected label when it disagrees + +Illustrative judge prompt (example only): + +``` +Here is the document and the fast model's label + rationale. +Do you agree? YES/NO +If NO, correct the label. +Explain briefly. +``` + +The judge is not a replacement for SMEs — it's a way to: + +- concentrate SME attention on the highest-value cases +- surface failure patterns faster +- create measurable disagreement signals for iteration + +> **Fast Model → Judge Model → SME Review → Ingest → Iterate** + +### Step 6 — SME Loop via Google Sheets (Focused Human Time) + +We exported: + +- **judge disagreements** (high priority) +- **a small random sample of agreements** (spot checks) + +SMEs reviewed the sheet and made edits. This is where human-in-the-loop scales: you're not asking SMEs to label everything — you're asking them to validate a small, high-value slice. + +Then we ingested the SME edits back into Databricks and versioned the golden dataset. + +## Sampling Strategy: 1% Validation + 0.1% Judge Sample + +Why not just rely on the small golden dataset? Because small golden datasets tend to be: + +- cleaner than the real corpus +- missing long-tail edge cases +- biased toward examples you already understand + +We used: + +- **1% of the corpus** as a validation slice for sanity checks +- **0.1% random sample** for judge review (plus targeted slices when needed) + +**What this catches:** + +- rare content patterns +- languages or formats underrepresented in golden data +- prompt ambiguity that doesn't show up in curated examples +- unexpected "UNKNOWN" spikes due to drift + +If random sampling isn't enough, stratify by patterns visible in the rationale and disagreements such as: + +- document type/format +- document length +- language + +## Prompt Engineering: Make It Iterative, Not Fragile + +A scalable, repeatable pattern: + +1. fast model outputs label + rationale +2. judge model explicitly agrees/disagrees +3. use disagreements as "training data" for prompt improvements: + - clarify definitions + - add a few targeted examples (carefully) + - tighten output formatting rules + - reduce ambiguity and edge-case confusion + +You should see: + +- disagreement rate drop over time +- SME correction rate drop over time +- fewer surprise failures on the 1% validation slice + +> **Fast Model → Judge Model → SME Review → Ingest → Iterate** + +## Metrics That Matter (Simple but Effective) + +You do not need an overly complex evaluation harness to start. Track: + +- disagreement rate (fast vs judge) +- SME correction rate (on judge disagreements) +- throughput (docs processed per job/day) +- iteration trend (disagreement should fall as prompts improve) + +The point is not "perfect metrics." The point is to make iteration measurable. + +## Common Failure Modes (And How We Handled Them) + +1. **Overconfident wrong answers** + Fix: capture context and prioritize big disagreements for SME review. +2. **Google Sheets becomes a bottleneck** + Fix: keep the sheet schema strict; export smaller batches; parallelize review across multiple sheets. +3. **Batch retries / partial failures** + Fix: keep batch requests idempotent; retry only failed batches; store intermediate results per batch. + +## What To Copy If You Want to Replicate This + +1. Standardize document inputs (`doc_id` + pdf or excerpt + minimal metadata) +2. Run a small fast-model probe (100–1k annotated docs) to sanity check outputs +3. Run a fast-model batch warmup for a larger slice +4. Run a judge model on 0.1% random sample +5. Export disagreements to Google Sheets and have SMEs review +6. Ingest edits as a versioned golden dataset +7. Iterate prompts, validate on 1%, then scale up + +Repeatable mantra (again): + +> **Fast Model → Judge Model → SME Review → Ingest → Iterate** + +## Summary + +LLM models, prompt quality, and cost will change over time. That's why we built a repeatable process — not a one-off prompt — and validated it across two large real-world projects. + +> **Fast Model → Judge Model → SME Review → Ingest → Iterate** + +That repeatability is what made it possible for a small cross-functional team to classify ~25M slideshows and ~400M documents in ~2 months — and it's the process we hope others can reuse and adapt as models and pricing evolve. + +## Appendix + +### A) Suggested Google Sheets Columns for SME Review + +| doc_id | snippet | fast_label | fast_rationale | judge_agree | judge_label | judge_rationale | sme_label | sme_notes | reviewer | review_ts | +|--------|---------|------------|----------------|-------------|-------------|-----------------|-----------|-----------|----------|-----------| + +### B) Mermaid Diagram + +```mermaid +flowchart LR + A[1) Source data in Databricks] --> B[2) Standardize doc snippets + metadata] + B --> C[3) Build JSONL batches (idempotent)] + C --> D[4) Batch inference: fast model] + D --> E[5) Store candidate label + rationale] + E --> F[6) Validate 1% slice; judge 0.1% sample] + F --> G[7) SME reviews disagreements in Sheets] + G --> H[8) Ingest edits → grow golden set → iterate] + H --> C +``` + +### C) One-Page Checklist + +- Standardized doc snippet schema +- JSONL batch builder +- Fast-model batch runner + parser +- Judge-model batch runner + disagreement computation +- Sheets export/import +- Golden dataset versioning +- Dashboard: disagreement rate + SME correction rate + unknown rate From b159a27a915d0614ee630a09aa36ee533f27958e Mon Sep 17 00:00:00 2001 From: Anish Kumar Date: Sat, 11 Jul 2026 11:11:52 -0700 Subject: [PATCH 2/2] Address review: add tag page, fix Mermaid, add workflow visuals - Add tag/llm/index.md so the new `llm` tag no longer 404s (Cursor Bugbot; matches the CONTRIBUTING generate-tags output format). - Fix the Appendix B Mermaid: quote node labels containing parentheses so it renders (previously broken syntax). Heading and text otherwise unchanged from the source doc. - Add three on-brand SVG diagrams (Scribd palette) grounded in the post text: - Workflow Overview: the repeatable Fast Model -> Judge -> SME -> Ingest -> Iterate loop. - Sampling Strategy: breadth-to-precision funnel (100% -> 1% -> 0.1% -> SME reviews the disagreements). - The Core Point: human-in-the-loop (model proposes labels, humans have the final say, corrections grow a versioned golden set, feeding back). - Add horizontal-rule dividers between top-level sections for readability (mirrors the section breaks in the source doc). Co-Authored-By: Claude Opus 4.8 (1M context) --- ...st-llm-human-in-the-loop-classification.md | 44 +++++++++++--- .../2026-fast-llm-hitl/human-in-the-loop.svg | 58 +++++++++++++++++++ .../2026-fast-llm-hitl/repeatable-loop.svg | 56 ++++++++++++++++++ .../2026-fast-llm-hitl/sampling-funnel.svg | 46 +++++++++++++++ tag/llm/index.md | 6 ++ 5 files changed, 202 insertions(+), 8 deletions(-) create mode 100644 post-images/2026-fast-llm-hitl/human-in-the-loop.svg create mode 100644 post-images/2026-fast-llm-hitl/repeatable-loop.svg create mode 100644 post-images/2026-fast-llm-hitl/sampling-funnel.svg create mode 100644 tag/llm/index.md diff --git a/_posts/2026-07-11-fast-llm-human-in-the-loop-classification.md b/_posts/2026-07-11-fast-llm-human-in-the-loop-classification.md index 31f6c95..0ca2bcc 100644 --- a/_posts/2026-07-11-fast-llm-human-in-the-loop-classification.md +++ b/_posts/2026-07-11-fast-llm-human-in-the-loop-classification.md @@ -12,7 +12,7 @@ team: ML Data Engineering author: anishk123 --- -*A repeatable "Fast Model → Judge → SME → Iterate" workflow using familiar tools.* +*A repeatable "Fast Model → Judge → SME → Iterate" workflow using familiar tools* ## TL;DR — Key Learnings @@ -32,6 +32,8 @@ Repeatable mantra (we used it constantly): > **Fast Model → Judge Model → SME Review → Ingest → Iterate** +--- + ## What We Achieved (With a Small Team) This process enabled projects that normally require far more people and time: @@ -41,10 +43,14 @@ This process enabled projects that normally require far more people and time: The specific labels and thresholds vary by project and policy requirements. The important part is that the **process** was portable across both efforts. +--- + ## The Core Point LLM models, prompt quality, and cost will change over time. The durable asset is not a single prompt or model — it's a **repeatable human-in-the-loop process**. +![Humans stay in the loop: the model proposes labels at scale, humans have the final say on the disagreements, and corrections grow a versioned golden set that feeds back into the model.](/post-images/2026-fast-llm-hitl/human-in-the-loop.svg) + That process is: 1. Use a fast model to generate candidate labels and short rationales at scale @@ -55,10 +61,14 @@ That process is: > **Fast Model → Judge Model → SME Review → Ingest → Iterate** +--- + ## Workflow Overview Here's the end-to-end loop we repeated: +![The repeatable loop: a fast model labels for breadth, a judge model checks a sample, SMEs make the final call, corrections grow the golden set, and the prompt is iterated — then the loop repeats.](/post-images/2026-fast-llm-hitl/repeatable-loop.svg) + 1. **Source documents in Databricks** 2. **Standardize a document PDF** and metadata 3. **Build JSONL batches** for LLM inference @@ -72,6 +82,8 @@ Here's the end-to-end loop we repeated: > **Fast Model → Judge Model → SME Review → Ingest → Iterate** +--- + ## Step-by-Step ### Step 1 — Standardize Inputs (Before You Touch Prompts) @@ -177,8 +189,12 @@ SMEs reviewed the sheet and made edits. This is where human-in-the-loop scales: Then we ingested the SME edits back into Databricks and versioned the golden dataset. +--- + ## Sampling Strategy: 1% Validation + 0.1% Judge Sample +![Breadth first, then concentrate human time: the fast model labels the full corpus, a 1% slice validates generalization, a judge model re-checks a 0.1% sample, and SMEs review only the disagreements.](/post-images/2026-fast-llm-hitl/sampling-funnel.svg) + Why not just rely on the small golden dataset? Because small golden datasets tend to be: - cleaner than the real corpus @@ -203,6 +219,8 @@ If random sampling isn't enough, stratify by patterns visible in the rationale a - document length - language +--- + ## Prompt Engineering: Make It Iterative, Not Fragile A scalable, repeatable pattern: @@ -223,6 +241,8 @@ You should see: > **Fast Model → Judge Model → SME Review → Ingest → Iterate** +--- + ## Metrics That Matter (Simple but Effective) You do not need an overly complex evaluation harness to start. Track: @@ -234,6 +254,8 @@ You do not need an overly complex evaluation harness to start. Track: The point is not "perfect metrics." The point is to make iteration measurable. +--- + ## Common Failure Modes (And How We Handled Them) 1. **Overconfident wrong answers** @@ -243,6 +265,8 @@ The point is not "perfect metrics." The point is to make iteration measurable. 3. **Batch retries / partial failures** Fix: keep batch requests idempotent; retry only failed batches; store intermediate results per batch. +--- + ## What To Copy If You Want to Replicate This 1. Standardize document inputs (`doc_id` + pdf or excerpt + minimal metadata) @@ -257,6 +281,8 @@ Repeatable mantra (again): > **Fast Model → Judge Model → SME Review → Ingest → Iterate** +--- + ## Summary LLM models, prompt quality, and cost will change over time. That's why we built a repeatable process — not a one-off prompt — and validated it across two large real-world projects. @@ -265,6 +291,8 @@ LLM models, prompt quality, and cost will change over time. That's why we built That repeatability is what made it possible for a small cross-functional team to classify ~25M slideshows and ~400M documents in ~2 months — and it's the process we hope others can reuse and adapt as models and pricing evolve. +--- + ## Appendix ### A) Suggested Google Sheets Columns for SME Review @@ -276,13 +304,13 @@ That repeatability is what made it possible for a small cross-functional team to ```mermaid flowchart LR - A[1) Source data in Databricks] --> B[2) Standardize doc snippets + metadata] - B --> C[3) Build JSONL batches (idempotent)] - C --> D[4) Batch inference: fast model] - D --> E[5) Store candidate label + rationale] - E --> F[6) Validate 1% slice; judge 0.1% sample] - F --> G[7) SME reviews disagreements in Sheets] - G --> H[8) Ingest edits → grow golden set → iterate] + A["1) Source data in Databricks"] --> B["2) Standardize doc snippets + metadata"] + B --> C["3) Build JSONL batches (idempotent)"] + C --> D["4) Batch inference: fast model"] + D --> E["5) Store candidate label + rationale"] + E --> F["6) Validate 1% slice; judge 0.1% sample"] + F --> G["7) SME reviews disagreements in Sheets"] + G --> H["8) Ingest edits → grow golden set → iterate"] H --> C ``` diff --git a/post-images/2026-fast-llm-hitl/human-in-the-loop.svg b/post-images/2026-fast-llm-hitl/human-in-the-loop.svg new file mode 100644 index 0000000..a6aa2ed --- /dev/null +++ b/post-images/2026-fast-llm-hitl/human-in-the-loop.svg @@ -0,0 +1,58 @@ + + + + + + + + + + + + + + + + Humans stay in the loop + The model proposes labels at scale; humans have the final say; corrections grow a versioned golden set. + + + + + + + + + + the golden set & improved prompts feed back into the model + + + + Model proposes + + Fast model + candidate label + rationale, at scale + + Judge model + flags disagreements on a sample + + + + Humans decide + + SME review + the final say on the label + + Focus on disagreements + not every document + + + + Ingest & iterate + + Golden set + corrections captured, versioned + + Measurable + disagreement & correction rates + diff --git a/post-images/2026-fast-llm-hitl/repeatable-loop.svg b/post-images/2026-fast-llm-hitl/repeatable-loop.svg new file mode 100644 index 0000000..9e274b0 --- /dev/null +++ b/post-images/2026-fast-llm-hitl/repeatable-loop.svg @@ -0,0 +1,56 @@ + + + + + + + + + + + + + + + + The repeatable loop + Breadth first with a fast model — then concentrate scarce human time where it matters. + + + + Iterate: improve prompts, then re-run only the slices / disagreements that changed + + + + + + + + + + + + + Fast Model + batch: label + rationale + + + + Judge Model + 0.1% sample, agree/disagree + + + + SME Review + Google Sheets, final say + + + + Ingest + grow the golden set + + + + Iterate + versioned, measurable + diff --git a/post-images/2026-fast-llm-hitl/sampling-funnel.svg b/post-images/2026-fast-llm-hitl/sampling-funnel.svg new file mode 100644 index 0000000..8be8f9e --- /dev/null +++ b/post-images/2026-fast-llm-hitl/sampling-funnel.svg @@ -0,0 +1,46 @@ + + + + Breadth first, then concentrate human time + The fast model labels everything; the judge and SMEs only ever touch a sample. + + + + + + + + + + + + Full corpus · 100% + fast model labels every document + + + Validation · 1% + sanity-check generalization + + + Judge · 0.1% + stronger model agrees / disagrees + + + SME review + review the disagreements + + + + One cheap batch pass + ~400M docs (Scribd), ~25M slideshows + + Beyond the golden set + catches drift & long-tail edge cases + + Quality signal at scale + measurable disagreement rate + + Highest-leverage human time + review disagreements, not everything + + diff --git a/tag/llm/index.md b/tag/llm/index.md new file mode 100644 index 0000000..a455404 --- /dev/null +++ b/tag/llm/index.md @@ -0,0 +1,6 @@ +--- +layout: tag_page +title: "Tag: llm" +tag: llm +robots: noindex +---