Transcript Intelligence — Take-Home Submission

A pipeline that processes 100 enterprise call transcripts and surfaces insights for product, engineering, sales, and CS leadership.

What's in the box:

• pipeline/ — modular Python pipeline (data loader, classifier, categorizer, sentiment, insights, evals, viz).
• analyze.py — end-to-end offline analysis (no API keys needed). Produces all JSON outputs and charts that feed the slide deck.
• run_pipeline.py — LLM-backed pipeline (OpenAI or Anthropic, switchable via one CLI flag) with a built-in eval pass against the dataset's ground truth.
• build_slides.py — generates the .pptx deck from the analysis outputs.
• slides/transcript_intelligence.pptx — 14-slide deck for the 30-min panel.
• notebooks/analysis.ipynb — companion notebook walking through the same numbers with the underlying code exposed (for Q&A).
• video_script.md — 7-min screen-recording walkthrough script.

---

Quick start

pip install -r requirements.txt

# 1. Offline analysis — no API keys needed. Produces all charts + JSON.
python analyze.py

# 2. Build the slide deck from the analysis outputs.
python build_slides.py

# 3. (Optional) LLM-backed pipeline with eval pass — set keys first.
cp .env.example .env
#   ... edit .env ...
python run_pipeline.py --provider anthropic         # default
python run_pipeline.py --provider openai
python run_pipeline.py --provider anthropic --sample 20    # cheap dry-run

The dataset path defaults to ../interview-assignment/dataset relative to this folder. Override with --dataset on the CLI or DATASET_PATH in .env.

---

What the pipeline does

Required Task #1 — Categorization

A hybrid approach:

1. Per-call topic tagging — LLM with a closed taxonomy of 11 canonical tags. Closed taxonomy prevents tag drift, the #1 failure mode in real categorization systems.
2. Theme synthesis — one reasoning-tier LLM call rolls tags up into 3-6 higher-level themes for leadership consumption.
3. Eval pass — compare LLM tags to a lexical projection of the dataset's existing summary.json::topics field. Tag F1 + primary-theme accuracy.

Why hybrid: pure rules are too brittle for free-text content; pure LLM is expensive and prone to tag drift; hybrid gives the cost profile of rules with the recall of LLM, plus a built-in evaluator.

Required Task #2 — Sentiment analysis

The dataset already ships per-line sentiment labels and a meeting-level sentimentScore. We use those as primary signal — they're labeled at finer granularity than a one-shot LLM call could produce, and they're free.

What this layer adds:

• Aggregations by call type, week, customer domain.
• At-risk account detection — accounts with low avg sentiment OR steep downward trajectory across calls.
• Optional LLM second-opinion mode — re-score a sample, emit a confusion matrix vs the existing labels (in pipeline/evals.py).

Bonus insights

1. Cross-silo churn — customers visible in BOTH support AND external silos. Composite risk score combining churn signals, feature gaps, support call count, and a sentiment-floor penalty.
2. Action-item follow-through — what % of action items committed in a call are referenced again in a later call with overlapping participants? Lexical proxy; the magnitude of the gap (~45%) is the headline.
3. Feature-gap pressure index — feature gaps clustered by shared keywords, ranked by distinct-customer count. Same gap from 7 different customers is roadmap input, not noise.

---

Architecture decisions

Why offline analysis AND LLM pipeline (the two-track design)

The dataset already has rich pre-existing labels. Treating them as ground truth gives us two wins simultaneously:

• Slides + charts come from real numbers, not LLM confabulations — anyone can re-run analyze.py and verify.
• The LLM pipeline gets a free eval set — run_pipeline.py produces tag F1 and primary-theme accuracy automatically. This is exactly the regression test you'd want in CI before swapping models.

Why provider abstraction (OpenAI + Anthropic)

The job spec calls out both SDKs. The abstraction in llm_provider.py:

• Hides SDK-specific differences behind one classify / reason API.
• Coerces JSON output the same way on both sides.
• Exposes input/output token counts (cost tracking).
• Switches with one CLI flag — no code change.

Why two-tier model routing

Real production agentic systems route by task tier for cost reasons. Per-call tagging is a high-volume classification task — gets the cheap tier (Haiku 4.5 / gpt-4o-mini). Theme synthesis is one-shot reasoning — gets the strong tier (Sonnet 4.6 / gpt-4o). Same pattern that took $12K/month off LLM spend in production at Solidminds.

Why closed taxonomy

Open-vocabulary categorization is the single biggest source of tag drift in real systems — the same concept gets six different names across calls. A closed taxonomy with explicit synonym lists keeps the LLM honest. The taxonomy was bootstrapped from the dataset's existing topic labels (so it's already grounded in real data).

---

What I'd do next (production-readiness)

1. Stand up the cross-silo at-risk dashboard first — labeled data is already there, value is immediate, no equivalent today.
2. Treat action-item follow-through as a north-star ops KPI — surface it weekly per-team. Expect the number to move with attention alone.
3. Wire the feature-gap pressure index into PM grooming. Already produces actionable clusters; needs a UI surface, not more code.
4. Scale categorization with LangGraph + Anthropic batches — current pipeline is single-threaded; ~5x speedup with no architecture change.
5. Invest in the eval framework before scaling LLM calls — the eval set we already have lets us test new models in <60 seconds.

---

Things I deliberately did NOT do

• Speech-to-text — text is already there.
• Vector DB / embedding clustering — would be honest signal but over-kill for 100 calls. Lexical projection onto a closed taxonomy was sufficient and produces auditable tags. (Embedding-based clustering is a 30-line addition if needed for scale.)
• Custom UI / dashboard — the brief asks for a slide deck and notebook, not a product surface.
• Synthesis of new transcripts — the existing 100 are sufficient to demonstrate every claim in the deck.

---

Run order for the demo recording

1. python analyze.py — produces JSON + charts. ~3 seconds.
2. python build_slides.py — produces the .pptx. ~1 second.
3. (Optional) python run_pipeline.py --provider anthropic --sample 20 — cheap LLM dry-run that produces eval metrics.
4. Open slides/transcript_intelligence.pptx to present.
5. Have notebooks/analysis.ipynb ready for Q&A.

---

Project layout

interview-solution/
├── README.md                       (this file)
├── requirements.txt
├── .env.example
├── analyze.py                      # offline analysis — no API key required
├── run_pipeline.py                 # LLM-backed pipeline + evals
├── build_slides.py                 # generate the .pptx from outputs/
├── pipeline/
│   ├── data_loader.py              # JSON folder -> normalized records
│   ├── llm_provider.py             # OpenAI + Anthropic abstraction
│   ├── call_classifier.py          # support / external / internal
│   ├── categorizer.py              # 11-tag closed taxonomy + theme synthesis
│   ├── sentiment.py                # aggregations + at-risk detection
│   ├── insights.py                 # cross-silo, action items, feature gaps
│   ├── evals.py                    # tag F1, primary-theme acc, confusion matrix
│   └── visualize.py                # matplotlib chart generation
├── notebooks/
│   └── analysis.ipynb              # companion walkthrough for Q&A
├── outputs/                        # generated by analyze.py / run_pipeline.py
│   ├── call_types.json
│   ├── categorization.json
│   ├── sentiment.json
│   ├── at_risk_accounts.json
│   ├── insights.json
│   ├── llm_categorization.json     # only after run_pipeline.py
│   └── charts/*.png
├── slides/
│   └── transcript_intelligence.pptx
└── video_script.md