support-classifier-svc

A small (fictional) B2B SaaS service that classifies inbound customer support tickets into one of four categories — billing, technical, account, or refund — using an LLM with structured JSON output.

This repo exists as a realistic testbed customer for driftless: it has the shape of a real LLM application that would feel the pain of model deprecation.

What's "production-like" here

• Externalized prompt + few-shot examples (prompts/) that are tuned to the current model and live separately from code.
• A strict JSON output contract validated against schemas/ticket.schema.json.
• A real parsing/post-processing path (src/support_classifier/postprocess.py) that rejects malformed output instead of silently rescuing it.
• An offline eval set (evals/tickets.inputs.jsonl + tickets.labels.jsonl) with a gold label per ticket.
• Model config via env var + config file (config/llm.yml, SUPPORT_CLASSIFIER_MODEL), the standard "override at runtime, default in config" pattern.
• Provider-agnostic calls via LiteLLM (src/support_classifier/llm_client.py), so swapping to a model from a different provider is just a change to the model string — no SDK rewiring.

Run it

The service runs offline by default using a deterministic simulated backend, so no API keys are required. Real calls go through LiteLLM: set any provider key it understands (OPENAI_API_KEY, ANTHROPIC_API_KEY, GEMINI_API_KEY, …) to hit a real model, or force the simulator with SUPPORT_CLASSIFIER_SIMULATE=1.

pip install -r requirements.txt

# classify the eval set with the current (configured) model
python evals/run_eval.py

# try a different model
SUPPORT_CLASSIFIER_MODEL=gpt-4o-mini python evals/run_eval.py

Built-in scenarios

A prompt has two dependencies — the model it's tuned for and the eval dataset that defines "correct." This repo reproduces a drift on each axis, both fixable by editing the prompt.

1. Model change (a swap regresses quality)

Offline (simulator, CI): the simulator is calibrated so the current model (gpt-3.5-turbo) scores cleanly, while a naive swap to gpt-4o-mini regresses two ways that are fixable by editing the prompt:

1. it wraps JSON in markdown ``` fences (breaking strict parsing), and
2. it misroutes refund requests to billing.

Both regressions disappear once the prompt is tightened (ask for raw JSON; spell out the refund-vs-billing rule) — exactly the kind of fix a migration tool should discover and validate.

Real models: on live API calls with the original hand-written prompt, gpt-3.5-turbo and gpt-4o-mini score nearly the same (~0.92 macro-F1) — the simulator's fenced-JSON / refund-misroute regressions do not appear. That doesn't mean model migration is a no-op; it means an unoptimized prompt hides the effect.

2×2 control (how to measure the switch honestly): optimize the prompt on the source model first, then switch. On real models (290 baseline labels):

Prompt	Source (gpt-3.5-turbo)	Target (gpt-4o-mini)
P0 — original	0.922	0.904
P_src* — optimized for source	0.993	0.921
P_tgt* — optimized for target	1.000	0.987

• Optimizing for the source alone lifts F1 0.922 → 0.993 — mostly prompt debt, not migration.
• From that strong baseline, the switch drops 0.993 → 0.921 — genuine model-induced drift (source-tuned few-shots transfer poorly).
• Re-tuning for the target recovers 0.921 → 0.987 — the gain attributable to migration repair.

So: report migration gains relative to a source-optimized baseline, not the raw hand prompt. Full methodology: migration-app docs/repair-and-generators.md. Reproduce: driftless refine pinned to each model in turn (see that doc).

2. Dataset change (the labels shift under a fixed model)

The model stays put; the labeling policy changes. "Charge-reversal" tickets ("reverse the charge on my card") read like billing today and are labeled billing, so the current prompt scores cleanly. When the team decides those are really refunds and re-labels them (evals/_apply_refund_policy.py), the same pinned model now under-performs until the prompt learns the new rule. This is the refine path — no model swap involved:

python evals/_apply_refund_policy.py     # relabel charge-reversal -> refund
driftless refine -w support_classifier   # re-tune the prompt toward the new labels

The simulator routes charge-reversal tickets by a prompt rule that applies to every model, so the recovery is reproducible offline. In CI, committing the relabeled tickets.labels.jsonl trips .github/workflows/refine-on-label-change.yml.

3. Real-model prompt repair (the honest end-to-end)

Scenarios 1 and 2 run against the simulator, which is deterministic and great for CI — but determinism can't prove a repair actually holds on a real model. For that, run the workflow with a provider key (so calls go through LiteLLM) and no SUPPORT_CLASSIFIER_SIMULATE.

This exposes something the simulator hides. The gold labels mark charge-reversal tickets as billing ("a request to reverse/correct a charge is a billing adjustment; a refund is for a correct charge the customer is just unhappy about"). That's a counterintuitive policy: it contradicts what an LLM assumes. Left to its own prior, real gpt-4o-mini calls reversals refund ~92% of the time — wrong under this policy — so the current prompt (which never states the rule) scores far worse on a real model than it does in simulation. This is the kind of rule (like "password-reset emails are security, not account") where a model silently underperforms until the prompt teaches it.

driftless refine recovers it by discovering the rule from the failure cluster (no hint) and rewriting the category definitions in prompts/system.md:

export OPENAI_API_KEY=...                       # real calls via LiteLLM
SUPPORT_CLASSIFIER_MODEL=gpt-4o-mini \
  driftless refine -w support_classifier --generator llm

Observed on real gpt-4o-mini (repair via gpt-4o): tuning F1 0.72 -> 0.96, accuracy 0.72 -> 0.98, and a never-tuned holdout F1 of 1.00 — committed only after the holdout validated. The winning edit redefines billing as "including requests to reverse or correct erroneous charges" and narrows refund to "charged correctly but dissatisfied."

Why --candidates matters: a single few-shot example can't flip a strong prior — only an explicit definition rewrite does. The default proposes a couple of candidates and auto-widens the search when an iteration stalls, so a hard counterintuitive rule still gets found. This run hits real APIs (a few hundred to a few thousand gpt-4o-mini calls depending on eval size), so it costs tokens and is non-deterministic — unlike scenarios 1–2.

4. Dataset evolves from feedback (new inputs + relabels)

Scenario 2 is a pure relabel; real eval sets rarely change that cleanly. After a round of customer feedback a team usually does both: it adds inputs that were a blind spot in v1 and corrects a few existing labels. evals/_apply_feedback_batch.py reproduces that on the real model:

• +22 new tickets — the original set had no subscription cancellation/lifecycle requests (every subscription mention was a charge complaint). Feedback surfaces ~20 real "cancel my plan / downgrade / pause / turn off auto-renew" tickets, plus a few unrelated ones so the batch is a realistic mix.
• 2 relabels — general subscription inquiries filed under billing are corrected to account (charge/amount complaints stay billing).

The new policy is counterintuitive like scenario 3: subscription lifecycle & management is account, not billing. A model's prior is "mentions a plan/subscription → billing", so on real gpt-4o-mini the current prompt gets account-recall 0.11 on the new tickets (16/18 called billing). Adding the lifecycle rule takes it to 1.00 — a genuine, prompt-fixable gap for refine to discover.

python evals/_apply_feedback_batch.py    # +22 inputs, 2 relabels (idempotent)
export OPENAI_API_KEY=...                 # real calls via LiteLLM
SUPPORT_CLASSIFIER_MODEL=gpt-4o-mini \
  driftless refine -w support_classifier --generator llm

This is the most realistic dataset-change event: it moves added, changed, and total-row signals at once, so it also exercises driftless poll's meaningful-change detection. Like scenario 2, the simulator routes subscription- lifecycle tickets by a prompt rule, so the recovery is also reproducible offline (SUPPORT_CLASSIFIER_SIMULATE=1: accuracy 0.936 → 1.000) — drop the flag and set a key to prove it on the real model (0.11 → 1.00 above). Committing the changed tickets.*.jsonl trips .github/workflows/refine-on-label-change.yml.

CI workflows

Both workflows install driftless from driftless-dev/driftless@main — this repo is an integration testbed for the product, not a PyPI consumer.

Workflow	Trigger	What it runs
Audit gold labels	PR/push to evals/tickets.*.jsonl	audit-labels --fail on support_classifier (blocks label conflicts)
Refine prompt on dataset change	Push to evals/tickets.*.jsonl	Offline simulator refine + opens PR/issue; job summary + artifacts
Migrate model	Manual dispatch	Real API compare + migrate --to (default gpt-4o-mini) + opens PR/issue. Requires OPENAI_API_KEY.
Real-model refine (290 tickets)	Manual dispatch / weekly schedule	Real API harness + driftless refine; job summary + artifacts (no PR). Requires OPENAI_API_KEY.

Test a detailed migration PR: Actions → Migrate model → Run workflow (defaults: target gpt-4o-mini, restore baseline prompt). The PR body includes summary, scorecard, unified diffs, and attempt log.

Test a detailed refine PR: apply a dataset change locally (evals/_apply_refund_policy.py), commit labels, push — or dispatch Refine prompt on dataset change.

Progress logs (phase 1/3, [iter 2/6], etc.) appear when DRIFTLESS_PROGRESS=1 (set in CI workflows).

Per-record cost capture + plan demos

Eval outputs now include cost_usd, prompt_tokens, and completion_tokens per ticket (simulated offline, real usage when a provider key is set). driftless.yml wires these as cost_field / prompt_tokens_field / completion_tokens_field so compare and plan can reason about total cost — driftless never fabricates token estimates when the workflow omits them.

A second workflow, quick_triage, decides whether tickets should escalate. Together with support_classifier, driftless plan can surface multiple migration rows. Policy defaults live in .driftless/policy.yml (cost savings, deprecation, dataset-change triggers).

Both workflows declare model.portable: true because routing goes through LiteLLM — cross-provider swaps do not require SDK rewiring.

# offline triage harness
python evals/run_triage.py

# plan across both workflows (from migration-app checkout)
driftless plan

Using it with driftless

From the migration-app checkout (with the CLI installed):

cd ../support-classifier-svc

# 1. find LLM usage and at-risk models
driftless scan

# 2. confirm the contract parses and the harness runs
driftless validate -w support_classifier

# 3. see the regression when swapping models
driftless compare -w support_classifier --to gpt-4o-mini

# 4. attempt an automated migration (LLM repair needs a provider key)
SUPPORT_CLASSIFIER_SIMULATE=1 driftless migrate -w support_classifier \
  --to gpt-4o-mini --generator llm

# 5. render the report / open a PR from the result
driftless report -w support_classifier
driftless open-pr -w support_classifier   # dry run by default

scan, validate, and compare work fully offline. With SUPPORT_CLASSIFIER_SIMULATE=1 the workflow runs on the simulator, so repairs are validated deterministically (good for CI). Drop that flag and set a provider key to run the real end-to-end (scenario 3 above) — the only way to prove a repair holds on the actual model.

Layout

src/support_classifier/   # service code (classifier, llm client, postprocess)
prompts/                  # editable system prompt + few-shot examples
schemas/                  # JSON schema for the output contract
config/                   # model configuration (default model id)
evals/                    # eval harness, dataset builder, dataset, dataset-change events
.github/workflows/        # refine, migrate, and real-model-refine CI
driftless.yml             # the migration contract (support_classifier + quick_triage)
.driftless/policy.yml     # when plan should propose migrations