txn-categorizer

A hybrid bank-transaction categorizer for consumer fintech: deterministic merchant rules handle the easy, recurring majority for free, and Claude handles the ambiguous tail with structured (schema-constrained) output. Ships with an evaluation harness, a FastAPI service, token-cost accounting, and a per-merchant cache.

Built to mirror what an AI Engineer actually owns in production: not just "call an LLM", but measure it, bound its cost, and wrap it in a service.

Why it's built this way

Decision	Reason
Rules first, LLM for the tail	Most transactions are recurring merchants (OXXO, CFE, Uber…). A keyword pass classifies them for $0 and 0 latency, so the LLM is only spent on genuinely ambiguous descriptions.
Per-merchant cache	A merchant string is classified once, not on every occurrence — the biggest lever on LLM spend for a real transaction stream.
Strict tool schema for output	The LLM is forced to call one tool whose category is an enum of the taxonomy, so it can only return a valid label — no free-text parsing or cleanup.
Cost-efficient model (claude-haiku-4-5)	Categorization is high-volume and low-complexity — the cheapest capable tier is the right engineering choice, and the harness reports the $/txn it costs.
Evaluation harness with a rules-only baseline	The honest way to judge the LLM is its lift over the free baseline, per category — not a single vibe-checked number.

Architecture

transaction ─▶ rule match? ──yes──▶ category (free, deterministic, conf 1.0)
                   │no
                   ▼
              cache hit? ──yes──▶ cached category
                   │no
                   ▼
        Claude (strict tool: enum category + confidence) ──▶ category  (+token cost)
                   │  (LLM disabled)
                   ▼
              fallback: "other"

Categories: groceries, dining, transport, utilities, entertainment, shopping, health, travel, income, transfer, fees, other.

Run the evaluation

pip install -e ".[dev]"

python -m categorizer.eval               # rules-only baseline (free, no API key)
export ANTHROPIC_API_KEY=sk-...
python -m categorizer.eval --llm         # rules + Claude, prints accuracy, macro-F1,
                                         # per-category P/R/F1, method mix, and $/txn

The baseline run needs no API key — it shows exactly how far merchant rules get you, which is the number the LLM has to beat.

Run the API

uvicorn app:app --reload
# or: docker build -t txn-categorizer . && docker run -p 8000:8000 -e ANTHROPIC_API_KEY=sk-... txn-categorizer

curl -s localhost:8000/categorize \
  -H 'content-type: application/json' \
  -d '{"description": "OXXO TIENDA 1234", "amount": -85.5}'
# {"category":"groceries","method":"rules","confidence":1.0,"reason":"merchant keyword match"}

Without ANTHROPIC_API_KEY the service runs in rules-only mode (still fully functional). Endpoints: GET /health, POST /categorize, POST /categorize/batch.

Monitoring

GET /metrics exposes Prometheus-format counters for post-deployment monitoring — total categorized, LLM calls, LLM errors (failed calls that degraded to a safe fallback instead of 500-ing), cumulative token spend in USD, and the rules/LLM/cache/fallback method mix. Point a Prometheus scraper at it to alert on error rate or cost. Requests are also logged via the stdlib logging module.

curl -s localhost:8000/metrics

Develop

ruff check .
pytest -q

Tests cover the rules, the taxonomy, the metric functions (hand-checked values), the orchestration (cache + cost accounting, LLM stubbed), and the API in rules-only mode — so the whole suite runs offline with no API key. CI runs lint + tests on every push (.github/workflows/ci.yml).

Next steps (ideas)

• Batch the LLM path via the Message Batches API (50% cheaper) for bulk backfills
• Add prompt caching for the system prompt + tool schema on high-volume runs
• Expand the labeled set and track macro-F1 as a regression gate in CI