Prompt Categories

Prompt Categories

Every user prompt is tagged with one category (its intent) and a complexity score (1–5). The category is what drives the Prompts page breakdown and the category cross-filter; it answers "where does my interaction actually go — building, debugging, steering, ops…?"

There are three classifiers, all writing the same 13-label taxonomy:

• Heuristic (default, prompt-analytics categorize) — a deterministic FR+EN regex scorer. No API key, no cost, instant. This is what the Refresh data button runs.
• Semantic (prompt-analytics categorize --semantic) — multilingual static embeddings, fully offline, no API key. Reads meaning like the LLM but with nothing leaving your machine. Opt-in, the heuristic stays the default. See The semantic classifier below.
• LLM (prompt-analytics categorize --llm) — Anthropic / OpenRouter / Ollama / Azure. Reads meaning, so it handles open-ended prompts the regex can't, at the price of API calls. Same labels, defined by the SYSTEM_PROMPT in categorize.py (generated from the same shared anchors the semantic mode uses).

The rules below describe the heuristic classifier (the source of truth is prompt_analytics/categorize.py).

⚠️ This is the most work-in-progress part of the tool — and the easiest to help with. The heuristic rules were tuned on one developer's real prompt history, which is not representative: it skews toward this project's own workflow (French + English, lots of git/ops and dashboard iteration). As the community runs the tool on their own logs, the keyword lists, weights and category boundaries will need to evolve — this is exactly where contributions have the most impact. If a category mislabels your prompts, that's a useful signal: open an issue with a few (anonymized) examples, or send a PR tweaking the rules in categorize.py. The --semantic (offline) and --llm modes are the escape hatches in the meantime for prompts the regex can't place.

The taxonomy

Category	What it represents	Example
plan	Planning, architecture, design decisions	"propose an architecture for the cache"
implementation	Writing new code or features	"add a login endpoint"
debug	Fixing errors, investigating failures/crashes	"fix the error, it crashes on merge"
refactor	Improving / restructuring existing code	"simplify this function"
review	Reviewing, auditing, verifying code or results	"analyze the project", "verify the file"
test	Writing/running tests, CI, coverage	"add tests for the parser"
docs	README, changelog, comments, status files	"update the changelog"
ops	Git (commit/push/merge/PR), deploys, running scripts	"commit and push the branch"
question	Explanations, understanding	"what does this function do?"
followup	Short conversational steering / acks / option picks	"ok", "yes go ahead", "ok for A"
feedback	Reacting to / correcting the assistant's work	"not bad but the order is wrong", "actually, change…"
notification	Harness task-notification (background task / sub-agent finished) — not a human prompt	<task-notification>…</task-notification>
other	Everything the rules can't place	open-ended prose with no keyword

How the heuristic classifier decides

For each prompt, in order:

1. Strip harness chrome. <system-reminder>…</system-reminder> and <task-notification>…</task-notification> blocks are removed first, so the classification reflects the user's actual words (a reminder prefixing a real instruction no longer masks it).
2. Pure notification? If the turn was nothing but a task-notification (empty after stripping), it becomes notification. Its token cost is untouched everywhere else — cost is derived from tokens, not the category.
3. Pure ack/steering? A short prompt (≤ 40 chars) made entirely of acknowledgement tokens — ok, yes, go, sounds good, lgtm (and French equivalents like oui, c'est bon, je valide), an option pick (ok A, ok option 1), or a restart nudge (continue, reprends) — becomes followup before any scoring.
4. Weighted keyword scoring. Every category has a list of FR+EN regex patterns (accent-tolerant — real prompts are mostly unaccented French). Each match adds the category's weight; the highest total wins.
5. Ties & fallback. Equal scores resolve by a fixed priority order (below), so a prompt that both "fixes" and mentions a "plan" is debug. If no rule fires, a prompt ending in ? is question, otherwise other.

Weights and tie-break priority

The weight sets how strongly a single keyword pulls; the priority breaks exact ties. Specific intents outweigh and outrank generic discourse.

Priority (ties)	Category	Weight	Representative triggers
1	debug	1.2	fix, error, bug, crash, traceback, broken, not working, why does…, root cause
2	docs	1.1	readme, changelog, docstring, markdown, comments, status file, update the docs
3	test	1.25	test(s), pytest, coverage, regression, fixtures, CI passes
4	review	1.1	review, audit, analyze, verify, check, inspect, look at
5	refactor	1.0	refactor, improve, clean up, simplify, optimize, rename, restructure
6	ops	1.0	commit, push, merge, PR, branch, rebase, git, deploy, run the script, publish, install, release
7	plan	1.2	architecture, design, propose, strategy, approach, how should I…, best way
8	implementation	0.8	implement, create, write, add, develop, generate, build, integrate
9	question	1.0	what is, how do, why, explain, understand, describe, difference between
10	followup	1.5	stalled-assistant nudges; restart orders at line start (continue, restart, resume)
11	feedback	0.5	actually, instead, on second thought, I think, I prefer, not convinced, looks good but, still the same, not quite, no change
—	notification	n/a	detected structurally (step 2), not scored
—	other	n/a	the remainder (step 5)

Bilingual. The triggers above are shown in English, but the classifier is FR + EN: each has a French counterpart matched too (debug also fires on erreur / corrige / plante, ops on déploie / lance le script, feedback on en fait / plutôt / par contre, …), accent-tolerant since real prompts are mostly unaccented French. See categorize.py for the full lists.

Why feedback is the lowest weight (0.5). Course-correction markers ("actually", "instead", "by the way") are extremely common discourse and appear inside real task prompts too. At weight 0.5 a feedback marker is always out-scored by any concrete intent — "actually, fix the bug" stays debug, "instead, add a test" stays test. feedback only wins when the prompt is pure steering with no task keyword, i.e. exactly the prompts that would otherwise fall through to other.

The semantic classifier (offline, opt-in)

prompt-analytics categorize --semantic is a third classifier that reads meaning instead of keywords — without any API key and without sending anything off your machine. It is built on multilingual embeddings, so it places the open-ended, FR+EN, typo-laden prompts the regex drops into other.

• Embeddings in the core package, no extra to install. It uses a static, torch-free model2vec model (minishlab/potion-multilingual-128M) — a few tens of MB, pure-numpy at inference, fetched once then cached locally for offline use. There is no heavy [nlp]/torch dependency: the semantic features ship by default.
• Mono-label. Each prompt gets exactly one category. Per category, several realistic FR+EN prototype examples are embedded and kept distinct (not averaged into one blurry centroid). A prompt's score for a category is the max (or top-k mean) cosine to its prototypes.
• Hybrid. Hard short-circuits handle the unambiguous cases first (<task-notification> → notification, a short pure ack → followup), reusing the heuristic guard rails. For the rest, the semantic scores are fused in one logit space with a lexical prime for ops / feedback (their regex evidence, scaled so they compete without overriding a stronger intent); argmax wins. If the best score is below the threshold τ, the prompt is other (which has no prototype — "everything else" is exactly the sub-threshold case).
• Shared anchors. The prototype definitions/examples live in prompt_analytics/data/semantic_anchors.yml and are the single source of truth the LLM SYSTEM_PROMPT is also generated from, so the two meaning-based modes can't drift apart. The file is editable — community contributions welcome.

Why it is opt-in, not the default. An evaluation on the demo corpus (scripts/eval_semantic.py: a hand-curated litmus set + an LLM judge) had the heuristic edge it out (≈80% vs 72% litmus agreement). Two honest reasons: the synthetic demo prompts are keyword-rich (which favors the regex), and human agreement on a prompt's intent tops out around ~0.7 anyway, so part of the gap is irreducible ambiguity, not a fixable bug. The semantic mode shines on real, multilingual, open-ended histories — hence opt-in, with the heuristic as the safe default.

Tuning. The calibrated defaults (τ = 0.325, lexical prime weight = 0.60, top-k = 1) are overridable for power users, with precedence CLI flag > config > calibrated default:

prompt-analytics categorize --semantic --tau 0.30 --prime-weight 0.7 --top-k 2

or a semantic: section in output/config.yml:

semantic:
  tau: 0.30
  prime_weight: 0.7
  top_k: 2

Re-classification. Semantic rows are stamped semantic-st-v1. The semantic mode supersedes heuristic rows (so --semantic upgrades the default labels) and redoes its own stale-version rows, but — like the heuristic — it never overwrites LLM-classified rows.

Auditing the taxonomy (--audit-categories)

prompt-analytics categorize --audit-categories is a diagnostic, not a classifier: it clusters the whole corpus with HDBSCAN (density-based, no k to guess, isolates noise) and compares the natural clusters to the thirteen categories. It produces an alignment matrix, labels each cluster (frequent c-TF-IDF terms + representative prompts), and flags candidates for merges, splits, transverse/latent themes, and the other bucket. It writes taxonomy_audit.txt + taxonomy_audit.csv only — it never changes categories.csv. Use it to inform deliberate, human taxonomy revisions (e.g. whether test should fold into implementation), not to auto-rewrite labels.

Reading your distribution

A few labels naturally dominate an agentic-coding history — that's signal, not noise:

• ops is often the largest bucket: in agent-driven work you commit, push, merge, run scripts, install and deploy constantly, and every one of those verbs is an ops prompt.
• debug and question follow for the same reason — a lot of a session is reporting breakage and asking how things work.
• feedback + followup together measure how much of your interaction is iterative steering (acks, corrections, course-changes) rather than net-new work — a genuinely useful ratio to watch.
• notification is harness plumbing (background tasks / sub-agents finishing), kept separate so it neither inflates other nor distorts the real intents, while its token cost still counts everywhere.
• other is the floor the heuristic can't pass: open-ended prose with no reliable keyword (and the unavoidable typos / source-corrupted accents). Run categorize --semantic (offline) or categorize --llm to classify that long tail by meaning.

Complexity (1–5)

Complexity is observed, not classified. Rather than guess difficulty from the wording, it is the quintile-banded average of four real effort signals per prompt: assistant turns, tool calls, prompt length, and prompt cost. So a "3" means median effort across your own history, and the same scale recomputes on every run.

Re-classification

The heuristic stamps each row with a version (HEURISTIC_VERSION, currently heuristic-v3). Bumping it makes the next categorize run re-classify every heuristic row with the current rules — so a rules upgrade propagates without a re-extract. LLM-classified rows are never overwritten by the heuristic.

See Dashboard for the Composition (Input) and Prompt Explorer views that render these, and CLI Commands for categorize / categorize --semantic / categorize --llm.