v1.4.0 — the deflection blind spot: AI agent chats as a third signal, plus a theme-matching eval

An AI support agent answers questions that never become tickets. That makes ticket-based prioritization blind to every theme the bot handles well, and the blind spot grows as the bot improves. This release measures that gap and closes it. Full details in the changelog.

Measured before any of it was built: across four products over 30 days, 1,394 AI agent conversations against 1,678 support tickets — roughly 83% of the ticket channel, previously invisible to the analysis.

Added

• Chatbase as a third signal class. DEFLECTED joins REACTIVE and PROACTIVE. Chat conversations count toward the frequency term and carry three evidence fields per theme: deflected_count, self_serve_failure_rate, and mean_answer_confidence. They do not enter the severity or vote-momentum terms and do not trigger the 2x convergence boost — the scoring formula and methodology version are unchanged. Chatbase does not document what its min_score field measures, so it is reported as evidence for the LLM to weigh rather than folded into a score.
• A theme-matching eval. npm run eval reports multi-label precision, recall and F1 per theme, a per-register breakdown, and a keyword-collision report. Nothing previously measured whether theme assignment was correct — only that the scoring mechanics behaved. Runs in CI at --min-f1 0.90.
• agent_name filter on both analysis tools, Chatbase agents in list_sources, and a chatbase block in preview_only naming exactly which fields are and are not sent.

Changed

• themes.config v3. Two new themes derived from real unmatched conversations — Giveaways & Contests and List & Contact Management — taking the config to 18 themes across 12 categories. Over-generic keywords scoped, and every duplicated keyword now belongs to exactly one theme.
• Validated on 1,100 held-out conversations from a window the new themes were not derived from: unmatched fell from 39.9% to 33.1%, with the biggest gains on the products whose vocabulary the config never covered.

Fixed

• Single-word keywords now match a regular plural suffix. plan missed "plans" and tier missed "tiers", and the config listed plurals only where someone had thought of it. On live data this was expensive: a recurring widget prompt, "what are your plans and prices?", matched no theme at all. Irregular plurals still need listing explicitly.
• buildEvidenceSummary produced NaN signals for a theme without a deflected_count.

Security

• Cleared the two high-severity transitive advisories that were failing audit:ci (fast-uri, ip-address), plus the remaining moderate and low findings. Lockfile-only; no direct dependency range changed.
• Chat is the widest PII surface of the three sources. Only role: "user" turns are read — assistant turns, captured lead form submissions, end-user identifiers and per-conversation country never leave the server, asserted by a leak-check test. Adding this source is what first made credit_card appear in pii_categories_redacted; tickets and roadmap posts over the same window produced only emails and phone numbers.

Known limits, all documented in the README

Canned widget prompts recur verbatim and are deliberately not deduplicated. Matching is English-only. Irregular plurals need explicit keywords. A third of live chat conversations still match no theme.

Install

Clone and build — see the README. Chatbase is optional; with no key configured the output shape is unchanged from 1.3.0.