A safety check that reads the current message sees only what the user chose to show. Risk Tree forecasts where the conversation can go next — then checks each reachable state against a curated risk store, so hidden intent surfaces as a path with evidence.
Maintained by NextSense AI — research notes at nextsense.ai/research. Originally published by Volkan Turk.
The illustrated tour · Method & full results · Threat model
Sanitized, defensive-only. No exploit steps, payloads, real targets, or credentials appear in this repository — by design. Retrieval similarity is treated as evidence for a routing decision, never as a verdict.
A conversation-safety layer usually scores the user's current message: embed it, compare it to a database of known-risky patterns, route the high-similarity ones to a slower check. That works until the current message is deliberately benign — a cover story, a slow roll, an innocent-sounding question whose next step is the harmful one. The risk isn't in the words on screen; it's one move ahead of them. Risk Tree carries over the one rule behind ReasonTree — the model never gets the last word on anything a machine can check — into misuse monitoring. Instead of judging the observed turn, it asks a cheap model to forecast a bounded tree of plausible next moves, scores each reachable state against a curated store of sanitized risky and benign patterns, and aggregates a probability-weighted (expected) risk. Every flag it raises is auditable: a reachable path plus the retrieved cases behind it.
Two frozen, sanitized evaluation sets, scored by the same pipeline. The honest headline is not "the tree is smarter" — a capable model's single forecast pass matches it. It's that looking ahead beats reading the current turn, and the tree's engineering contribution is staying calibrated while doing so.
Set A — 30 mixed cases (hidden-intent, obvious-risk, authorized-defensive controls, ambiguous):
| Condition | Hidden-intent recall | Obvious recall | False positives on controls | Ambiguous → clarify |
|---|---|---|---|---|
| Keyword gate | 25% | 33% | 50% | 0% |
| Single-turn retrieval | 75% | 83% | 12.5% | 75% |
| Full-conversation retrieval | 83% | 100% | 12.5% | 75% |
| Single model forecast (1 pass) | 100% | 100% | 12.5% | 0% |
| Risk Tree | 100% | 100% | 37.5% (all soft "monitor") | 75% |
Set B — 14 genuinely surface-benign cases (the last turn reads as an innocent question; the risk is only in the trajectory):
| Condition | Hidden-intent recall |
|---|---|
| Keyword gate | 0% |
| Single-turn retrieval | 85.7% |
| Single model forecast (1 pass) | 100% |
| Risk Tree | 100% |
Read those honestly:
- Keyword matching is blind to hidden intent — 0% on cases with no risk vocabulary in the visible turn. This is the easy, expected result.
- Forecasting recovers what the surface hides. On Set A, single-turn retrieval misses 3 of 12 hidden cases that the tree catches — e.g. a device-cataloguing chat sliding toward "what would tell me a login panel is worth trying default credentials?", a load-test drifting into rate-limit evasion, and a novel-writing frame used to unwrap literal restricted instructions. In each, the observed turn looks ordinary; the forecast branch lands on a known-risky pattern with evidence attached.
- A single model forecast pass is a strong baseline — 100% recall, 12.5% FP. The tree does not beat it on accuracy. Its distinct value is elsewhere (below).
The interesting failure is what happens when you forecast naively. Almost any conversation can continue toward something risky — including a legitimate penetration tester's, an incident responder's, a benchmark author's. If you route on the worst imaginable branch, you flag them all:
| Aggregation over the forecast tree | False positives on authorized-defensive controls |
|---|---|
| Worst reachable branch (max) | 87.5% |
| Expected peak, probability-weighted | 37.5%, and every one a soft "monitor," none reaching "review" |
Probability-weighting is the fix: a legit pentester's likely next move is "document findings in scope," a low-probability "goes rogue" branch barely contributes. Measured, the expected margin for control cases is negative (correctly allow); for hidden-intent cases it's positive (correctly flag). The residual soft flags are borderline — "expand these controls to more systems," "broad taxonomy coverage" — the kind of thing a human reviewer might reasonably glance at. At a "review-or-block" operating point the tree's control false-positive rate is effectively 0%.
This is the honest contribution: anticipatory risk monitoring is only usable if the look-ahead is probability-weighted; the worst-case reading is unshippable.
Adds
- Recovers hidden intent that keyword and single-turn checks miss, as a reachable path + retrieved evidence rather than an opaque score.
- Stays calibrated on authorized defensive work via expected (not worst-case) aggregation.
- Handles genuinely-ambiguous cases by routing to a clarifying question (75%) where a one-pass model forces allow/block (0%).
- Dataset-driven: new attack shapes are added to the store, no retraining; the decision is grounded in retrieval, not the model's unaided say-so.
Doesn't
- Doesn't beat a capable model's single forecast pass on recall — both hit 100%. If you only need a verdict and can trust the model, one call is cheaper.
- Doesn't eliminate false positives on security-adjacent legitimate work; forecasting inherently imagines risky continuations there (37.5% soft-monitor vs the one-pass model's 12.5%). Its FP profile differs from the one-pass model's rather than dominating it.
- Isn't a production moderation stack: no account-level signals, no novel-pattern coverage beyond the store, and it can be probed and adapted against. It's one cheap, parallel, auditable early-warning layer in a defense-in-depth design.
Ask a frontier model "is this conversation heading somewhere risky?" and it often answers well — the single-forecast column proves it. The point of the tree isn't to replace that judgment; it's to make it auditable and dataset-grounded: the decision is a probability-weighted aggregate over enumerated trajectories, each scored against a curated store you own and can update, and each flag ships with the specific reachable path and the retrieved cases that justify it. Where the domain gives you a store, the store — not the model's confidence — settles the routing. (Same lineage as Tree-of-Thoughts / MCTS branching and Reflexion-style critique; the deliberate difference is the retrieval gate and the honest publication of where it only matches a simpler baseline.)
Gemini cosine similarity has a high baseline (~0.6 for almost any pair), so raw similarity is a degenerate classifier — it flags everything. Risk Tree scores with a centered margin instead: subtract the store's mean embedding, then take (nearest risky case) − (nearest benign case). The boundary at 0 is principled — "closer to a known-risky than to any known-benign pattern" — and label-free. Benign controls land negative; risky trajectories land positive. Full derivation and the ablation are in docs/METHOD.md.
# Tests run offline, no API key required:
PYTHONPATH=src python3 -m unittest discover -s tests
# Score a single conversation window (needs a GEMINI_API_KEY for the semantic embedder):
PYTHONPATH=src python3 -m risk_tree.cli --cases data/seed_cases.jsonl \
--text "User asks, after a benign backup-tooling chat, what keeps a job's log entries from standing out in an audit."
# Reproduce the full comparison (~a few minutes; forecasts + embeddings are disk-cached):
PYTHONPATH=src python3 scripts/run_conditions.py --backend gemini --embedder gemini --score cmargin \
--eval data/risk_tree_eval.jsonl --out data/risk_tree_results_std.json
PYTHONPATH=src python3 scripts/run_conditions.py --backend gemini --embedder gemini --score cmargin \
--eval data/risk_tree_eval_hidden.jsonl --out data/risk_tree_results_hidden.json
PYTHONPATH=src python3 scripts/analyze_results.py data/risk_tree_results_std.jsonThe forecasting step is provider-agnostic — --backend gemini or --backend claude (or a mock for
offline tests). The reported runs use Gemini 2.5 Flash for both the tree's forecasting and the single-pass
baseline, so the comparison isolates the structure, not the model.
data/seed_cases.jsonl curated, sanitized risk store (risky patterns + benign controls)
data/risk_tree_eval.jsonl Set A — 30 mixed cases (hidden / obvious / control / ambiguous)
data/risk_tree_eval_hidden.jsonl Set B — 14 genuinely surface-benign cases
data/risk_tree_results_final_*.json frozen results behind the tables above
src/risk_tree/core.py forecasting backends, tree parsing, expected-margin aggregation, routing
src/risk_tree/scoring.py centered-margin scorer
scripts/run_conditions.py the five-condition harness
docs/METHOD.md method, scorer derivation, aggregation ablation, full honest limits
Frontier labs describe deployment safety as layered: model behavior, real-time classifiers, account-level signals, differentiated access, monitoring, review. This is a small, honest prototype of one layer — an early-warning signal that reads a conversation's trajectory, not just its latest turn, and shows its work. Built by Volkan Turk as independent AI-safety research, sibling to ReasonTree.