Add Keyword/pattern-based canned response matching for completion endpoints

[Taka-cst]

Ideas

Currently, completion_text(profile) only returns a fixed string configured via PROMPTPOT_RESPONSE_TEXT, completely ignoring the actual prompt content.
Looking at the promptpot.messages keyword aggregation in Kibana, canary-style liveness-check prompts like "say pong" and "Reply with OK" show up near the top. These are almost certainly used by scanners to verify that an endpoint is actually running a live LLM before proceeding further. If the expected response isn't returned, the scanner likely bails out right there instead of continuing to more interesting probing (jailbreak attempts, tool/agent discovery, sensitive-data exfiltration attempts, etc.).
Implementing keyword-based response matching would:

• Pass the liveness check and encourage scanners into deeper, more revealing follow-up behavior
• Capture richer multi-turn attacker interaction patterns

[ta-061]

Thanks, I see the same pattern on my sensors: most sessions stop right after the canary prompt fails, so passing the check should get us much more interesting follow-up traffic. Let's do this.
Plan: add "response_rules" to the JSON config. First match wins, PROMPTPOT_RESPONSE_TEXT stays as the fallback.
"response_rules": [
{"contains": "pong", "response": "pong"},
{"contains": "reply with ok", "response": "OK"}
]

Two rules for the implementation:

• Matching is case-insensitive substring, not regex. Prompts are attacker input and Python's re has no timeout, so regex rules would allow ReDoS.
• Responses are always static strings from the config, never built from the prompt. This keeps the "never executes attacker input" guarantee.

Each event logs which rule matched (promptpot.matched_rule), so Kibana can show which canaries were passed and what the attacker did next. I'll also ship a small default ruleset (pong / OK / simple arithmetic) so it works without any config.

Do you want to send a PR? If not, I'll implement it next week.

[Taka-cst]

Agreed on avoiding regex here — since prompts are fully attacker-controlled, even a "simple looking" pattern could be crafted for catastrophic backtracking against Python's re (no timeout). Substring matching is the right call for safety.

+1 on the config-driven response_rules idea too — that solves the "different operators want different canned responses"　problem nicely without touching code. A couple of small additions that might help, still regex-free:

• multiple match types per rule (contains / starts_with / ends_with), still plain substring checks
• allow a list of keywords per rule (OR match)
• optionally pick randomly among several response strings for a rule, so repeated hits don't look identical

[ta-061]

All three sound good to me, and they keep the matching logic simple and easy to reason about.
match could support contains, starts_with, and ends_with, with contains as the default. Accepting either a string or a list for keywords and responses would also keep simple configurations concise.
Randomly choosing from multiple responses could make repeated hits less fingerprintable. It may still be useful to allow deterministic behavior, though, especially when reproducing and analyzing sessions. This intentionally would not cover checks that require echoing a per-request random token. Supporting those would mean building a response from attacker-controlled input, which goes against the current “never executes or reflects attacker input” guarantee.
From what I have seen so far, fixed canaries seem much more common anyway. For example, prompts such as Return exactly this text: LAYERCLOUD_AI_TEST_OK, Reply with exactly one word: blue, and even xyz123random are reused across many requests. A small ruleset based on observed traffic should therefore handle many of the liveness checks we currently see.
The promptpot.matched_rule field should also make it easy to spot new campaigns and add rules over time.

[Taka-cst]

Looked at promptpot.py — this looks quite feasible with the current structure. selected_model(parsed) already resolves a model name (from the request's model field, or MODEL_NAMES[0] as fallback) and it's already echoed back in the model key of every response, so the honeypot already "knows" which model to claim.

Idea: let response_rules support a {model} placeholder in the response
string, e.g.:

"response_rules": [
{"contains": "which model", "response": "{model}"}
]

completion_text() would need to take the resolved model as well, then do a simple substitution when the matched rule's response contains "{model}".

One thing worth being careful about: selected_model() currently trusts the request's own "model" field verbatim if present, so an attacker could send an arbitrary string there. To keep the "never reflects attacker input" guarantee airtight, I'd suggest only using the request's model value for substitution when it matches something in MODEL_NAMES, and falling back to MODEL_NAMES[0] otherwise. That way {model} only ever expands to a value this project controls.

[ta-061]

Nice, you read the code closer than I expected. Adopted: response strings support a {model} placeholder, resolved per request.

Your caveat is correct, I checked: selected_model() returns parsed["model"] verbatim whenever it's a non-empty string, so a naive {model} substitution would reflect arbitrary attacker text into the response body. I'll do exactly what you suggest: use the request's model value only when it's in MODEL_NAMES, otherwise fall back to MODEL_NAMES[0]. That way {model} only ever expands to operator-configured values.

Implementation-wise, {model} won't be special-cased: substitution goes through a small per-request dict of placeholders, so adding {version} or similar later is a one-line change. The rule stays the same for every placeholder: the dict only holds values this project controls, never anything derived from the request. That boundary is what keeps the whitelist meaningful.

One note: the model key in the response JSON will keep echoing the requested name verbatim, because that's what the real APIs do and changing it would hurt fidelity. The whitelist applies to rule text substitution only.

At this point you know the code well enough that the PR should be yours if you want it. Say the word and I'll leave the implementation to you, happy to review. Otherwise I'll start this week, with you credited for the design.

[ta-061]

Shipped in #23, thanks @Taka-cst. It's on main and in ghcr.io/ta-061/promptpot:latest now. The default ruleset in config.example.json covers the canaries we discussed, and matched_rule is in every event, so you can watch which canaries get passed and add rules when a new campaign shows up.