promptbadger 0.3.1

Packaging release: promptbadger is now on PyPI. The rules, the classifier and their metrics are unchanged from 0.3.0.

pip install promptbadger

• Uploaded by a new publish.yml workflow with PyPI trusted publishing (no stored token). Before uploading, it checks that the release tag matches the package version, then builds the package and tests the built wheel in a clean environment.
• Package metadata: SPDX license expression, project URLs, supported Python versions.
• README images and links are absolute URLs so they render on PyPI.

Full Changelog: v0.3.0...v0.3.1

promptbadger 0.3.0

Adds an optional ML classifier layer, PB-ML, alongside the rule pack. The rules are unchanged from 0.2.0, and so are their metrics.

pip install "promptbadger @ git+https://github.com/cyberchup/promptbadger.git@v0.3.0"

What's new

Classifier layer: PB-ML (opt-in). A TF-IDF + logistic regression model that scores the prompt as a whole and adds one detection to the same noisy-OR score as the rules. Its confidence is the model's probability, and the detection lists the terms that drove it. Enable it with Scanner(classifier=True) or promptbadger scan --ml. It is medium from p ≥ 0.572 (suspicious on its own) and high from p ≥ 0.834 (malicious on its own), and adds nothing below that.

• Inference is pure Python from a 443 KiB bundled model: no new runtime dependency. scikit-learn is only needed to retrain (.[train]).
• About 160 ms to load, about 3 ms for a 29,000-character prompt. It runs on scan() only. Documents and email (scan_context) would need their own measurement.
• run_eval.py gains --ml and --ml-only. eval/train_classifier.py and eval/classifier_eval.py reproduce the model and its held-out report.

Demo. A checkbox adds the classifier, plus two examples: a reworded override only the classifier catches, and a benign story prompt it flags (a known false positive).

Measured

Built to a protocol written before training: trained on train splits only, with any text that also appears in an evaluation set dropped. Thresholds were set from a false-positive budget on training data, then each held-out set was scored once with nothing changed afterwards.

Recall at suspicious or worse:

Dataset	Rules	Rules + classifier
reshabhs/SPML_Chatbot_Prompt_Injection (independent)	0.182	0.403
Lakera/gandalf_ignore_instructions (independent)	0.517	0.741

False positives are the price. On 52,002 benign Alpaca instructions the rules flag nothing and the classifier flags 162 (0.31%), 13 of them at malicious. Most are creative-writing requests and ordinary security vocabulary. On SPML it adds 1 false positive in 3,470 benign prompts. That profile suits a review queue better than a block decision, which is why the classifier is off by default.

The same-source test splits (deepset, safe-guard, jailbreak-classification) score much higher with the classifier, but it trained on their train splits, so those numbers are optimistic. The independent sets above are the generalisation numbers. Full tables: Classifier (v0.3).

All rule reports were regenerated at 0.3.0 and reproduce the 0.2.0 numbers exactly.

Limits

A bag-of-words model learns each dataset's vocabulary. Paraphrased attacks with no telltale words still get through: the demo's "known miss" example scores 0.14. Next up is a stronger model or an LLM-as-judge layer for inputs the fast layers mark suspicious, each measured under its own protocol.

Full reports: eval/results/

Full Changelog: v0.2.0...v0.3.0

promptbadger 0.2.0

First tagged release since v0.1.0. It adds obfuscation handling, indirect-injection scanning, output checks and four more direct-injection rules (18 rules in total).

pip install "promptbadger @ git+https://github.com/cyberchup/promptbadger.git@v0.2.0"

What's new

Obfuscation handling (decode-and-rescan). Each input is also scanned in decoded views: leetspeak, letter spacing, base64 / hex / binary, escape sequences and HTML entities, Cyrillic look-alike letters, Unicode tag characters, rot13 and reversed text. A rule fires once, in the first view that matches. A match in a decoded view adds ATLAS AML.T0068 and records view and decoded on the detection, and match spans point into the raw input. New rule PI-018: an encoded payload paired with a decode-and-obey instruction.

Indirect injection: scan_context(). Scans retrieved documents, emails, web pages and tool output. Rules now carry a scope. PI-015 (content addresses the AI reading it), PI-016 (content tells the model what to do with the user) and PI-017 (instructions hidden in markup) run on context only, because talking to the AI is normal in a user's own prompt but a red flag inside an email.

Output checks: scan_output().

• PB-CANARY: plant a token from make_canary() in the system prompt and flag any model reply that contains it, even if the token is spaced out or de-hyphenated.
• PB-EXFIL-IMAGE / PB-EXFIL-LINK: data carried in rendered image or link URLs to untrusted hosts (EchoLeak style). Tune with --trusted-domain.
• PB-SMUGGLE: Unicode tag characters (ASCII smuggling). This also runs on prompts and context.

More direct-injection coverage. PI-012 context reset, PI-013 RAG grounding override, PI-014 secret and credential requests (medium, so it reaches suspicious on its own), and wider DE / ES / FR / RU / BCS wording in existing rules.

SIEM. Events gain Direction and optional User and SessionId fields. New CLI flags: --direction, --canary, --trusted-domain, --user, --session-id. Sentinel content adds context_injection.kql and output_leak.kql (canary leaks, URL exfiltration), and the hunting queries gain a per-session one hunting query that combines per-turn scores with the scanner's noisy-OR to surface slow-burn conversations. All queries use column_ifexists(), so tables built from v0.1 events still work.

Breaking change

OWASP tags move to the LLM Top 10 2026 edition. LLM01:2025 becomes LLM01:2026, and system prompt leakage moves from LLM07:2025 to LLM08:2026 (Hidden Context Exposure; LLM07 is now Misinformation). Update any SIEM filters or workbooks that key on OWASP IDs.

Measured

Held-out test splits, alerting on suspicious or worse. None were used to tune rules (one caveat below).

Dataset	Recall v0.1.0 → v0.2.0	Precision	FPR
deepset/prompt-injections	0.200 → 0.250	1.000	0.000
xTRam1/safe-guard-prompt-injection	0.260 → 0.282*	1.000	0.000
jackhhao/jailbreak-classification	0.655 → 0.655	1.000	0.000

*The PI-014 part of this gain (0.263 → 0.282) is optimistic: similar safe-guard misses had been read before the rule was written.

• Indirect injection, LLMail-Inject phase 2 (21,007 attacks): recall 0.149 with input rules only → 0.165 with scan_context and v0.2 views. 0 false positives on 203 LLMail benign emails, 16,545 Enron emails and 4,467 Wikipedia passages.
• Obfuscation: 95–100% of held-out injections that are caught in plain form are still caught after each supported transform, against 0–2% with views off (tag characters excepted: PB-SMUGGLE flags those either way). The decoders were built against this table, so it is a development measurement of technique coverage, not recall. Caesar shifts other than rot13 and words split into chunks are not decoded (0%).
• False positives: 0 suspicious-or-worse on 52,002 Alpaca prompts.
• Cost: views add about 0.1 ms to a short prompt and about 50% to a long email (3.6 → 5.4 ms).

Limits

The regex layer is a high-precision, low-recall tier. Paraphrased attacks, long custom-persona jailbreaks and unsupported encodings get through; see Known gaps. Next up is v0.3: an ML classifier layer combined with the rule scores.

Full reports: eval/results/

Full Changelog: v0.1.0...v0.2.0