feat: obfuscation is no longer a bypass (#26)

[hermanngeorge15]

Closes #26 (SCAN-05, engine E1). Completes Phase 3's detection work.

Every pattern matched raw bytes, so a find-and-replace defeated the whole library. All five evasions from the issue now land:

Evasion	Example
separator	ignore-all-previous-instructions
spacing	i g n o r e a l l p r e v i o u s
homoglyph	іgnore all previous instructions (Cyrillic і)
fullwidth	ｉｇｎｏｒｅ all previous instructions
zero-width	ig<U+200B>nore all previous instructions

One pass beats all five — far more leverage than a literal pattern per spelling, since an attacker has unbounded spellings and we don't.

Offsets are the whole problem

Normalizing is easy; reporting is not. Every byte of normalized text carries the offset it came from, so findings name a real line and quote the original:

:1 CRITICAL  Attempts to override agent instructions  (PI001)
             matched: "ignore-all-previous-instructions"

Telling a user their file contains ignore all previous instructions when it visibly contains the hyphenated form is a quote they can't search for, about a file they're being asked to fix.

Three bugs, all found by running it rather than reading it

The confusable table maps m onto rn. The skeleton rewrote "normal clean text" → "norrnal clean text" — every ASCII word in every document, quietly mangled before matching. It now applies only to non-ASCII, which is the only place it has a job: ASCII is the form it maps toward.

\s matches a newline. Scanning the whole normalized document let a pattern span line breaks — quietly making this a second multi-line pass, but without the paragraph boundaries #24 established. It joined across a blank list item that #24 correctly refuses to cross. Now matched line by line.

A trailing full stop counted as separator injection, so the pass ran on essentially every document for nothing. A separator only counts between two word characters: the hyphens in ignore-all-previous are load-bearing, the full stop ending "…about prompts." is punctuation.

Measured

Zero new false positives on the ecosystem tree.

The six findings that disappear are this repo's own pattern library: a name: field like ignore-previous-instructions normalizes into the attack it names. That's correct behaviour on any other document, so they're suppressed with the tool's own inline directives — and still reported as suppressed, so nothing goes silent.

	before	after
500 clean files (PERF-01)	20ms	20ms — clean ASCII short-circuits
This repo	30ms	40ms (budget: 200ms)
Eight dense repos	0.77s	1.67s

The last row is the honest cost: a deep package path is separator-shaped, so hyphen-heavy trees defeat the short-circuit.

Documented limit

Fully despaced text (i g n o r e a l l p r e v i o u s, no double spaces) is not rejoined. The result would be ignoreallprevious, and every pattern joins its words with \s+ — matching that means rewriting the pattern set, not the input. Recorded rather than papered over.

Spacing rejoin is also decided per line, not per word: judging words separately left "i g n o r e a l l" as "ignore a l l", because no per-word threshold is both low enough for a three-letter word and high enough to leave "a b" in prose alone.

Tests

10 new, one per evasion plus one per bug above. A corpus specimen guards the separator fold against ordinary hyphenated technical prose — and caught me quoting a payload in its own HTML comment header, which scores 1.0 by design. Full suite 22 binaries green, clippy clean.