Skip to content

feat: Unicode normalization pass — defeats homoglyph, spacing and separator evasion #26

Description

@hermanngeorge15

Every pattern is matched against raw bytes, so trivial obfuscation defeats the whole library. All of these are currently undetected:

ignore-all-previous-instructions        # separator injection
i g n o r e   a l l   p r e v i o u s   # spacing
іgnore all previous instructions        # Cyrillic і (U+0456) homoglyph
ignore all previous instructions      # fullwidth
ig<U+200B>nore all previous instructions # zero-width interleaved

Proposed fix (engine E1)

A normalization pass run in addition to the raw pass:

  1. NFKC normalize (folds fullwidth, ligatures, compatibility forms)
  2. Strip zero-width, bidi control, soft hyphen, variation selectors, combining marks
  3. Fold Unicode confusables to their ASCII skeleton (unicode-security crate)
  4. Collapse repeated whitespace and separator punctuation (-, _, ., *)

Re-run all patterns against the normalized text and map hits back to original byte offsets for reporting. Tag such findings with a normalized: true flag and a slightly reduced confidence.

This single engine defeats homoglyph, spacing, separator, fullwidth and zero-width-interleave evasion at once — far more leverage than adding more literal patterns.

Crates: unicode-normalization, unicode-security.

Backlog engine: E1

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1High prioritydetection-gapAttack class the scanner missesenhancementNew feature or requestphase-3Milestone phase 3 — Signal Quality

    Type

    No type

    Projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions