v4.5.0

Hindi and Nepali, opt-in lookalike folding, and a set of word-list and config additions, all contributed by @prolaxu (split from #53 into #59–#63). Thank you!

Nothing here changes results for single-language checks with default config, except for one small improvement to accented Latin text (below). inAllLanguages() now includes Hindi and Nepali.

New languages

• Hindi and Nepali (#60): Blasp::hindi(), Blasp::nepali(), or Blasp::in('english', 'hindi').
  • Both Devanagari and romanized spellings are covered.
  • Spelling variants (nukta, chandrabindu, vowel length) are handled by a new DevanagariNormalizer.
  • Plain words that would flag ordinary text (animal nouns, or words that are ordinary in other languages) were left out.

Opt-in matching options (#61, all off by default)

• homoglyphs folds Cyrillic and Greek lookalikes, fullwidth letters and unlisted accented letters before matching, so fuсk (Cyrillic с) and ｆｕｃｋ are caught. The original text is what gets masked.
  • Accent folding needs ext-intl or symfony/polyfill-intl-normalizer (suggested, not required).
  • Note that Cyrillic and Greek text is folded too.
• extended_substitutions adds 1→l, 7/+→t and 2→z.
• flexible_phrases makes a space in a list entry match any separator or none (sieg heil, sieg-heil, siegheil).

Config

• substitutions_append (#62) adds characters to the substitution table without copying the whole table:

'substitutions_append' => ['/u/' => ['v']], // "fvck" now matches

Word lists

• New entries (#63) for English, French, German and Spanish: hate terms, obfuscated slurs and slang not already covered. False positives are included for words that contain them (spoof, clitoris, cream pie, …).

Matching

• Unicode-aware word boundaries and separators (#59) for non-Latin scripts, so मा-दर-चोद matches like f-u-c-k.
• Accented Latin words: a match inside an accented word (e.g. "ass" in "classé") is now treated as embedded and skipped, like it already was for unaccented words.