PersianTextGuard 1.5.0

PersianTextGuard 1.5.0

New: Rust

PersianTextGuard is now on crates.io as persian-text-guard, used as persian_text_guard. It is a
native Rust crate for Rust 1.85 and later (edition 2024), with one dependency,
unicode-normalization, and #![forbid(unsafe_code)].

cargo add persian-text-guard

use persian_text_guard::{ProfanityFilter, WordList};

let filter = ProfanityFilter::with_defaults(WordList::persian_default());
assert!(filter.contains_profanity("ک.ی.ر"));
assert_eq!(filter.censor("kir and motherfucker"), "**** and ****");

It gives the same answers as the .NET, JavaScript and Python packages for every case in the shared
conformance corpus. Positions are byte offsets of the string you passed, so &text[m.range()] is the
matched region. A filter is immutable and Send + Sync, so threads share one without a lock; no function
panics for any input, and 100,000 arbitrary strings and 100,000 arbitrary byte sequences are run through
every entry point on each CI run. WordList::load and load_reader read a word-list file from a path or
any reader. See the
Rust README.

The crate's Unicode categories and invariant lower-casing come from tables generated from .NET 10 itself
and checked in CI, so they never drift with the compiler's Unicode version.

New: byte versions, for input that may not be UTF-8

The Rust crate adds contains_profanity_bytes, find_match_bytes, find_matches_bytes, censor_bytes
and censor_bytes_with, which take &[u8]. Invalid sequences are read as U+FFFD, exactly as
String::from_utf8_lossy reads them, and every byte outside a censored region is returned unchanged,
invalid ones included:

# use persian_text_guard::{ProfanityFilter, WordList};
# let filter = ProfanityFilter::with_defaults(WordList::persian_default());
assert_eq!(filter.censor_bytes(b"kir \xFF fuck"), b"**** \xFF ****");

The conformance corpus runner rules

A Rust String cannot hold a lone surrogate, and Rust has no missing string value. The runner
obligations in the corpus contract gained two readings for ports in that position: such an Input is built
as UTF-16 units with each lone surrogate replaced by U+FFFD (in the Input and in the expected text), a
mask that does not build into one character counts as refused, and a null Input is read as the empty
string. Both readings were measured against .NET itself and give its own results.

No corpus case changed, the format version stays 1, and the .NET, JavaScript and Python runners are
unaffected. The Rust runner runs all 523 cases with none reported as not applicable.

.NET, JavaScript and Python

NuGet PersianTextGuard 1.5.0 and persian-text-guard 1.5.0 on npm and PyPI are identical to 1.4.0
apart from the version number: no change to the API, behaviour or bundled word lists. Each package is now
validated against 1.4.0, the previous release.

---

خلاصهٔ فارسی

جدید: راست. این کتابخانه اکنون برای زبان راست هم منتشر شده است، با نام persian-text-guard روی مخزن crates.io، برای راست ۱٫۸۵ و بالاتر، تنها با یک وابستگی و بدون هیچ کد ناامن. برای هر پیام همان پاسخی را می‌دهد که بسته‌های دات‌نت، جاوااسکریپت و پایتون می‌دهند، و جایگاه‌ها را با بایت‌های همان رشته‌ای می‌شمارد که به آن داده‌اید.

cargo add persian-text-guard

جدید: نسخه‌های بایتی. برای ورودی‌هایی که شاید یوتی‌اف‑۸ معتبر نباشند، پنج تابع تازه روی بایت‌ها کار می‌کنند: دنباله‌های نامعتبر مثل جایگزینی خود زبان خوانده می‌شوند و بایت‌های بیرون از ناحیهٔ سانسورشده دست‌نخورده برمی‌گردند.

قواعد اجرای مجموعهٔ آزمون. چون رشتهٔ راست نمی‌تواند نیم‌جانشین تنها را نگه دارد و مقدار تهی هم ندارد، دو خوانش تازه به قواعد اجراکننده‌ها افزوده شد؛ هیچ موردی از مجموعهٔ آزمون تغییر نکرد و هر ۵۲۳ مورد در راست اجرا می‌شوند.

دات‌نت، جاوااسکریپت و پایتون. نسخهٔ ۱٫۵٫۰ این سه بسته جز شمارهٔ نسخه با ۱٫۴٫۰ یکسان است و رفتار و فهرست کلمات آن‌ها تغییری نکرده است.

PersianTextGuard 1.4.0

PersianTextGuard 1.4.0

New: Python

PersianTextGuard is now on PyPI as persian-text-guard, imported as persian_text_guard. It is pure
Python with full type hints, for CPython 3.11 and later, free-threaded 3.14 included, with no
dependencies.

pip install persian-text-guard

from persian_text_guard import ProfanityFilter, WordList

filter = ProfanityFilter(WordList.persian_default())
filter.contains_profanity("ک.ی.ر")  # True
filter.censor("kir and motherfucker")  # '**** and ****'

It gives the same answers as the .NET and JavaScript packages for every case in the shared conformance
corpus. Positions are Python string indexes (code points). A filter is immutable and safe to share
across threads, and WordList.load reads a word-list file from a path or an open file. See the
Python README.

The PyPI files are published from CI with trusted publishing, and carry attestations that name this
repository and workflow.

New conformance cases

The corpus gained ten cases for characters outside the Basic Multilingual Plane: emoji, a CJK
Extension B letter, mathematical bold letters and a Deseret letter, next to or inside a banned word, and
inside two ordinary Persian and English messages that must stay unflagged. They pin what every port
already did, as .NET reads such text in UTF-16 units, and change no behaviour in any package.

.NET and JavaScript

NuGet PersianTextGuard 1.4.0 and npm persian-text-guard 1.4.0 are identical to 1.3.0 apart from the
version number: no change to the API, behaviour or bundled word lists. The .NET package is now validated
against 1.3.0, the previous release, and the npm package's API is checked against v1.3.0.

---

خلاصهٔ فارسی

جدید: پایتون. این کتابخانه اکنون برای پایتون هم منتشر شده است، با نام persian-text-guard روی مخزن PyPI، برای پایتون ۳٫۱۱ و بالاتر و بدون هیچ وابستگی. برای هر پیام همان پاسخی را می‌دهد که بسته‌های دات‌نت و جاوااسکریپت می‌دهند.

pip install persian-text-guard

موارد تازهٔ آزمون. ده مورد تازه به مجموعهٔ آزمون مشترک اضافه شده است، برای ایموجی و نویسه‌های بیرون از صفحهٔ پایهٔ یونیکد. این موارد رفتار فعلی همهٔ نسخه‌ها را ثبت می‌کنند و چیزی را تغییر نمی‌دهند.

دات‌نت و جاوااسکریپت. نسخهٔ ۱٫۴٫۰ این دو بسته جز شمارهٔ نسخه با ۱٫۳٫۰ یکسان است و رفتار و فهرست کلمات آن‌ها تغییری نکرده است.

PersianTextGuard 1.3.0

PersianTextGuard 1.3.0

New: JavaScript and TypeScript

PersianTextGuard is now on npm as persian-text-guard, with TypeScript types included. It works with
import and require, on Node.js 22 and later, and in browsers through a bundler, with no dependencies.

npm install persian-text-guard

import { ProfanityFilter, WordList } from 'persian-text-guard';

const filter = new ProfanityFilter(WordList.persianDefault);
filter.containsProfanity('ک.ی.ر'); // true
filter.censor('kir and motherfucker'); // '**** and ****'

It gives the same answers as the .NET package for every case in the shared conformance corpus. Positions
are JavaScript string indexes. See the JavaScript README.

Changes for .NET users

These change behaviour only for the inputs named here:

• Fixed: messages containing Unicode noncharacters no longer throw. In 1.2.0, ContainsProfanity,
  FindMatch, FindMatches, Censor and PersianNormalizer.Normalize threw ArgumentException for a
  message containing U+FFFE (on .NET 8 and .NET 10), or any of the 66 noncharacters (on .NET Framework
  4.8). They now return normally. Every message that worked in 1.2.0 gives the same result.
• Changed: word-list section headings must be category names. WordList.Parse and WordList.Load
  accept [insult] in any letter case, as before. A number such as [3] or [+4], which 1.2.0 read as
  the category with that internal number, now throws FormatException as an unknown category. The
  bundled lists are not affected.

No other change to the .NET API, behaviour or bundled word lists. The package passes API compatibility
validation against 1.2.0.

---

خلاصهٔ فارسی

جدید: جاوااسکریپت و تایپ‌اسکریپت. PersianTextGuard اکنون با نام persian-text-guard روی npm منتشر
شده است، همراه با تایپ‌های تایپ‌اسکریپت و بدون هیچ وابستگی. با import و require، روی Node.js 22 و
بالاتر و در مرورگر کار می‌کند و برای هر پیام همان پاسخی را می‌دهد که بستهٔ ‎.NET‎ می‌دهد.

تغییرات برای کاربران ‎.NET‎:

• رفع اشکال: پیامی که «نانویسه» (noncharacter) یونیکد مثل U+FFFE دارد دیگر خطا نمی‌دهد. در نسخهٔ
  1.2.0 این پیام‌ها باعث ArgumentException می‌شدند. پاسخ همهٔ پیام‌هایی که قبلاً درست کار می‌کردند
  تغییری نکرده است.
• تغییر: در فایل فهرست کلمات، عنوان هر بخش باید نام دسته باشد، مثل [insult]. عددی مثل [3] که
  نسخهٔ 1.2.0 می‌پذیرفت، اکنون دستهٔ ناشناخته به شمار می‌آید. فهرست‌های همراه بسته تغییری نکرده‌اند.

رابط برنامه‌نویسی و رفتار ‎.NET‎ جز این دو مورد تغییری نکرده است.

v1.2.0

Find every banned word in a message, with where it is, and censor messages.

New

FindMatches — every match, with its position

foreach (var match in filter.FindMatches("sh1t and f u c k"))
{
    Console.WriteLine($"{match.Word.Text} ({match.Word.Category}) at {match.Index}, length {match.Length}");
}
// shit (Profanity) at 0, length 4
// fuck (Profanity) at 9, length 7

• Every banned word, ordered by position, each with its entry, category and the evasion undone to
  find it.
• Index and Length refer to the text exactly as you passed it, not a normalized copy, and
  cover whole words, so they can be used directly to highlight or slice the original string.
• Overlapping entries become one match: "motherfucker" is one match, not "motherfucker" plus the
  "fuck" inside it.

Censor — hide the words, keep the message

filter.Censor("kir and motherfucker");   // "**** and ****"
filter.Censor("جنده‌ها رو ببین");         // "**** رو ببین"
filter.Censor("this is kir", '#');       // "this is ####"

• Whole words are hidden — a Persian suffix or the rest of a longer word goes with the banned
  word, so no fragment is left — and disguised words are hidden with their separators.
• Every mask is four characters, so a reader cannot tell how long the hidden word was.
• The output is always clean: checking it with the same filter finds nothing, even where masking
  would otherwise join the surrounding letters into a new word.
• Clean text is returned as the same string; nothing outside a hidden word is normalized.
• Letters, digits, whitespace and control characters are refused as mask characters.

FindMatch reports a position too

ProfanityMatch has new Index and Length properties, and FindMatch fills them.

Behaviour changes

• ProfanityMatch equality now includes Index and Length. A FindMatch result no longer
  equals a hand-built new ProfanityMatch(word, evasion). Compare Word and Evasion instead if
  you relied on that. Which entry and evasion FindMatch returns is unchanged.
• Nothing else changes in what ContainsProfanity or FindMatch return: all 1.1.0 tests pass
  unmodified, and a new consistency suite checks that all four methods agree on every message in
  the test corpus.

Performance

BenchmarkDotNet, .NET 10, Intel Core i7-9700K. See benchmarks.md.

Operation	1.1.0	1.2.0
Short clean message	2.399 µs	2.391 µs
Long clean message (60 words)	21.870 µs	21.302 µs
Message with evasions	1.735 µs	1.905 µs
FindMatches, clean short message	—	2.368 µs
FindMatches, message with three banned words	—	6.567 µs
Censor, 60-word message with three banned words	—	92.967 µs

Positions are only worked out for a message that contains a banned word, so FindMatches and
Censor cost the same as ContainsProfanity on clean text. The message-with-evasions case is
+0.17 µs: joining spaced-out letters now also records where each letter sits so a match can be
located in the original text.

v1.1.0

Fixes what testing 1.0.1 turned up: one crash, several ways past the filter, and ordinary messages it rejected.

Fixed

• Crash on truncated input. A message cut in the middle of an emoji threw ArgumentException out of string.Normalize. User input can no longer throw.
• Words hidden by neighbouring characters. «کیر», کیر😂, @kir passed as clean. The tokenizer now splits on Unicode punctuation and symbols.
• Numbers read as words. 455 folded to "ass", so prices, room numbers and phone numbers were profanity. Digits are only read as letters in a run that has letters in it.
• New evasions caught: masked letters (f**k, c*nt, f@ck, ahole), punctuation inside a word (ج.نده, kos_kesh), a word split once (fu ck, کی ر), accents and letters with no decomposition (fück, ƒuck, 🅵🆄🅲🅺), emoji inside a word (f🖕ck).
• Ordinary words no longer rejected: shiitake, Matsushita, sniggered, Pussycat Dolls. shit, pussy, nigger, whore and slut are whole-word entries now, with their compounds listed.
• Persian suffixes are handled by the matcher: جنده‌ها, کیرتون, کونیا match without entries of their own.

Added

Every entry now has a WordCategory — Profanity, Sexual, Insult, Slur, Harassment or Mild:

WordList.PersianDefault   // everything except Mild — the sane default
WordList.All              // Mild too
WordList.Bundled(WordCategory.Slur, WordCategory.Harassment)

filter.FindMatch("nigger")!.Word.Category;   // WordCategory.Slur

The list grew from about 400 to about 1,260 entries across persian.txt, finglish.txt and english.txt, drawing on Surge AI's Obscenity List, LDNOOBW, amirshnll/Persian-Swear-Words, kaveh-dev/persian-bad-words and dsojevic/profanity-list — all credited in THIRD-PARTY-NOTICES.md.

Faster

Whole-word entries are looked up by token rather than scanned one by one: a short message takes 2.4 µs instead of 4.9 µs, against three times as many entries.

Upgrading

Additive, with one behaviour change: shit is reported as a whole-word rather than an anywhere entry. Ordinary words such as آشغال and damn are categorised Mild and are not in WordList.PersianDefault; pass WordList.All to keep blocking them.