Skip to content

v0.12.0 — 14 Languages, Placeholder Safety, Watermark Pipeline

Choose a tag to compare

@ksanyok ksanyok released this 26 Feb 14:54

What's New

5 New Languages (14 total)

  • Arabic (ar) — 81 bureaucratic, 80 synonyms, 49 AI connectors, 47 abbreviations
  • Chinese Simplified (zh) — 80 bureaucratic, 80 synonyms, 36 AI connectors
  • Japanese (ja) — 60+ per category, keigo to casual register replacements
  • Korean (ko) — 60+ per category, honorific to casual register
  • Turkish (tr) — 60+ per category, Ottoman to modern Turkish

Critical Bug Fixes

  • Placeholder safety — all 6 processing modules now skip placeholder tokens; no more leaked placeholders in output
  • 3-pass restore() — exact match, case-insensitive, orphan cleanup
  • HTML block protection — ul, ol, table, pre, blockquote preserved as single segments
  • Bare domain protection — site.com.ua, portal.kh.ua, example.co.uk etc.
  • Homoglyph fix — removed Cyrillic characters from special homoglyphs table (was corrupting all Cyrillic text)

Pipeline Improvements

  • Watermark cleaning — automatic first stage (12 stages total), removes zero-width chars, homoglyphs, invisible Unicode
  • Language detection — Arabic/CJK/Turkish script detection added

Tests

  • 1,509 tests passed (54 new)