Skip to content

v1.6.0

Choose a tag to compare

View all tags
@amirivojdan amirivojdan released this 27 May 17:05
· 15 commits to main since this release
v1.6.0
325e295

What's New in v1.6.0

This release introduces transliteration support, a suite of new text transforms for data augmentation and Persianization, faster and more reliable model downloads, and a refactored conjugation system. It also includes several enhancements to the Shekar Studio Web UI.

Highlights

  • Farsi ↔ Tajik Transliteration – added a new transliteration module powered by a quantized ByT5 model. Use FarsiToTajik and TajikToFarsi (both exposed at the package top level) to convert between Persian (Arabic script) and Tajik (Cyrillic script).

  • Persianizer for Loanword Replacement – the new Persianizer transform suggests and substitutes native Persian alternatives for foreign loanwords, backed by a curated mapping of 1,700+ entries sourced from beparsi.com. Use Persianizer() to auto-replace, or .suggest() to get ranked alternatives with positions.

  • NumberToWords Transform – the new NumberToWords transform converts numeric digits (both Persian and Arabic-Indic) into their Persian word form (e.g. ۱۲۳ → صد و بیست و سه).

  • Text Noise Transforms for Data Augmentation – three new transforms for synthetic noise generation, useful for training robust models and benchmarking:

    • KeyboardNoise – simulates typos based on Persian keyboard adjacency (substitution, insertion, deletion, repeat, shift).
    • OCRNoise – simulates OCR errors based on visual character confusions.
    • WhitespaceNoise – corrupts whitespace and ZWNJ structure (deletion or swapping between space ↔ ZWNJ).
      All three accept independent per-operation probabilities and a seed for reproducibility.

  • Rule-Based Informal Classifier – added RuleBasedInformalClassifier for fast, dependency-free detection of colloquial/informal Persian text using keyword matching over informal vocabulary and conjugated verb forms. Inspired by Persian-Informal-Text-Detector.

  • Iran Mirror with Latency-Based Selection – the model hub now selects the fastest available mirror automatically (currently shekar.ai and ir.shekar.ai), significantly improving download speed for users in Iran and reducing failures when one mirror is unreachable.

  • Shekar Studio Enhancements – the built-in web interface (shekar serve) now includes:

    • A new Transliteration panel with Persian (Tajik) support
    • Localized UI with English and Persian (Tajik) translations alongside Persian (Farsi)

  • Refactored Conjugation Engine – conjugation logic has been split into separate formal and informal pipelines for cleaner code, better test coverage, and more accurate informal verb generation.

Other Changes

  • Dropped Python 3.10 support; the package now requires Python 3.11+.
  • Bumped onnxruntime minimum to >=1.26.0.
  • Changed YaNormalizer default to 'standard'.
Assets 2
Loading