Skip to content

Sindhi Tokenizer v1.0.0 — First Stable Release

Latest
Latest

Choose a tag to compare

View all tags
@alinawazcode alinawazcode released this 04 Nov 20:02
· 1 commit to main since this release
v1.0.0
f448ef0
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Sindhi Tokenizer v1.0.0 — First Stable Release

This is the first versioned release of Sindhi Tokenizer, a corpus-driven SentencePiece Unigram tokenizer developed for Sindhi written in the Perso-Arabic script.

Highlights

  • Four pretrained tokenizer configurations: 32K, 64K, 100K, and 150K vocabularies.
  • SentencePiece Unigram architecture with byte fallback.
  • Full observed-character coverage during training.
  • Flagship 64K tokenizer with a macro-average fertility of 1.147 tokens per word.
  • Evaluation against 14 baseline tokenizer entries across four evaluation registers.
  • Documented training pipeline, corpus provenance, and benchmark methodology.
  • Python evaluation utilities and an interactive Gradio demonstration.
  • Comprehensive technical documentation and reproducibility resources.

Recommended Model

The 64K vocabulary model is the recommended balance between tokenization efficiency and vocabulary size.

Installation

git clone https://github.com/alinawazcode/sindhi-tokenizer.git
cd sindhi-tokenizer
pip install -e .

Model and Documentation

Citation

Please cite this software when using the tokenizer, evaluation code, or reported findings in academic research.

See CITATION.cff in the repository for author and citation metadata.

Authors

Ali Nawaz, Muqadas Fareed, Soonh Sadaf, And Attaullah Sahito.

Department of Computer Science, Shaikh Ayaz University, Shikarpur, Pakistan.

License

MIT License for the software code. See the repository's dataset and model documentation for their respective licensing terms.

Note: The associated research manuscript is a draft and has not yet been published.

Assets 2
Loading