Skip to content

rougers 0.2.0

Choose a tag to compare

View all tags
@kholdrex kholdrex released this 11 Oct 03:19
· 3 commits to main since this release
v0.2.0
1ed3a3b
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Opt-in tokenizers for languages other than English. rouge-score keeps only [a-z0-9], so Russian, Arabic, Hindi or Chinese text scores zero; the default tokenizer and its scores stay exactly as before.

from rouge_score_rs import rouge_scorer, tokenizers

scorer = rouge_scorer.RougeScorer(["rouge1", "rougeL"], tokenizer=tokenizers.UnicodeTokenizer())
chinese = rouge_scorer.RougeScorer(["rouge1", "rougeL"], tokenizer=tokenizers.CharacterTokenizer())
  • UnicodeTokenizer (profile unicode-v1): Unicode word boundaries after NFKC and full
    case folding, for languages in any script.
  • CharacterTokenizer (profile character-v1): one token per grapheme cluster after the
    same normalization, for character-level ROUGE such as in Chinese.
  • Both profiles use pinned Unicode 17.0 data, so tokens do not depend on the platform or
    the Rust version.
  • Both run natively, including in parallel score_batch. The default tokenizer is
    unchanged.
  • Rust: Tokenizer::unicode, Tokenizer::character and Tokenizer::profile.
    Minimum Rust version 1.88.

This release also replaces the synthetic benchmark with measurements on CNN/DailyMail, XSum and PubMed.

pip install -U rouge-score-rs
Assets 2
Loading