0.1.0

The first release of anySeg: segmentation of text into tokens, words and sentences with their positions in the source text, with a Rust core and a Python package without dependencies. The text is never changed, and the tokens follow the conventions of Universal Dependencies. This release brings Russian.

pip install anyseg

Segmenter

• Segmenter("ru") gives the tokens and the sentences of a text as spans of it: tokenize, sentenize, and segment for a document. Positions are offsets in the characters of the Python string; the Rust crate gives them in bytes.
• A segmenter is immutable. It can be shared between threads and pickled, and it segments long texts without the GIL, so threads run in parallel.
• Lists of your own: more abbreviations, and strings that are one token each (Segmenter("ru", abbreviations=["сущ"], tokens=["C++"])).
• tokenize, sentenize and segment are also functions of the module with a required lang.

Russian

• Tokens:
  • words with stress marks and soft hyphens;
  • hyphenated words split by the UD guidelines (particles -то, -ка, ranges, approximate numbers, compounds whose first part inflects);
  • numbers with groups of digits, decimals, dates and times; phone numbers and ISBNs; 20%-ный;
  • URLs, e-mail addresses, domains and file names, mentions and hashtags, emoticons and runs of emoji;
  • abbreviations and initials that keep their period.
• Sentences:
  • final marks with the closing quotes and brackets after them;
  • abbreviations that end a sentence (и т. д., в 1914 г.);
  • lines of dialogue, list items, headings and verse;
  • smiles and emoji at the end of a post, Markdown emphasis.
• Paragraphs and lines: paragraphs between blank lines. Every line is marked as a break of the author or a wrap at the width of the page, as in text from PDF.

Document

segment gives a Document:

• tokens, words with the columns of CoNLL-U, sentences, paragraphs and lines, as views of the text;
• set_columns fills the columns;
• to_conllu writes CoNLL-U and read_conllu reads it: a document read from CoNLL-U is written back as it was;
• Document.from_tokens makes a document of the tokens of another tool.

Quality and speed

On the test files of the Russian treebanks of UD 2.18 (F1 of the CoNLL 2018 script):

	SynTagRus	Taiga	GSD	PUD
Tokens	99.61	99.10	99.14	99.78
Sentence boundaries	99.46	95.94	98.66	99.75

Token F1 is the highest of all tools compared on Taiga, GSD and PUD. Sentence boundaries are above razdel on all four sets.

Speed on the raw text of SynTagRus test, one thread, Apple M3 Pro:

• from Python, 3.12 M tokens per second with sentences, six times faster than razdel;
• from Rust, 13.4 M tokens per second.

All tools and sets: Comparison.

Platforms

• Wheels for Linux (x86_64 and aarch64; manylinux and musllinux), macOS (Intel and Apple Silicon) and Windows (x64 and arm64).
• Python versions: one abi3 wheel serves Python 3.11 and newer; the free-threaded Python 3.14t and 3.15t have wheels of their own.
• Rust crate: anyseg, with Russian behind the ru feature.

Documentation: https://sergeyshk.github.io/anySeg/