0.2.0

anySeg 0.2.0 brings integrations: a spaCy bridge, batches of texts on several threads and the anyseg command. Segmentation does not change: the tokens and sentences are those of 0.1.0.

pip install anyseg              # the package and the command
pip install "anyseg[spacy]"     # with the spaCy bridge

spaCy

• The tokenizer anyseg.Tokenizer.v1 and the component anyseg_sentencizer register with spaCy through entry points. They are used by name in a configuration, without importing anyseg, and a saved pipeline loads with them.

• The tokenizer keeps the text of the Doc as given, and the whitespace as spaCy keeps it. nlp.tokenizer.pipe segments many texts in batches without the GIL.

• The component sets the sentence starts before the parser, which keeps them. It works with any tokenizer and reports sents_p, sents_r and sents_f in an evaluation.

• anyseg.spacy.to_doc and from_doc convert between a Doc and a document, with lemmas, parts of speech, tags, features (in the order of UD), heads and relations.

• With ru_core_news_sm 3.8.0 on the test files of UD 2.18, from raw text, each paragraph a document (F1):

  	SynTagRus	Taiga	GSD	PUD
  Sentences, spaCy → with anySeg	95.39 → 98.53	80.42 → 89.92	85.85 → 96.32	96.45 → 99.25
  LAS, spaCy → with anySeg	81.83 → 82.96	71.55 → 73.39	75.80 → 79.34	86.14 → 87.26

• The tokenizer gives 2.7 M tokens per second, nearly ten times the tokenizer of spaCy.

• spaCy has no wheels for the free-threaded Python 3.14t and 3.15t.

Batches and positions

• Segmenter.pipe(texts, threads=4, batch_size=1000) gives the documents of many texts in order. The batches are segmented without the GIL on several threads.
• starts and ends of tokens, sentences, paragraphs and lines give their positions as lists, many times faster than reading the parts one by one.

Documents

• Less memory. The columns of CoNLL-U are kept in one buffer, and read_conllu makes a document at every # newdoc. On SynTagRus test, read_conllu keeps 133 bytes per word against 317, with a peak of 37 MB against 93.
• The parts of a document are read without a lock, so threads that share a document no longer contend for it on the free-threaded Python.
• Documents stay out of the cycle collector: a million documents come three times faster.

The anyseg command

• anyseg --lang ru [--format tokens|sentences|conllu] [-o FILE] [--abbreviations FILE] [--tokens FILE] [file ...] writes tokens, sentences or CoNLL-U.
  • It comes with the package and runs as python -m anyseg.
  • cargo install anyseg --features cli installs it without Python.
• The text is segmented paragraph by paragraph, with the same result as for the whole text, so the command can be kept open in a pipeline.
• A text without blank lines is segmented in pieces of 2 MiB, so the memory stays bounded.

Rust crate

• The feature cli adds the module anyseg::cli and the binary anyseg. The library API is compatible with 0.1.0.
• The wheels carry the licenses of the Rust crates linked into the module.

Documentation: spaCy, Command, Segmenter.

0.1.0

The first release of anySeg: segmentation of text into tokens, words and sentences with their positions in the source text, with a Rust core and a Python package without dependencies. The text is never changed, and the tokens follow the conventions of Universal Dependencies. This release brings Russian.

pip install anyseg

Segmenter

• Segmenter("ru") gives the tokens and the sentences of a text as spans of it: tokenize, sentenize, and segment for a document. Positions are offsets in the characters of the Python string; the Rust crate gives them in bytes.
• A segmenter is immutable. It can be shared between threads and pickled, and it segments long texts without the GIL, so threads run in parallel.
• Lists of your own: more abbreviations, and strings that are one token each (Segmenter("ru", abbreviations=["сущ"], tokens=["C++"])).
• tokenize, sentenize and segment are also functions of the module with a required lang.

Russian

• Tokens:
  • words with stress marks and soft hyphens;
  • hyphenated words split by the UD guidelines (particles -то, -ка, ranges, approximate numbers, compounds whose first part inflects);
  • numbers with groups of digits, decimals, dates and times; phone numbers and ISBNs; 20%-ный;
  • URLs, e-mail addresses, domains and file names, mentions and hashtags, emoticons and runs of emoji;
  • abbreviations and initials that keep their period.
• Sentences:
  • final marks with the closing quotes and brackets after them;
  • abbreviations that end a sentence (и т. д., в 1914 г.);
  • lines of dialogue, list items, headings and verse;
  • smiles and emoji at the end of a post, Markdown emphasis.
• Paragraphs and lines: paragraphs between blank lines. Every line is marked as a break of the author or a wrap at the width of the page, as in text from PDF.

Document

segment gives a Document:

• tokens, words with the columns of CoNLL-U, sentences, paragraphs and lines, as views of the text;
• set_columns fills the columns;
• to_conllu writes CoNLL-U and read_conllu reads it: a document read from CoNLL-U is written back as it was;
• Document.from_tokens makes a document of the tokens of another tool.

Quality and speed

On the test files of the Russian treebanks of UD 2.18 (F1 of the CoNLL 2018 script):

	SynTagRus	Taiga	GSD	PUD
Tokens	99.61	99.10	99.14	99.78
Sentence boundaries	99.46	95.94	98.66	99.75

Token F1 is the highest of all tools compared on Taiga, GSD and PUD. Sentence boundaries are above razdel on all four sets.

Speed on the raw text of SynTagRus test, one thread, Apple M3 Pro:

• from Python, 3.12 M tokens per second with sentences, six times faster than razdel;
• from Rust, 13.4 M tokens per second.

All tools and sets: Comparison.

Platforms

• Wheels for Linux (x86_64 and aarch64; manylinux and musllinux), macOS (Intel and Apple Silicon) and Windows (x64 and arm64).
• Python versions: one abi3 wheel serves Python 3.11 and newer; the free-threaded Python 3.14t and 3.15t have wheels of their own.
• Rust crate: anyseg, with Russian behind the ru feature.

Documentation: https://sergeyshk.github.io/anySeg/