0.2.0

anySeg 0.2.0 brings integrations: a spaCy bridge, batches of texts on several threads and the anyseg command. Segmentation does not change: the tokens and sentences are those of 0.1.0.

pip install anyseg              # the package and the command
pip install "anyseg[spacy]"     # with the spaCy bridge

spaCy

• The tokenizer anyseg.Tokenizer.v1 and the component anyseg_sentencizer register with spaCy through entry points. They are used by name in a configuration, without importing anyseg, and a saved pipeline loads with them.

• The tokenizer keeps the text of the Doc as given, and the whitespace as spaCy keeps it. nlp.tokenizer.pipe segments many texts in batches without the GIL.

• The component sets the sentence starts before the parser, which keeps them. It works with any tokenizer and reports sents_p, sents_r and sents_f in an evaluation.

• anyseg.spacy.to_doc and from_doc convert between a Doc and a document, with lemmas, parts of speech, tags, features (in the order of UD), heads and relations.

• With ru_core_news_sm 3.8.0 on the test files of UD 2.18, from raw text, each paragraph a document (F1):

  	SynTagRus	Taiga	GSD	PUD
  Sentences, spaCy → with anySeg	95.39 → 98.53	80.42 → 89.92	85.85 → 96.32	96.45 → 99.25
  LAS, spaCy → with anySeg	81.83 → 82.96	71.55 → 73.39	75.80 → 79.34	86.14 → 87.26

• The tokenizer gives 2.7 M tokens per second, nearly ten times the tokenizer of spaCy.

• spaCy has no wheels for the free-threaded Python 3.14t and 3.15t.

Batches and positions

• Segmenter.pipe(texts, threads=4, batch_size=1000) gives the documents of many texts in order. The batches are segmented without the GIL on several threads.
• starts and ends of tokens, sentences, paragraphs and lines give their positions as lists, many times faster than reading the parts one by one.

Documents

• Less memory. The columns of CoNLL-U are kept in one buffer, and read_conllu makes a document at every # newdoc. On SynTagRus test, read_conllu keeps 133 bytes per word against 317, with a peak of 37 MB against 93.
• The parts of a document are read without a lock, so threads that share a document no longer contend for it on the free-threaded Python.
• Documents stay out of the cycle collector: a million documents come three times faster.

The anyseg command

• anyseg --lang ru [--format tokens|sentences|conllu] [-o FILE] [--abbreviations FILE] [--tokens FILE] [file ...] writes tokens, sentences or CoNLL-U.
  • It comes with the package and runs as python -m anyseg.
  • cargo install anyseg --features cli installs it without Python.
• The text is segmented paragraph by paragraph, with the same result as for the whole text, so the command can be kept open in a pipeline.
• A text without blank lines is segmented in pieces of 2 MiB, so the memory stays bounded.

Rust crate

• The feature cli adds the module anyseg::cli and the binary anyseg. The library API is compatible with 0.1.0.
• The wheels carry the licenses of the Rust crates linked into the module.

Documentation: spaCy, Command, Segmenter.