Skip to content

0.2.0

Choose a tag to compare

View all tags
@SergeyShk SergeyShk released this 12 Nov 04:58
· 36 commits to master since this release
0.2.0
173d16a
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

esMorph 0.2.0 analyzes text, not only words. It fills the lemmas, parts of speech and UD features of a document of the anySeg segmenter, and the new esmorph command does the same for files. The release also brings dictionary verification and faster inflection, analysis of lists and pickling. The analyses of words are the same as in 0.1.0, apart from the order of the features of ordinals in digits.

pip install esmorph anyseg

Analysis of documents

  • Analyzer.annotate(document) fills the lemma, UPOS and FEATS columns of an anyseg.Document with the most probable analysis of every word. In Rust it is Dictionary::annotate with the feature anyseg.
  • The words of a multiword token are analyzed by the token: dime, split into di + me, is the imperative of decir with the pronoun me. Clitics after a verb are personal pronouns, and se before lo, la, los and las is a dative.
  • A token that the segmenter leaves as one word gets the analysis of the whole token.
  • Any object with the same column("form"), multiword_tokens and set_columns works too. The analysis runs without the GIL.
  • On raw text segmented by anySeg 0.3, the CoNLL 2018 evaluation gives these F1 scores on the AnCora, PUD and COSER test sets:
    • UPOS: 93.13, 92.24 and 91.92;
    • lemma: 96.43, 96.92 and 95.21;
    • UFeats: 91.92, 83.28 and 90.43.
  • annotate analyzes about 1.5 million words per second from Python.

The esmorph command

  • The command is installed with the package and also runs as python -m esmorph. Without Python, cargo install esmorph --features cli installs it.
  • Input: text, one token per line, or CoNLL-U.
  • Output: lemmas, CoNLL-U, TSV with probabilities, or JSON Lines with all analyses.
  • Streaming: text is read paragraph by paragraph, so the command can be kept open on a stream, and memory does not grow with the input.
  • CoNLL-U input: an invalid sentence is written as it is and reported, and the rest of the file is analyzed.
  • Safety: an output file or a standard output that is an input or the dictionary is refused, and a dictionary given with -d is verified.
  • Speed: in one thread it lemmatizes about 1.8 million words per second (13.6 million words of a literary corpus in 7.6 s).

Dictionaries

  • Analyzer(path, verify=True) and Dictionary::verify() check the whole dictionary, for a file from a source that is not trusted. A deliberately crafted file with a right checksum is rejected instead of making the analysis fail. The check takes about 20 ms, and pickle keeps it.
  • Analyzer.dictionary_info describes the dictionary: the source, its snapshot and SHA-256, the license, the UD release and the treebanks of the probabilities, the number of word forms and the checksum. Its type is esmorph.DictionaryInfo.

Speed and memory

Measured against 0.1.0 in the same session (Apple M3 Pro, Python 3.11):

  • analyze_many: 0.92-0.98 → 1.59-1.61 million words per second. The lists of analyses are no longer tracked by the garbage collector.
  • inflect:
    • verbs: 348-355 → 559-567 thousand per second;
    • nouns and adjectives: +8 %.
  • Import, loading and the first analysis: 3.5 → 3.1 ms.
  • Pickles of analyses:
    • size: 40 % smaller (9.3 → 5.5 MB for 94,801 analyses);
    • dumps: −30 %;
    • loads: −18 %.

Fixes

  • The features of ordinals in digits (2.ª) are in the order of UD.
  • A pickle of an analysis keeps the path of its dictionary as a string, so it loads on another operating system. Pickles of 0.1.0 load.
  • Pickling analyzers of dictionary files, or their analyses, from several threads at once no longer deadlocks. This affected 0.1.0 too.

Comparison

Test parts of the UD 2.18 treebanks. The scores are the share of words whose first analysis has the right lemma, and the right part of speech, features and lemma together, %. Speed is text analysis on an Apple M3 Pro with Python 3.11.

System Lemma, AnCora Lemma, PUD Lemma, COSER All together, PUD Tokens per second
esMorph 97.80 97.74 94.49 87.90 1,744,647
simplemma 96.94 96.96 93.50 - 2,516,195
spaCy es_core_news_sm 98.66 96.69 93.39 83.59 44,659
Stanza 99.16 97.80 94.98 86.34 2,649

The methods and all the scores are on the Comparison page.

Platforms

The platforms are the same as in 0.1.0:

  • Python wheels: an abi3 wheel serves Python 3.11 and newer, and the free-threaded 3.14t has a wheel of its own. Linux, macOS and Windows are covered.
  • Rust: the crates esmorph and esmorph-dict-es are on crates.io.

Documentation: https://sergeyshk.github.io/esMorph/

Assets 2
Loading