0.2.0

esMorph 0.2.0 analyzes text, not only words. It fills the lemmas, parts of speech and UD features of a document of the anySeg segmenter, and the new esmorph command does the same for files. The release also brings dictionary verification and faster inflection, analysis of lists and pickling. The analyses of words are the same as in 0.1.0, apart from the order of the features of ordinals in digits.

pip install esmorph anyseg

Analysis of documents

• Analyzer.annotate(document) fills the lemma, UPOS and FEATS columns of an anyseg.Document with the most probable analysis of every word. In Rust it is Dictionary::annotate with the feature anyseg.
• The words of a multiword token are analyzed by the token: dime, split into di + me, is the imperative of decir with the pronoun me. Clitics after a verb are personal pronouns, and se before lo, la, los and las is a dative.
• A token that the segmenter leaves as one word gets the analysis of the whole token.
• Any object with the same column("form"), multiword_tokens and set_columns works too. The analysis runs without the GIL.
• On raw text segmented by anySeg 0.3, the CoNLL 2018 evaluation gives these F1 scores on the AnCora, PUD and COSER test sets:
  • UPOS: 93.13, 92.24 and 91.92;
  • lemma: 96.43, 96.92 and 95.21;
  • UFeats: 91.92, 83.28 and 90.43.
• annotate analyzes about 1.5 million words per second from Python.

The esmorph command

• The command is installed with the package and also runs as python -m esmorph. Without Python, cargo install esmorph --features cli installs it.
• Input: text, one token per line, or CoNLL-U.
• Output: lemmas, CoNLL-U, TSV with probabilities, or JSON Lines with all analyses.
• Streaming: text is read paragraph by paragraph, so the command can be kept open on a stream, and memory does not grow with the input.
• CoNLL-U input: an invalid sentence is written as it is and reported, and the rest of the file is analyzed.
• Safety: an output file or a standard output that is an input or the dictionary is refused, and a dictionary given with -d is verified.
• Speed: in one thread it lemmatizes about 1.8 million words per second (13.6 million words of a literary corpus in 7.6 s).

Dictionaries

• Analyzer(path, verify=True) and Dictionary::verify() check the whole dictionary, for a file from a source that is not trusted. A deliberately crafted file with a right checksum is rejected instead of making the analysis fail. The check takes about 20 ms, and pickle keeps it.
• Analyzer.dictionary_info describes the dictionary: the source, its snapshot and SHA-256, the license, the UD release and the treebanks of the probabilities, the number of word forms and the checksum. Its type is esmorph.DictionaryInfo.

Speed and memory

Measured against 0.1.0 in the same session (Apple M3 Pro, Python 3.11):

• analyze_many: 0.92-0.98 → 1.59-1.61 million words per second. The lists of analyses are no longer tracked by the garbage collector.
• inflect:
  • verbs: 348-355 → 559-567 thousand per second;
  • nouns and adjectives: +8 %.
• Import, loading and the first analysis: 3.5 → 3.1 ms.
• Pickles of analyses:
  • size: 40 % smaller (9.3 → 5.5 MB for 94,801 analyses);
  • dumps: −30 %;
  • loads: −18 %.

Fixes

• The features of ordinals in digits (2.ª) are in the order of UD.
• A pickle of an analysis keeps the path of its dictionary as a string, so it loads on another operating system. Pickles of 0.1.0 load.
• Pickling analyzers of dictionary files, or their analyses, from several threads at once no longer deadlocks. This affected 0.1.0 too.

Comparison

Test parts of the UD 2.18 treebanks. The scores are the share of words whose first analysis has the right lemma, and the right part of speech, features and lemma together, %. Speed is text analysis on an Apple M3 Pro with Python 3.11.

System	Lemma, AnCora	Lemma, PUD	Lemma, COSER	All together, PUD	Tokens per second
esMorph	97.80	97.74	94.49	87.90	1,744,647
simplemma	96.94	96.96	93.50	-	2,516,195
spaCy es_core_news_sm	98.66	96.69	93.39	83.59	44,659
Stanza	99.16	97.80	94.98	86.34	2,649

The methods and all the scores are on the Comparison page.

Platforms

The platforms are the same as in 0.1.0:

• Python wheels: an abi3 wheel serves Python 3.11 and newer, and the free-threaded 3.14t has a wheel of its own. Linux, macOS and Windows are covered.
• Rust: the crates esmorph and esmorph-dict-es are on crates.io.

Documentation: https://sergeyshk.github.io/esMorph/

0.1.0

The first release of esMorph, a fast morphological analyzer of Spanish with a Rust core and a Python API. For a word form it finds the analyses in a dictionary built from Wiktionary: the lemma, the part of speech and the Universal Dependencies features in the conventions of the AnCora treebank. It orders them by probability and builds the forms of the word.

pip install esmorph

Analysis

• Analyzer loads the dictionary in 3.6 ms. The file is mapped into memory, so processes with the same dictionary share it, and every further process adds 3.6 MB.
• analyze gives the analyses of a word form, the most probable first. Each analysis has the lemma, the part of speech, the UD features, the probability, the source and the words of the token. The probabilities come from the training parts of AnCora and GSD.
• Case does not matter for finding a word, though a word with a capital letter may also be a name. A word absent as written is looked for without its accents and with a lost diaeresis.
• analyze_many analyzes lists of words, and lemmatize releases the GIL. Both read their input in chunks.
• The analyzer and its analyses are pickled, so they work with multiprocessing and joblib. is_known checks a word form against the dictionary.

Parts of speech and features of UD

Parts of speech, features and lemmas follow the UD guidelines and, where they are silent, the conventions of UD Spanish AnCora:

• determiners, pronouns, prepositions and conjunctions have the lemmas and features of AnCora;
• the auxiliaries have both AUX and VERB analyses;
• a participle is a verb and an adjective, and verbs have no -se in the lemma;
• names have a lemma with a capital letter.

A verb with enclitic pronouns is analyzed as its verb, with the words of the token: dámelo is da + me + lo. So are the finite forms of older writing (díjole), the contractions del and al, and older ones such as desta.

Tokens out of the dictionary and unknown words

• Rules analyze:

  • punctuation and symbols;
  • numbers in digits, with signs, fractions, powers, spaces between thousands and units (500cc);
  • ordinals (2.ª) and Roman numerals;
  • names, links, emails, hashtags and mentions;
  • abbreviations and letters.

• A lowercase word that neither the dictionary nor the rules know is guessed:

  • by its last part after a hyphen;
  • by the word without a known prefix;
  • as a verb with clitics of a guessed host;
  • by the dictionary words with its ending.

  A guessed word has its forms too.

Inflection

• forms() gives the forms of a lexeme in the order of its paradigm. inflect gives the form with the given UD features and keeps the other features of the analysis.
• A variant, such as a form of vos or an older spelling, comes only from a variant.
• If there is no such form, FormNotFoundError is raised. A wrong feature or value raises ValueError with the allowed values.

Comparison

Test parts of the UD 2.18 treebanks. The scores are the share of words whose first analysis has the right lemma, and the right part of speech, features and lemma together, %. Speed is text analysis on an Apple M3 Pro with Python 3.11.

System	Lemma, AnCora	Lemma, PUD	Lemma, COSER	All together, PUD	Tokens per second
esMorph	97.80	97.74	94.49	87.90	1,228,661
simplemma	96.94	96.96	93.50	-	2,507,937
spaCy es_core_news_sm	98.66	96.69	93.39	83.59	44,100
Stanza	99.16	97.80	94.98	86.34	2,694

An analysis with the right part of speech and lemma is among the analyses of esMorph for 96-99 % of words. For the verbs of AnCora, PUD and COSER, esMorph gives the gold form of 99.65, 93.29 and 95.55 %, and ablaut of 98.87, 92.71 and 93.61 %. The methods and all the scores are on the Comparison page.

Platforms

• Python wheels:
  • an abi3 wheel serves Python 3.11 and newer, and the free-threaded Python 3.14t has a wheel of its own;
  • Linux is covered on x86_64 and aarch64, manylinux and musllinux; macOS on Intel and Apple Silicon; Windows on x64 and arm64;
  • the dictionary is in the package.
• Rust: the crates are esmorph and esmorph-dict-es on crates.io.
• Elsewhere, the package builds from source with Rust 1.88 or newer.

Documentation: https://sergeyshk.github.io/esMorph/