Skip to content

0.1.0

Choose a tag to compare

@SergeyShk SergeyShk released this 10 Nov 23:09
· 113 commits to master since this release
0ea3869

The first release of esMorph, a fast morphological analyzer of Spanish with a Rust core and a Python API. For a word form it finds the analyses in a dictionary built from Wiktionary: the lemma, the part of speech and the Universal Dependencies features in the conventions of the AnCora treebank. It orders them by probability and builds the forms of the word.

pip install esmorph

Analysis

  • Analyzer loads the dictionary in 3.6 ms. The file is mapped into memory, so processes with the same dictionary share it, and every further process adds 3.6 MB.
  • analyze gives the analyses of a word form, the most probable first. Each analysis has the lemma, the part of speech, the UD features, the probability, the source and the words of the token. The probabilities come from the training parts of AnCora and GSD.
  • Case does not matter for finding a word, though a word with a capital letter may also be a name. A word absent as written is looked for without its accents and with a lost diaeresis.
  • analyze_many analyzes lists of words, and lemmatize releases the GIL. Both read their input in chunks.
  • The analyzer and its analyses are pickled, so they work with multiprocessing and joblib. is_known checks a word form against the dictionary.

Parts of speech and features of UD

Parts of speech, features and lemmas follow the UD guidelines and, where they are silent, the conventions of UD Spanish AnCora:

  • determiners, pronouns, prepositions and conjunctions have the lemmas and features of AnCora;
  • the auxiliaries have both AUX and VERB analyses;
  • a participle is a verb and an adjective, and verbs have no -se in the lemma;
  • names have a lemma with a capital letter.

A verb with enclitic pronouns is analyzed as its verb, with the words of the token: dámelo is da + me + lo. So are the finite forms of older writing (díjole), the contractions del and al, and older ones such as desta.

Tokens out of the dictionary and unknown words

  • Rules analyze:

    • punctuation and symbols;
    • numbers in digits, with signs, fractions, powers, spaces between thousands and units (500cc);
    • ordinals (2.ª) and Roman numerals;
    • names, links, emails, hashtags and mentions;
    • abbreviations and letters.
  • A lowercase word that neither the dictionary nor the rules know is guessed:

    • by its last part after a hyphen;
    • by the word without a known prefix;
    • as a verb with clitics of a guessed host;
    • by the dictionary words with its ending.

    A guessed word has its forms too.

Inflection

  • forms() gives the forms of a lexeme in the order of its paradigm. inflect gives the form with the given UD features and keeps the other features of the analysis.
  • A variant, such as a form of vos or an older spelling, comes only from a variant.
  • If there is no such form, FormNotFoundError is raised. A wrong feature or value raises ValueError with the allowed values.

Comparison

Test parts of the UD 2.18 treebanks. The scores are the share of words whose first analysis has the right lemma, and the right part of speech, features and lemma together, %. Speed is text analysis on an Apple M3 Pro with Python 3.11.

System Lemma, AnCora Lemma, PUD Lemma, COSER All together, PUD Tokens per second
esMorph 97.80 97.74 94.49 87.90 1,228,661
simplemma 96.94 96.96 93.50 - 2,507,937
spaCy es_core_news_sm 98.66 96.69 93.39 83.59 44,100
Stanza 99.16 97.80 94.98 86.34 2,694

An analysis with the right part of speech and lemma is among the analyses of esMorph for 96-99 % of words. For the verbs of AnCora, PUD and COSER, esMorph gives the gold form of 99.65, 93.29 and 95.55 %, and ablaut of 98.87, 92.71 and 93.61 %. The methods and all the scores are on the Comparison page.

Platforms

  • Python wheels:
    • an abi3 wheel serves Python 3.11 and newer, and the free-threaded Python 3.14t has a wheel of its own;
    • Linux is covered on x86_64 and aarch64, manylinux and musllinux; macOS on Intel and Apple Silicon; Windows on x64 and arm64;
    • the dictionary is in the package.
  • Rust: the crates are esmorph and esmorph-dict-es on crates.io.
  • Elsewhere, the package builds from source with Rust 1.88 or newer.

Documentation: https://sergeyshk.github.io/esMorph/