Repository navigation
0.1.0
The first release of esMorph, a fast morphological analyzer of Spanish with a Rust core and a Python API. For a word form it finds the analyses in a dictionary built from Wiktionary: the lemma, the part of speech and the Universal Dependencies features in the conventions of the AnCora treebank. It orders them by probability and builds the forms of the word.
pip install esmorphAnalysis
Analyzerloads the dictionary in 3.6 ms. The file is mapped into memory, so processes with the same dictionary share it, and every further process adds 3.6 MB.analyzegives the analyses of a word form, the most probable first. Each analysis has the lemma, the part of speech, the UD features, the probability, the source and the words of the token. The probabilities come from the training parts of AnCora and GSD.- Case does not matter for finding a word, though a word with a capital letter may also be a name. A word absent as written is looked for without its accents and with a lost diaeresis.
analyze_manyanalyzes lists of words, andlemmatizereleases the GIL. Both read their input in chunks.- The analyzer and its analyses are pickled, so they work with
multiprocessingand joblib.is_knownchecks a word form against the dictionary.
Parts of speech and features of UD
Parts of speech, features and lemmas follow the UD guidelines and, where they are silent, the conventions of UD Spanish AnCora:
- determiners, pronouns, prepositions and conjunctions have the lemmas and features of AnCora;
- the auxiliaries have both
AUXandVERBanalyses; - a participle is a verb and an adjective, and verbs have no
-sein the lemma; - names have a lemma with a capital letter.
A verb with enclitic pronouns is analyzed as its verb, with the words of the token: dámelo is da + me + lo. So are the finite forms of older writing (díjole), the contractions del and al, and older ones such as desta.
Tokens out of the dictionary and unknown words
-
Rules analyze:
- punctuation and symbols;
- numbers in digits, with signs, fractions, powers, spaces between thousands and units (
500cc); - ordinals (
2.ª) and Roman numerals; - names, links, emails, hashtags and mentions;
- abbreviations and letters.
-
A lowercase word that neither the dictionary nor the rules know is guessed:
- by its last part after a hyphen;
- by the word without a known prefix;
- as a verb with clitics of a guessed host;
- by the dictionary words with its ending.
A guessed word has its forms too.
Inflection
forms()gives the forms of a lexeme in the order of its paradigm.inflectgives the form with the given UD features and keeps the other features of the analysis.- A variant, such as a form of
vosor an older spelling, comes only from a variant. - If there is no such form,
FormNotFoundErroris raised. A wrong feature or value raisesValueErrorwith the allowed values.
Comparison
Test parts of the UD 2.18 treebanks. The scores are the share of words whose first analysis has the right lemma, and the right part of speech, features and lemma together, %. Speed is text analysis on an Apple M3 Pro with Python 3.11.
| System | Lemma, AnCora | Lemma, PUD | Lemma, COSER | All together, PUD | Tokens per second |
|---|---|---|---|---|---|
| esMorph | 97.80 | 97.74 | 94.49 | 87.90 | 1,228,661 |
| simplemma | 96.94 | 96.96 | 93.50 | - | 2,507,937 |
| spaCy es_core_news_sm | 98.66 | 96.69 | 93.39 | 83.59 | 44,100 |
| Stanza | 99.16 | 97.80 | 94.98 | 86.34 | 2,694 |
An analysis with the right part of speech and lemma is among the analyses of esMorph for 96-99 % of words. For the verbs of AnCora, PUD and COSER, esMorph gives the gold form of 99.65, 93.29 and 95.55 %, and ablaut of 98.87, 92.71 and 93.61 %. The methods and all the scores are on the Comparison page.
Platforms
- Python wheels:
- an abi3 wheel serves Python 3.11 and newer, and the free-threaded Python 3.14t has a wheel of its own;
- Linux is covered on x86_64 and aarch64, manylinux and musllinux; macOS on Intel and Apple Silicon; Windows on x64 and arm64;
- the dictionary is in the package.
- Rust: the crates are
esmorphandesmorph-dict-eson crates.io. - Elsewhere, the package builds from source with Rust 1.88 or newer.
Documentation: https://sergeyshk.github.io/esMorph/