0.2.0

The second release of esTS adds the statistics that read the annotation of Universal Dependencies - morphology, syntax and cohesion - and puts every class of the library on a spaCy pipeline as a component.

Requires Python 3.11 or newer. The distribution on PyPI is pyests, the package it installs is ests: pip install pyests, import ests. The statistics of 0.1 still need no trained model; the three new ones do, and python -m spacy download es_core_news_sm is what they ask for.

No incompatible changes. The API of 0.1 is untouched and everything here is an addition: the six classes it shipped behave as they did, and the helpers that grew (iter_doc_tokens, get_nlp, has_words, is_verbal_noun) are new names beside the old ones.

Morphology

• MorphStats - parts of speech and fifteen grammatical features of Universal Dependencies as the Spanish models annotate them: case, definiteness, degree, gender, mood, numeral type, number, person, polarity, politeness, possessive, pronoun type, reflexive, tense and verb form, with the whole annotation kept in tags in the CoNLL-U form. get_stats counts them, explain_text parses a text word by word, print_stats prints them with their descriptions.
• get_markers and print_markers - the markers of Spanish as shares of their own base, so texts of different lengths compare directly: the moods among the finite forms, the non-finite forms among the verb forms, ser among the two copulas, the adverbs in -mente among the adverbs. The base of the verb forms is the verbs and the auxiliaries, so a participle the model reads as an adjective stays out of it, and the copula of a passive - fue escrito, es financiado - is no choice between the copulas.
• What the models can and cannot do is measured and written down: es_core_news_sm, es_core_news_md and es_core_news_lg find the imperative in 2, 3 and 4 of 24 imperative sentences, so p_imperative under-reports whichever one is used, and the enclitics (dámelo, vámonos) are mis-analysed by all three.

Syntax

• SyntaxStats - 30 statistics on the dependency tree: the dependency distances of Liu (2008) with their mean, deviation and share of adjacent links, the depth of the tree, leaves and subtrees, the valency of the finite verbs, the coordination chains, the clauses and the subordinate clauses, the modifiers of a noun and the ratio of nouns to verbs, in the terms of Ivanov, Solnyshkina and Solovyev (2018).
• The constructions the Spanish guides to clear language warn about: the passive with ser and with se and the agentless passive, the participial and the gerund clauses with their lengths, the chains of de (el aumento de la eficiencia del uso de los recursos), the impersonal se, the infinitives, the words of negation and the split predicates (hacer una revisión instead of revisar), which split_predicates lists as it finds them.
• Every rule is written against what the models actually annotate: they use almost no subtypes of the relations, so a passive is recognized by the lemma of its auxiliary and an agent by its preposition; a participle that modifies a noun is an adjective for them; the periphrases with a gerund hang from their verb as xcomp or advcl.

Cohesion

• CohesionStats - referential cohesion in the manner of Coh-Metrix and of Coh-Metrix-Esp: the overlap of nouns, of arguments and of content words between adjacent sentences and between all pairs of sentences, binary and proportional, the share of pronouns and of demonstratives, p_given, and temporal cohesion, where the mood takes the place of the aspect Spanish has no feature for.
• 255 discourse markers in the classification of Martín Zorraquino and Portolés, shipped with the package: causal, adversative, concessive, temporal, additive, conditional and reformulative, each primary or secondary, with the density per 1000 words and the occurrences in connector_spans. A marker that heads a prepositional phrase (antes de la reunión, sobre todo el texto) is not counted, and a one-word marker is checked against its part of speech.
• The overlaps are computed over bit masks and histograms instead of every pair of sentences, so the time grows with the number of occurrences and not with the square of the number of sentences.

Components of a spaCy pipeline

• Six factories - ests_basic, ests_readability, ests_diversity, ests_morph, ests_syntax, ests_cohesion - each putting its object of statistics into doc._.<name>, so a text is annotated and measured in one pass. The name of the pipe is free and is what the extension is called: nlp.add_pipe("ests_basic", name="basic") gives doc._.basic.
• They are entry points of spacy_factories, so a pipeline saved with them loads with spacy.load(path) in a process that never imports the library. A document with no words passes through untouched, so one empty text does not stop nlp.pipe over a corpus.
• ests_readability takes its preset and ests_diversity its five parameters, validated when the component is added; ests_readability can also take the basic statistics of another component instead of computing them a second time.
• The objects they attach are not serializable: Doc.to_bytes(), DocBin(store_user_data=True) and nlp.pipe(n_process > 1) need the user data left out, or get_stats() kept apart.

Quality

• 1051 tests, coverage 100 % by lines and branches, ruff and mypy clean; the CI runs on Python 3.11 to 3.14 and on Linux, Windows and macOS.
• The release went through a QA pass of its own, and the ten findings of it are fixed or decided and written down; the documentation is bilingual, and every example in it is executed and compared with its printed output.

Documentation: https://sergeyshk.github.io/esTS/ · Commits: 0.1.0...0.2.0