0.9.0

The ninth release of esTS moves onto anyTS 0.4, the language-independent core of text statistics, and gives its new metrics a Spanish layer. It also:

• counts the clauses by the rules of L2SCA, with the Spanish subordinating conjunctions;
• stops reading the si of an indirect question as a condition;
• adds the readability formulas of other languages, rolling Delta and Zeta in segments;
• adds the cohesion and the syntax to the comparison of corpora;
• highlights the anaphoric el mismo of the administrative style.

Requires Python 3.11 or newer. Install with pip install -U pyests and import with import ests. The dependencies are now anyts>=0.4.5,<0.5 and spacy>=3.8.15,<4, both installed with the package.

New

• New metrics of the core, with Spanish descriptions, cards and tests.
  • SyntaxStats, 14 keys:
    • the L2SCA indices by T-units: t_units_per_sent, mean_t_unit_len, clauses_per_t_unit, p_complex_t_units, verb_phrases_per_t_unit, coordinate_phrases_per_clause, complex_nominals_per_clause;
    • the words before the root (words_before_root);
    • the prepositional chains (prep_chains_per_sent, max_prep_chain_len);
    • the nominal subjects and objects before their verb (p_preverbal_subjects, p_preverbal_objects);
    • the similarity of adjacent sentences (pos_edit_distance, syntax_similarity).
  • CohesionStats, 6 keys: the overlaps of the nouns, the arguments and the content words with the next two sentences (*_2_sent) and between adjacent paragraphs (*_para), as in TAACO, and the attribute n_paragraphs.
  • DiversityStats: Orlov's Z (orlov_z) and the burstiness of the frequent lexemes (burstiness); calc_mamtld takes step.
  • BasicStats: the entropies of the word lengths in letters and in syllables (letters_entropy, syllables_entropy).
  • The functions of the new metrics are re-exported from the modules of their classes: calc_forward_overlap, calc_orlov_z, calc_burstiness, is_prep_complement, calc_edit_distance, calc_multiset_similarity, dependency_triples.
• Clauses.
  • An infinitive under csubj, advcl or ccomp heads an infinitive clause, not a subordinate one (salió para ayudar, es difícil decirlo).
  • A clause under conj or parataxis with a Spanish subordinating conjunction is subordinate. The model attaches such a clause after y, pero or sino as a conjunct (y como dormía no quiso despertarle).
    • The conjunctions are in SUBORDINATING_MARKS of ests.constants, the hook is SyntaxStats.is_subordinating_mark and is_subordinating_mark(token) of ests.syntax_stats.
    • que after sino coordinates (NON_SUBORDINATING_PREV_PHRASES).
  • C/T, CP/C and CN/C divide by the clauses of L2SCA (n_l2sca_clauses): finite clauses with a subject of their own. A sentence without a clause (Sí., a heading, the name of a speaker in a play) is one T-unit and one clause, as the fragments of L2SCA.
• The non-conditional si. is_connector_blocked(token) of ests.cohesion_stats blocks:
  • the si of an indirect question after a verb of CONNECTOR_QUESTION_VERBS: no sé si vendrá, a ver si llega, Indique si desea..., La comisión valorará si...;
    • a verb with a direct object before si still takes a condition (lo veré si tengo tiempo);
  • si after an opening ¡ (¡Si ya te lo dije!);
  • si after como, cual, vaya, maldito (como si fuera, vaya si lo sé).
    CohesionStats, the connectors layer of the highlighting and find_connectors(..., blocked=) follow the same rule. On 300 fragments of SpanishLiterature the conditional connectors fall by a quarter.
• Paragraphs as lines.
  • CohesionStats starts a paragraph after every line break, as TAACO does. A line without content words, such as the number of a chapter, joins the next paragraph.
  • With blank_lines=True a paragraph starts only after a blank line, for a text whose lines are wrapped inside its paragraphs, such as the texts of SpanishLiterature.
  • ests_cohesion takes the same switch: nlp.add_pipe("ests_cohesion", config={"blank_lines": True}).
• Formulas of other languages.
  • UnadaptedReadabilityStats counts these formulas over the Spanish syllables and letters:
    • FORCAST, Linsear Write, Powers-Sumner-Kearl, EFLAW;
    • the four Wiener Sachtextformeln;
    • Dickes-Steiwer, Gulpease, Fucks, Strain, ELF, Wheeler-Smith.
  • It has a component, ests_unadapted_readability.
  • The formulas are not calibrated for Spanish. A Spanish word has more syllables, so they read Spanish texts as harder: they have no scales and stay out of the consensus grade and of text_features.
• Stylometry and corpora.
  • rolling_delta gives the Delta distance of the windows of a text to reference authors, and zeta_segments the markers of Zeta in segments.
  • The plots delta_histogram_plot, rolling_delta_plot and zeta_scatter_plot.
  • split_paragraphs of ests.extractors.
  • keyness(n=) takes keywords as n-grams, and collocations(side=) the collocates on one side.
• Comparison of corpora. text_features, corpus_features and compare_corpora give the group cohesion_: 163 features instead of 132.
  • With a pipeline that has a parser (partial(text_features, nlp=nlp)) they also give the group syntax_: 209 features.
  • The cohesion and the syntax count the sentences of the group sents_.
• MorphStats on the frame of the core.
  • get_stats, explain_text, print_stats and print_markers are those of the core.
  • The Spanish markers and the parse of a string stay in esTS.
  • The object of a parsed novel holds 10.9 MB instead of 17.6 MB.
• The anaphoric el mismo. The highlighting layer anaphoric_mismo of the group "Officialese" marks se archivó el mismo, la duración del mismo and contra la misma cabe recurso, which the Diccionario panhispánico de dudas takes for unnecessary.
  • The identity, the emphasis and the comparison are not marked: es el mismo, el mismo día, él mismo, el mismo que el año pasado.
  • The functions are is_anaphoric_mismo(token) and find_anaphoric_mismo(doc) of ests.visualizers.highlight. The layer needs a Doc with the dependency parse, the parts of speech and the lemmas.
• Components. The factories require the annotations their classes read: ests_syntax the parse, the parts of speech and the lemmas, ests_morph and ests_cohesion the parts of speech and the lemmas, ests_lexical the parts of speech. So nlp.analyze_pipes() names a pipeline that lacks them.
• Documentation. The cards of the new metrics and functions, the page of the formulas of other languages, the stylometric plots on twelve works of Galdós, Pardo Bazán and Unamuno, and the cohesion and the syntax on the page of the comparison of corpora. The notebooks are run again on 0.9.

Incompatible changes

The public classes and functions keep their names. Their signatures change only by new optional last parameters: keyness(n=), collocations(side=), CohesionStats(blank_lines=), find_connectors(blocked=), find_connector_highlights(doc=), calc_mamtld(step=).

• Dependencies. anyTS 0.4.5 or newer within 0.4 is required.
• New keys. get_stats, to_dataframe and print_stats of SyntaxStats, CohesionStats, DiversityStats and BasicStats give the new metrics, so the tables of print_stats are longer.
• Clauses. subordinate_clauses_per_sent and p_complex_sents fall and infinitive_clauses_per_sent grows on texts with infinitives under advcl, ccomp or csubj. A coordinated clause with a subordinating conjunction is subordinate now.
• Connectors. The non-conditional si is no connector, so connectors, connectors_intra, connectors_conditional and connectors_primary are lower.
• Paragraphs. CohesionStats starts a paragraph after every line break. Pass blank_lines=True for the paragraphs between blank lines. n_paragraphs counts the paragraphs with content words.
• Features of the corpora.
  • There are 163 features instead of 132 (diversity_burstiness and the group cohesion_), and 209 with a parser.
  • The Holm correction runs over more features, so p_holm changes.
  • The bootstrap intervals take a generator of their own for every feature.
• A Doc without the parts of speech. SyntaxStats needs them now.
  • A Doc of a pipeline without its morphologizer still has SPACE on its line breaks. LexicalStats, StyleStats.verbal_nouns, SyntaxStats and CohesionStats raise SourceError for it, where they used to accept it without the parts of speech.
  • function_words_profile tags such a Doc with the model.
• Messages. DiversityStats, MorphStats, LexicalStats and VerseStats raise SourceTypeError with the text of the core: The data source cannot be of type int. A Span gets the hint pass span.as_doc().
• MorphStats.
  • print_stats joins the descriptions of a value of several values with a slash (Interrogative / Relative).
  • print_markers prints the table of the core.
  • get_markers counts a feature of several values (Mood=Ind,Sub) in the share of each value.
• collocations sorts the pairs of an equal score by frequency.
• Pickles. A SyntaxStats, CohesionStats or MorphStats pickled by 0.8 has to be computed again.

Numbers

On a fixed set of texts, 633 of 4620 results changed against 0.8.0: 563 only gain keys, and 66 results are new. The 70 whose old values move are all in these categories. Everything else is bitwise equal.

• Clauses: SyntaxStats and ests_syntax, 38 results. On 100 excerpts of the prose of SpanishLiterature (441 thousand tokens):
  • the subordinate clauses fall from 22,440 to 19,392;
  • the infinitive clauses grow from 2,196 to 5,415.
• The non-conditional si: CohesionStats and ests_cohesion, 20 results. The change of connectors equals that of connectors_conditional everywhere.
• Messages, 3 results: DiversityStats(42), the list of the metrics of windowed, collocations(min_freq=0).
• Order of collocations, 5 results.
• Features of the corpora, 4 results. The 132 old columns of corpus_features are bitwise equal. In compare_features and compare_corpora the intervals and p_holm of one feature change.

Fixes

• A Doc without the parts of speech (above) used to pass the check of the lexical and the stylistic statistics by its SPACE tags.
• The si of a question was counted as a condition (QA of 0.8.0).
• corpus_features raises SourceTypeError when the function of the features gives no mapping of numbers by name, instead of the errors of pandas.

Speed

• The new metrics have a cost on a parsed novel: SyntaxStats.get_stats takes about 50 % longer than in 0.8, CohesionStats.get_stats about 37 % longer.
• A window of 1000 words of text_features takes 3-18 % longer by default and 14-24 % longer with a parser, which now also gives the syntax.
• import ests, MorphStats, the highlighting and the corpus functions take the time of 0.8.

Quality

• 2251 tests pass with 100% line and branch coverage.

• ruff and mypy are clean.

• CI runs on Python 3.11–3.14, on Linux, Windows and macOS, and with the lowest versions of the dependencies.

• A QA pass before the release covered:

  • the rules of 0.9 on 80 excerpts of prose;
  • about 250 calls of the new classes on 20 edge strings and 13 kinds of Doc, and a fuzzing of 800 texts;
  • pickles and copies of 11 classes;
  • the import time and the speed against 0.8.0;
  • the lowest and the newest dependencies, Python 3.11–3.14t, the notebooks and the demo;
  • the documentation in English and in Spanish, with 328 examples run.

  Its findings are fixed in anyTS 0.4.5 and in this release.

Links

• Documentation: https://sergeyshk.github.io/esTS/
• Notebooks: https://github.com/SergeyShk/esTS/tree/master/examples
• Demo: https://huggingface.co/spaces/SergeyShk/esTS
• PyPI: https://pypi.org/project/pyests/0.9.0/
• anyTS: https://github.com/SergeyShk/anyTS