0.9.0

The ninth release of esTS moves onto anyTS 0.4, the language-independent core of text statistics, and gives its new metrics a Spanish layer. It also:

• counts the clauses by the rules of L2SCA, with the Spanish subordinating conjunctions;
• stops reading the si of an indirect question as a condition;
• adds the readability formulas of other languages, rolling Delta and Zeta in segments;
• adds the cohesion and the syntax to the comparison of corpora;
• highlights the anaphoric el mismo of the administrative style.

Requires Python 3.11 or newer. Install with pip install -U pyests and import with import ests. The dependencies are now anyts>=0.4.5,<0.5 and spacy>=3.8.15,<4, both installed with the package.

New

• New metrics of the core, with Spanish descriptions, cards and tests.
  • SyntaxStats, 14 keys:
    • the L2SCA indices by T-units: t_units_per_sent, mean_t_unit_len, clauses_per_t_unit, p_complex_t_units, verb_phrases_per_t_unit, coordinate_phrases_per_clause, complex_nominals_per_clause;
    • the words before the root (words_before_root);
    • the prepositional chains (prep_chains_per_sent, max_prep_chain_len);
    • the nominal subjects and objects before their verb (p_preverbal_subjects, p_preverbal_objects);
    • the similarity of adjacent sentences (pos_edit_distance, syntax_similarity).
  • CohesionStats, 6 keys: the overlaps of the nouns, the arguments and the content words with the next two sentences (*_2_sent) and between adjacent paragraphs (*_para), as in TAACO, and the attribute n_paragraphs.
  • DiversityStats: Orlov's Z (orlov_z) and the burstiness of the frequent lexemes (burstiness); calc_mamtld takes step.
  • BasicStats: the entropies of the word lengths in letters and in syllables (letters_entropy, syllables_entropy).
  • The functions of the new metrics are re-exported from the modules of their classes: calc_forward_overlap, calc_orlov_z, calc_burstiness, is_prep_complement, calc_edit_distance, calc_multiset_similarity, dependency_triples.
• Clauses.
  • An infinitive under csubj, advcl or ccomp heads an infinitive clause, not a subordinate one (salió para ayudar, es difícil decirlo).
  • A clause under conj or parataxis with a Spanish subordinating conjunction is subordinate. The model attaches such a clause after y, pero or sino as a conjunct (y como dormía no quiso despertarle).
    • The conjunctions are in SUBORDINATING_MARKS of ests.constants, the hook is SyntaxStats.is_subordinating_mark and is_subordinating_mark(token) of ests.syntax_stats.
    • que after sino coordinates (NON_SUBORDINATING_PREV_PHRASES).
  • C/T, CP/C and CN/C divide by the clauses of L2SCA (n_l2sca_clauses): finite clauses with a subject of their own. A sentence without a clause (Sí., a heading, the name of a speaker in a play) is one T-unit and one clause, as the fragments of L2SCA.
• The non-conditional si. is_connector_blocked(token) of ests.cohesion_stats blocks:
  • the si of an indirect question after a verb of CONNECTOR_QUESTION_VERBS: no sé si vendrá, a ver si llega, Indique si desea..., La comisión valorará si...;
    • a verb with a direct object before si still takes a condition (lo veré si tengo tiempo);
  • si after an opening ¡ (¡Si ya te lo dije!);
  • si after como, cual, vaya, maldito (como si fuera, vaya si lo sé).
    CohesionStats, the connectors layer of the highlighting and find_connectors(..., blocked=) follow the same rule. On 300 fragments of SpanishLiterature the conditional connectors fall by a quarter.
• Paragraphs as lines.
  • CohesionStats starts a paragraph after every line break, as TAACO does. A line without content words, such as the number of a chapter, joins the next paragraph.
  • With blank_lines=True a paragraph starts only after a blank line, for a text whose lines are wrapped inside its paragraphs, such as the texts of SpanishLiterature.
  • ests_cohesion takes the same switch: nlp.add_pipe("ests_cohesion", config={"blank_lines": True}).
• Formulas of other languages.
  • UnadaptedReadabilityStats counts these formulas over the Spanish syllables and letters:
    • FORCAST, Linsear Write, Powers-Sumner-Kearl, EFLAW;
    • the four Wiener Sachtextformeln;
    • Dickes-Steiwer, Gulpease, Fucks, Strain, ELF, Wheeler-Smith.
  • It has a component, ests_unadapted_readability.
  • The formulas are not calibrated for Spanish. A Spanish word has more syllables, so they read Spanish texts as harder: they have no scales and stay out of the consensus grade and of text_features.
• Stylometry and corpora.
  • rolling_delta gives the Delta distance of the windows of a text to reference authors, and zeta_segments the markers of Zeta in segments.
  • The plots delta_histogram_plot, rolling_delta_plot and zeta_scatter_plot.
  • split_paragraphs of ests.extractors.
  • keyness(n=) takes keywords as n-grams, and collocations(side=) the collocates on one side.
• Comparison of corpora. text_features, corpus_features and compare_corpora give the group cohesion_: 163 features instead of 132.
  • With a pipeline that has a parser (partial(text_features, nlp=nlp)) they also give the group syntax_: 209 features.
  • The cohesion and the syntax count the sentences of the group sents_.
• MorphStats on the frame of the core.
  • get_stats, explain_text, print_stats and print_markers are those of the core.
  • The Spanish markers and the parse of a string stay in esTS.
  • The object of a parsed novel holds 10.9 MB instead of 17.6 MB.
• The anaphoric el mismo. The highlighting layer anaphoric_mismo of the group "Officialese" marks se archivó el mismo, la duración del mismo and contra la misma cabe recurso, which the Diccionario panhispánico de dudas takes for unnecessary.
  • The identity, the emphasis and the comparison are not marked: es el mismo, el mismo día, él mismo, el mismo que el año pasado.
  • The functions are is_anaphoric_mismo(token) and find_anaphoric_mismo(doc) of ests.visualizers.highlight. The layer needs a Doc with the dependency parse, the parts of speech and the lemmas.
• Components. The factories require the annotations their classes read: ests_syntax the parse, the parts of speech and the lemmas, ests_morph and ests_cohesion the parts of speech and the lemmas, ests_lexical the parts of speech. So nlp.analyze_pipes() names a pipeline that lacks them.
• Documentation. The cards of the new metrics and functions, the page of the formulas of other languages, the stylometric plots on twelve works of Galdós, Pardo Bazán and Unamuno, and the cohesion and the syntax on the page of the comparison of corpora. The notebooks are run again on 0.9.

Incompatible changes

The public classes and functions keep their names. Their signatures change only by new optional last parameters: keyness(n=), collocations(side=), CohesionStats(blank_lines=), find_connectors(blocked=), find_connector_highlights(doc=), calc_mamtld(step=).

• Dependencies. anyTS 0.4.5 or newer within 0.4 is required.
• New keys. get_stats, to_dataframe and print_stats of SyntaxStats, CohesionStats, DiversityStats and BasicStats give the new metrics, so the tables of print_stats are longer.
• Clauses. subordinate_clauses_per_sent and p_complex_sents fall and infinitive_clauses_per_sent grows on texts with infinitives under advcl, ccomp or csubj. A coordinated clause with a subordinating conjunction is subordinate now.
• Connectors. The non-conditional si is no connector, so connectors, connectors_intra, connectors_conditional and connectors_primary are lower.
• Paragraphs. CohesionStats starts a paragraph after every line break. Pass blank_lines=True for the paragraphs between blank lines. n_paragraphs counts the paragraphs with content words.
• Features of the corpora.
  • There are 163 features instead of 132 (diversity_burstiness and the group cohesion_), and 209 with a parser.
  • The Holm correction runs over more features, so p_holm changes.
  • The bootstrap intervals take a generator of their own for every feature.
• A Doc without the parts of speech. SyntaxStats needs them now.
  • A Doc of a pipeline without its morphologizer still has SPACE on its line breaks. LexicalStats, StyleStats.verbal_nouns, SyntaxStats and CohesionStats raise SourceError for it, where they used to accept it without the parts of speech.
  • function_words_profile tags such a Doc with the model.
• Messages. DiversityStats, MorphStats, LexicalStats and VerseStats raise SourceTypeError with the text of the core: The data source cannot be of type int. A Span gets the hint pass span.as_doc().
• MorphStats.
  • print_stats joins the descriptions of a value of several values with a slash (Interrogative / Relative).
  • print_markers prints the table of the core.
  • get_markers counts a feature of several values (Mood=Ind,Sub) in the share of each value.
• collocations sorts the pairs of an equal score by frequency.
• Pickles. A SyntaxStats, CohesionStats or MorphStats pickled by 0.8 has to be computed again.

Numbers

On a fixed set of texts, 633 of 4620 results changed against 0.8.0: 563 only gain keys, and 66 results are new. The 70 whose old values move are all in these categories. Everything else is bitwise equal.

• Clauses: SyntaxStats and ests_syntax, 38 results. On 100 excerpts of the prose of SpanishLiterature (441 thousand tokens):
  • the subordinate clauses fall from 22,440 to 19,392;
  • the infinitive clauses grow from 2,196 to 5,415.
• The non-conditional si: CohesionStats and ests_cohesion, 20 results. The change of `conn...

0.8.0

The eighth release of esTS moves every class of statistics onto the frames of anyTS 0.3, the language-independent core of text statistics. esTS keeps the Spanish layer: the tokenizer and the sentences, the transcription and the syllables, the clause rules, the lists of words and the dictionary of connectors. It also:

• counts the infinitive clauses apart from the subordinate ones;
• counts the connectors by their place in the sentence;
• imports about ten times faster, without loading spaCy;
• gives every class of statistics the same interface: get_stats(flat=True), to_dataframe, describe and print_stats.

Requires Python 3.11 or newer. Install with pip install -U pyests and import with import ests. The dependencies are now anyts>=0.3.9,<0.4 and spacy>=3.8.15,<4, both installed with the package.

New

• The frames of anyTS 0.3. StyleStats, LexicalStats, PhonStats, SyntaxStats and CohesionStats are subclasses of the frames of the core. So are the spaCy components and the finders of the highlighting.
• One interface for all classes. MorphStats and VerseStats are now Stats of the core too.
  • Every class has get_stats(flat=True), to_dataframe(), describe(stat) and the table of print_stats.
  • MorphStats.get_stats gives the counts by keys such as mood_Sub with flat=True.
  • describe gives None for a statistic without a scale.
• Infinitive clauses.
  • The new key infinitive_clauses_per_sent counts the infinitive clauses under a verb (quiere salir). They no longer count as subordinate clauses.
  • is_infinitive_clause_head is new in ests.syntax_stats.
  • is_infinitive_periphrasis also covers the infinitive of a raising verb under xcomp (parece dormir).
• Connectors by place.
  • The new keys connectors_inter and connectors_intra count, per 1000 words, the connectors that link a sentence to the ones before and those that link the parts of a sentence.
  • A subordinator that opens a sentence begins its own clause or phrase, so it is intra-sentential.
  • SpanishConnectorMatcher is public. It carries the Spanish rules as class attributes, with three new lists of ests.constants:
    • CONNECTOR_SUBORDINATORS: 66 conjunctive and prepositional phrases (ya que, a pesar de, siempre que);
    • CONNECTOR_PARCELLATION: porque and conque, which link a sentence they open to the one before;
    • CONNECTOR_SPLIT_BY_MARKS: 51 phrases ending in que, como or si, which a punctuation mark inside splits (No tire así, que se romperá).
  • The dictionary gains 23 anaphoric forms (a pesar de esto, gracias a ello) and now has 273 connectors.
  • find_connectors(..., breaks=) takes the positions of the marks, and find_connector_highlights(..., text=) the text, so the layer of the highlighting follows the same rules as CohesionStats.
• Fast import. import ests takes 0.08 s instead of 0.89 s and loads no spaCy, pandas, matplotlib or SciPy. ests.visualizers and ests.corpus load their modules on first use, and the spaCy components load when they are first used or found by their entry points.
• Clichés of the component. nlp.add_pipe("ests_style", config={"cliches": [...]}) counts the clichés of a list.
• Presets of a subclass. A subclass of ReadabilityStats may narrow the presets, and a preset it adds is read by the scales of general (preset_level_scales, preset_reading_ease_grades).
• Documentation. Every page has metric cards and signature blocks, with the errors of every class and function. The examples are run in English and in Spanish.

Incompatible changes

The public classes and functions keep their names. Their signatures change only by new optional parameters, except the name that a component class built directly needs.

• Dependencies.
  • anyTS 0.3.9 or newer within 0.3 is required.
  • spaCy is bounded below 4: the model es_core_news_sm 3.8.0 bounds it only in its meta.json.
• Names removed from ests.constants. Their values live in anyts.constants:
  • ALLITERATION_MIN_WORD_LEN, ALLITERATION_THRESHOLD;
  • CLAUSE_DEPS, SUBORDINATE_CLAUSE_DEPS, SUBJECT_DEPS, NOUN_MODIFIER_DEPS;
  • PHON_STATS_DESC, PHON_WINDOW_LEN;
  • CONTENT_UD_POS, NAUSEA_TOP_N;
  • FREQUENCY_BANDS, which ests.lexical_stats still re-exports.
• Other names.
  • ests.phon_stats.PHON_STATS_DESC is gone: use PhonStats.stats_desc.
  • ests.cohesion_stats has no ConnectorIndex and split_doc_sents (the core has split_doc_units). Import CONNECTOR_CLASSES and CONNECTOR_TYPES from ests.constants.
  • is_finite_verb and is_infinitive come from anyts.utils, not from ests.syntax_stats. AUXILIARY_DEPS there is now a frozenset.
  • ests.utils.BLANK_LINE is gone.
  • ests.lexical_stats.is_number is the function of the core, is_number(word, pattern). The Spanish pattern is ests.extractors.NUMBER_PATTERN.
• Components.
  • The component classes leave ests.__all__, so from ests import * no longer brings them. ests.StyleStatsComponent and the factories work as before.
  • A component class built directly needs its name: LexicalStatsComponent(nlp, "lexical"). add_pipe is unaffected.
  • The preset of ests_readability is None by default, the preset of the class; a saved config has null.
  • A wrong parameter raises the error of the core at add_pipe. A wrong seseo or data_dir or an unknown key still raises the ConfigValidationError of spaCy.
• Syntax.
  • An infinitive clause under a verb no longer counts in subordinate_clauses_per_sent and p_complex_sents; it has its own key.
  • An infinitive in a gerund or participial phrase belongs to that phrase and heads no clause.
  • An infinitive that the model tags as csubj of its periphrasis (llega a convertirse) heads no clause.
• Connectors.
  • A rule of a connector reads no word across a punctuation mark.
  • A phrase of CONNECTOR_SPLIT_BY_MARKS with a mark inside is no connector.
  • An anaphoric form counts as a connector of its own text in c_connectors.
• Windows. split_windows cuts consecutive windows of exactly window words. The rest is a window only with at least min_words words, a whole window by default; min_words above window raises ParameterError. Before, the windows were equal parts of about window words.
• Messages and tables.
  • The errors have the texts of the core, with hints (The data source has no parts of speech: parse the text with a model…).
  • print_stats prints the table of the core, with the columns fitted to the text, in every class but MorphStats.
  • ests.constants.CORE_MESSAGES is gone.
• Checks.
  • seseo of VerseStats must be a bool.
  • So must the switches of BasicStats, WordsExtractor, CharNgramsExtractor and kwic.
  • The finders of the highlighting, find_phrases, sentence_rhythm, is_parenthetical and noun_lemmas check their arguments. SourceTypeError or ParameterError replaces a bare TypeError or AttributeError.
• Readability attributes. ReadabilityStats.level_scales has the key flesch_reading_easy, and reading_ease_grades is the scale of general. The results of the methods are the same.
• StyleStats.verbal_nouns is a property instead of a cached_property.

Numbers

On a fixed set of texts, 137 of 4620 results changed against 0.7.0, all in these categories. Everything else is bitwise equal.

• New keys. infinitive_clauses_per_sent in every SyntaxStats, and connectors_inter and connectors_intra in every CohesionStats, with their components.
• Clauses. The clause keys of SyntaxStats move in the texts with infinitive clauses: the subordinate clauses only fall. On 1050 fragments of SpanishLiterature:
  • the subordinate clauses fall from 43,989 to 39,875;
  • 3,766 infinitive clauses are counted apart;
  • the clauses go from 89,293 to 89,223. The infinitives in gerund and participial phrases no longer count, while a coordinated nominal predicate (y estaba cansado) and an infinitive under a periphrasis attached as conj now do.
• Connectors. A few texts gain a connector after a comma (sin gracia alguna, y el amor). On the same fragments the connectors go from 36,050 to 36,092: 64 more by the marks, 22 fewer split by a mark. 11.8% of them link a sentence to the ones before.
• Chains of de. A punctuation mark is no link of a chain: one text goes from 11 chains to 10.
• Windows, as above: split_windows, corpus_features, compare_features and compare_corpora.
• Error texts, of the core.

Fixes

• Pickles.
  • SyntaxStats keeps no Doc of the user: a pickle of 30k characters takes 141 KB instead of 885 KB.
  • LexicalStats, StyleStats, SyntaxStats and CohesionStats built with nlp= leave the pipeline out of a pickle and a copy. A LexicalStats took 17 MB.
• Speed. VerseStats is linear in the length of a line: 0.03 s instead of 0.41 s on a line of 8,000 words.
• Extractors. StyleStats and PhonStats of a Doc with a words_extractor read the words with the extractor, as the other classes do.
• expand_phrases gives its keys in one order whatever the hash seed.
• is_agentless of a word that is no passive form gives False.
• Demo.
  • It started again.
  • It cuts a long run of punctuation marks, on which the tokenizer of spaCy hung for minutes.

Quality

• 2073 tests pass with 100% line and branch coverage.
• ruff and mypy are clean.
• CI runs on Python 3.11–3.14, on Linux, Windows and macOS, and with the lowest versions of the dependencies.
• A QA pass before the release covered:
  • the rules of 0.8 on 1050 fragments of SpanishLiterature and 1500 sonnets;
  • 2218 outcomes of the ten classes on 29 sources, and a fuzzing of 17 entry points;
  • the import time, the lowest and the newest dependencies, Python 3....

0.7.0

The seventh release of esTS moves the library onto anyTS 0.2.13, the language-independent core of text statistics. It also:

• reads every readability statistic by a scale;
• starts a sentence after a blank line in a parsed Doc;
• shows the words behind the officialese markers and the academic nausea, which now leaves out the stopwords;
• keeps abbreviations whole in every case.

Requires Python 3.11 or newer. Install with pip install -U pyests and import with import ests. The dependencies are now anyts>=0.2.13,<0.3 and spacy>=3.8.15, both installed with the package.

New

• Readability readings.
  • ReadabilityStats.describe(stat) reads every statistic that has a scale. The reading ease is read by the scale of the preset, INFLESZ for general and Fernández Huerta for classic; level_scale(stat) returns that scale. Legibilidad µ is read by its bands, and LIX by the text types of Björnsson (the English labels of the core).
  • The grades of Crawford, SOL, RIX and the consensus grade are read as a stage of the Spanish school system with its ages.
  • describe gives None for a statistic without a scale or with an undefined value.
  • scale_level, the reading of a value by any scale, is exported from ests.readability_stats.
• Blank lines.
  • The ests_blank_lines component starts a sentence after a blank line, so the parser no longer glues a paragraph to the next sentence.
    • get_nlp puts it before the parser (or the senter or sentencizer).
    • It runs in linear time (0.025 s on 4000 paragraphs), so get_nlp parses as fast as in 0.6.1.
    • It has an entry point, so a saved pipeline loads without importing esTS.
  • SentsExtractor splits at blank lines with a tokenizer of your own too.
  • A Doc with only partial sentence starts, such as one run without the parser, is read by its text in BasicStats, CohesionStats, kwic, sentence_lengths and the highlighting.
• Style.
  • StyleStats.markers() returns the words and phrases each officialese marker counts: the verbal nouns, the compound prepositions, the parenthetical expressions and the clichés, in the order of the text.
  • find_parentheticals(words) returns the bounds of the parenthetical expressions in a list of words, as pairs of start and end.
  • StyleStats.nausea_words() and find_nausea_words return the words of the academic nausea with their occurrences. The nausea_stopwords parameter of StyleStats and of the ests_style component sets its stopwords; it is the last parameter, so positional calls keep working.
• Abbreviations. ests.constants.TOKENIZER_ABBREVIATIONS lists the 46 abbreviations that spaCy keeps whole in some cases only (Sr., pág., etc., a.m.). Every tokenizer of esTS keeps them whole in lower case, capitalized and in upper case, so kwic(text, "sr. pérez") finds Sr. Pérez too.
• Corpus. ests.corpus.check_keyness_params checks the parameters of keyness and is documented.
• Demo. The readability table has a "Reading" column.

Incompatible changes

The public classes and functions keep their names. Their signatures change only by new optional parameters:

• nausea_stopwords of StyleStats and ests_style, and stopwords of calc_academic_nausea, both last;
• the keyword-only messages of calc_windowed and check_params in ests.diversity_stats.

• Dependencies.
  • anyTS 0.2.13 or newer within 0.2 is required.
  • spaCy 3.8.15 or newer is required: spaCy 3.8.2-3.8.14 import click without requiring it, so import ests failed there without it.
• Readability levels.
  • describe_level, flesch_reading_easy_to_level and mu_to_level raise ParameterError for a value that is not a finite number. Before:
    • for nan they returned "";
    • for inf and -inf they returned the top and the bottom band;
    • for a value that is not a number they raised TypeError.
  • describe_level and flesch_reading_easy_to_level raise ParameterError for a scale that is not a known name. A list or a set raised TypeError, and a false scale ("", [], 0) in describe_level fell back to the scale of the preset.
  • describe_grade raises ParameterError for an array of names, where it raised TypeError.
  • describe_level("lix") returns a band instead of raising.
• Academic nausea. It sums the top_n most frequent words without the stopwords, divided by all words. Before, the stopwords were counted, so the top ten of almost every text was de, la, que, el, and none of 300 excerpts of SpanishLiterature was within the norm of Advego (5-15%). On these excerpts the median falls from 29.2% to 6.3%, and 245 of them are within the norm.
• Abbreviations as words. The tokenizer of get_tokenizer and get_nlp keeps the 46 abbreviations of TOKENIZER_ABBREVIATIONS whole in every case, so sr., ud., Pág. and SR. are one word with the period, where 0.6.1 split the period off. The words of the extractors and the frequency counts change with them, and a keyword without the period no longer finds such a word: kwic("Habló el sr. Pérez ayer.", "sr") finds nothing, "sr." finds it.
• Pipeline.
  • get_nlp().pipe_names has ests_blank_lines before the parser, so the sentences of a parsed Doc start after blank lines.
  • Every component of esTS applies the tokenizer rules to the whole pipeline it is added to, abbreviations included.
• LexicalStats.band_coverage raises ParameterError for bounds that are not an iterable (100, None), where it raised TypeError. An iterator is read once.
• Characters without a UTF-8 form. A text with a lone surrogate raises SourceError in every class that parses it, where spaCy raised UnicodeEncodeError.

Numbers

On a fixed set of texts 534 of 4620 results changed against 0.6.1, all in these categories; everything else is bitwise equal.

• Sentences of a Doc. A string or a Doc parsed through get_nlp now has a sentence boundary at every blank line. A paragraph opened by a dash or a quote keeps the mark with its first word.
  • Everything built on the sentences of a Doc moves for the texts with blank lines: syntax, cohesion, and BasicStats and ReadabilityStats of a Doc.
  • SyntaxStats and CohesionStats of a string parse it with get_nlp, so they move with the Doc.
  • BasicStats and ReadabilityStats of a string split it with sentenize, so these changes do not touch them.
• Sentences of a string.
  • A sentence goes on after an abbreviation written without spaces (a.c., p.ej., EE.UU.), as it does after the spaced forms, and after esq. and apdo..
  • A line of Unicode spaces (a no-break space, U+3000) between two line breaks is now blank and ends a sentence.
• Abbreviations.
  • Texts with SR., Pág., ud., ee. uu. or a.m. change their words and their parse.
  • A dash of a dialogue glued to a dotted acronym (S.A.—dijo, »P.D.--El) no longer splits its last period off.
  • LexicalStats looks an abbreviation up without its period, so Sr., SRA., pág. and etc. find sr, sra, página and etcétera in the frequency dictionary. An initial (D., J.) keeps its period and stays out of the dictionary.
• Academic nausea, as described above.
• PhonStats of a text without sounds. Every statistic is nan; the shares were 0 while the ratios were nan. p_heavy_clusters is nan for a text without consonants.
• zipf_naturalness is computed in integers, so only its last bits change: frequencies 11, 8, 8, 2, 2 give 50.0 instead of 49.999999999999986, just under the norm.
• n_spaces and p_spaces count every Unicode space (Zs), so they rise in texts with no-break spaces.

Fixes

• SyntaxStats follows a long chain of coordinated clauses without recursion: 600 of them raised RecursionError.
• dispersion_plot and frequency_spectrum_plot accept a numpy array or a pandas Series of words, and fingerprinting and wordtree one of texts, instead of failing on their truth value; an empty one raises SourceError (anyTS 0.2.10).
• ests.utils.count_words_by_spans reads a pandas Series by position, not by its index labels, and returns Python integers.
• A component of esTS added to a pipeline that has its tokenizer rules already does not reload the tokenizer. Loading es_core_news_sm with the ten components takes about half the time of 0.6.1.
• The demo shows an infinite value as a dash.

Quality

• 1995 tests pass with 100% line and branch coverage, which is now required.

• ruff and mypy are clean.

• A QA pass before the release covered:

  • a fuzzing of 2.6 million calls on 3504 inputs;
  • the lowest versions of the dependencies and Python 3.14;
  • the notebooks and the demo;
  • the documentation in English and Spanish, with its 132 examples run on each language.

  Its findings are fixed in anyTS 0.2.13 and in this release.

Links

• Documentation: https://sergeyshk.github.io/esTS/
• Notebooks: https://github.com/SergeyShk/esTS/tree/master/examples
• Demo: https://huggingface.co/spaces/SergeyShk/esTS
• PyPI: https://pypi.org/project/pyests/0.7.0/
• anyTS: https://github.com/SergeyShk/anyTS

0.6.1

A patch release of esTS: the library moves to anyTS 0.2.8, the language-independent core of text statistics shared with ruTS, and fixes two gaps found on the way.

Requires Python 3.11 or newer. Install with pip install -U pyests and import with import ests. The core is now anyts>=0.2.8,<0.3, installed with the package.

Incompatible changes

The public classes and functions keep their names and signatures.

• anyTS 0.2.8 or newer within 0.2 is required.
• ests.utils.count_words_by_spans is the function of the core (anyts.utils.count_words_by_spans), with the same results. It also checks its arguments: spans that are not a sequence (an iterator, a generator, dict_keys, a 2-D array) raise SourceTypeError where 0.6.0 looped over them, and positions that are not a sequence raise SourceTypeError instead of a bare TypeError.
• keyness with a FreqDict checks its parameters and target before it reads the dictionary. A wrong measure, top_n or min_freq raises ParameterError, and a target that is not a list of words or a mapping of frequencies raises SourceTypeError, at once and also when the dictionary is not downloaded, where they used to raise DatasetNotFoundError.

Numbers

The numbers of 0.6.0 are kept except for D₂ of Carroll in dispersion, which the core computes without rounding noise: on a fixed set of texts 2 of 4620 results changed, eight values of carroll_d2: four in the last digit, one from 4.5e-17 to 0.0 and three from -0.0 to 0.0. χ² and BIC of keyness give no -0.0 either.

Fixes

• subtree_len, and with it the participle and gerund clauses of SyntaxStats, returns on a broken parse whose heads form a loop instead of never returning; on a tree the counts do not change.
• The participle and gerund layers of the highlighting return on such a parse as well, and count the words of a clause as SyntaxStats does. On 40 texts of SpanishLiterature all fragments are the same as in 0.6.0.

Quality

• 1812 tests pass with 100% line and branch coverage.
• ruff and mypy are clean.
• The numbers of all statistics were compared with 0.6.0 on a fixed set of texts; everything outside the change above is bitwise equal.

Links

• Documentation: https://sergeyshk.github.io/esTS/
• Notebooks: https://github.com/SergeyShk/esTS/tree/master/examples
• Demo: https://huggingface.co/spaces/SergeyShk/esTS
• PyPI: https://pypi.org/project/pyests/0.6.1/
• anyTS: https://github.com/SergeyShk/anyTS

0.6.0

The sixth release of esTS moves the library onto the second slice of anyTS, the language-independent core of text statistics. It also:

• revisits how sentences, clauses and connectors are counted;
• computes alliteration and assonance against the words in random order;
• adds a gallery of notebooks.

Requires Python 3.11 or newer. Install with pip install pyests and import with import ests. The dependencies are now anyts>=0.2.3,<0.3 and spacy>=3.8, both installed with the package.

Incompatible changes

The public classes and functions keep their names, and their signatures change only where noted below.

• Dependencies. spaCy 3.8 or newer is required: on spaCy 3.7 import ests fails without click, which typer no longer installs. The model es_core_news_sm 3.8 and the demo were on 3.8 already. anyTS 0.2.3 or newer within 0.2 is also required.
• calc_smog_index(n_complex, n_sents, a, b, c). b is now the number of sentences of the sample (30), and the constant is c. A positional b changes meaning: calc_smog_index(10, 30, 1.043, 3.1291) gave 6.427 and now gives 4.194.
• Names that left esTS. They live in the core; import them from anyts.
  • ests.constants no longer has COMPLEX_SYL_FACTOR, LONG_WORD_LETTER_FACTOR, SPACES, PUNCTUATION_TYPES, BASIC_STATS_DESC, SMOG_COMPLEX_SYL_FACTOR and LIX_LONG_WORD_LETTER_FACTOR (they are in anyts.constants), nor PUNCTUATIONS, HIGHLIGHT_SYNTAX_LAYERS and HIGHLIGHT_TAGGED_LAYERS (replaced by HIGHLIGHT_LAYER_ANNOTATIONS).
  • ests.utils no longer has to_path, download_file, extract_archive, sha256 (in anyts.datasets), DOWNLOAD_TIMEOUT or TAR_DATA_FILTER. The module ests.datasets.dataset is gone.
  • Undocumented names are gone:
    • DASH_PATTERN, ELLIPSIS_PATTERN and PUNCTUATION_CHARS of ests.basic_stats;
    • WordTree, TreeDrawer, FreqNode and Direction of ests.visualizers.word_tree;
    • MARGIN, MISSING_COLOR and SQUARE of ests.visualizers.fingerprinting;
    • LINE_BREAK, CSS, select_layers, get_text_words, get_text_sents, get_doc_words, get_doc_sents and split_segments of the highlighting;
    • SOUNDS, SOUNDS_ORDER, CONSONANT_COLUMNS and the private helpers of the repetition index in ests.phon_stats.
  • Word, Sent and Highlight are the classes of the core.
• Stricter checks. These raise ParameterError or SourceTypeError instead of being accepted or failing with a bare Python error:
  • Readability.
    • calc_consensus_grade checks the preset even without a reading ease.
    • A grade or a reading ease that is not a finite number raises ParameterError when it is converted or averaged: flesch_reading_easy_to_grade, ReadabilityStats.reading_ease_to_grade, calc_consensus_grade and grade_to_age. flesch_reading_easy_to_grade(nan) gave 13.
    • A reading speed that is not a positive number also raises: calc_reading_time(10, nan) gave nan.
    • ReadabilityStats checks the extractors given with ready basic statistics and refuses basic statistics of another class.
  • Types of words and features.
    • A word that is not a string in the syllables (syllabify, count_syllables, word_stress, word_stresses, stress_type), in transcribe and in is_stopword.
    • Words or parts of speech of find_connectors that are not a list: generators are refused now. Parts of speech must be as many as the words.
    • Features of compare_corpora and corpus_features that are not a function: features=0 no longer falls back to the default.
  • Plots.
    • The axes of every plot (ax) must be a matplotlib Axes.
    • The keywords of keyness_plot, the collocations of collocation_network and the concordances of format_kwic are checked; iterators are still accepted.
    • highlight refuses a bool or nan as alliteration_threshold, and sentence_lengths refuses sets, mappings and negative lengths.
    • A component name taken by an extension of another package raises ParameterError.
• The formulas of counts return nan when the number of words or sentences they divide by is zero, instead of raising ZeroDivisionError. This covers Flesch, Flesch-Kincaid, Coleman-Liau, ARI, SMOG, fog, LIX, RIX, Gutiérrez de Polini, Crawford and SOL. On other inputs the values are unchanged.
• kwic compares the word forms in NFC without soft hyphens, so a decomposed accent matches. A phrase no longer runs across the end of a sentence (by the Spanish rules) or of a paragraph.
• expand_phrases: a phrase written as such wins over the forms of another phrase (a efectos del next to a efectos de). The built-in lists do not change.
• fingerprinting makes every segment segment_len words long (anyTS 0.2).

Numbers

The numbers of 0.5.0 are kept except for the changes below. On a fixed set of texts 156 of 4620 results changed, all in these categories.

• Sentences.
  • A dash glued to the terminal mark, as the editions of Project Gutenberg type it, now ends the sentence: baja.--Tiene opens a new line of dialogue.
  • A glued dash before whitespace closes its line of dialogue and stays with it: ¡Traidores!--, viuda.-- El niño.
  • A remark after a spaced dash no longer splits the sentence: Sí. --dijo él.
  • Abbreviations, initials and list markers still end nothing.
  • The corpus of literature has 634,712 sentences instead of 624,200 (+1.7%).
  • n_sents and the readability of a string move with the sentence count. A parsed Doc keeps the sentences of the parser.
• Clauses (SyntaxStats). On 450 windows of the corpus of literature, clauses_per_sent rises by 11.7%, subordinate_clauses_per_sent by 21.2% and p_complex_sents by 8.6%, while mean_clause_len falls by 10.2%. The models es_core_news_lg and es_dep_news_trf shift by about as much. The rules behind this:
  • An infinitive under xcomp heads a subordinate clause (quiere salir), unless it makes a periphrasis with its verb (suele salir, vuelve a salir: INFINITIVE_PERIPHRASES, NGLE ch. 28) or the verb is parecer.
  • A periphrasis whose infinitive the models attach as conj (tiene que salir) is one predicate.
  • The form of a predicate is read from its first auxiliary or copula (predicate_form):
    • compound tenses, passives and progressives head their clauses (había llegado, estaba cantando), and so do finite predicates under acl and xcomp;
    • an infinitive form under acl is no clause (el miedo de ser robado);
    • compound and copular gerunds are gerund clauses (habiendo llegado tarde, siendo elegido presidente).
• Connectors (CohesionStats).
  • The dictionary has 250 markers. apenas, nada más, lo mismo que, esto es and es más are removed. después de todo is now reformulative; en primer/segundo/tercer/último lugar and primeramente are now additive.
  • Rules of place:
    • pues opening a sentence is no connector;
    • encima counts only at the start of a sentence or after y, e, ni or pero;
    • en cuanto a and algo así como count nothing.
  • On the 450 windows, connectors per 1000 words fall by 2.3%: causal −16.1%, reformulative −10.1%, temporal −5.9%.
  • The documentation now credits the markers correctly (Martín Zorraquino & Portolés and the NGLE, grouped in the seven classes of the library), with the corrected Coh-Metrix-Esp link.
• Alliteration and assonance (PhonStats, calc_alliteration, calc_assonance).
  • The expected number of repetitions is the exact one for the words in random order (hypergeometric), through anyts.phonetics.calc_repetition_index.
  • Before, it was binomial and read below 1 on random text: 0.966 on shuffled sonnets, now 0.996.
  • Books change in the third digit.
  • A text where no sound is shared by two words gives nan instead of 0.0.

Notebooks

A gallery of four notebooks in examples/ shows the library on its datasets, and a Colab button opens each:

• a short story of Quiroga through every tool;
• ten prose authors compared by stylometry;
• the Spanish sonnet across four centuries;
• the corpus measures, Spain against Spanish America.

The outputs are kept, and a weekly workflow runs the notebooks and fails when a number their text quotes has moved.

Fixes

• Sentence splitting is linear on long runs of spaces after a dash and on runs of terminal marks, which took seconds.
• Highlighting a Doc is linear again: 400,000 characters take 0.16 s instead of 42 s (anyTS 0.2.3).
• zipf(show_theory=True) accepts an integer alpha.
• SyntaxStats finds the clause heads once, so it is about as fast as in 0.5.0 despite the new rules.
• The documentation examples, the rules on the pages, the Spanish of the translations and the notes on the models (the recall of the imperative per model) are corrected.

Quality

• 1811 tests pass with 100% line and branch coverage.
• ruff and mypy are clean.
• The notebooks are checked weekly.
• The numbers of all statistics were compared with 0.5.0 on a fixed set of texts; everything outside the changes above is bitwise equal.

Links

• Documentation: https://sergeyshk.github.io/esTS/
• Notebooks: https://github.com/SergeyShk/esTS/tree/master/examples
• Demo: https://huggingface.co/spaces/SergeyShk/esTS
• PyPI: https://pypi.org/project/pyests/0.6.0/
• anyTS: https://github.com/SergeyShk/anyTS

0.5.0

The fifth release of esTS moves the library onto anyTS, the language-independent core of text statistics, and adds a demo on Hugging Face Spaces.

Requires Python 3.11 or newer. Install with pip install pyests and import with import ests. The new dependency is anyts>=0.1,<0.2, installed with the package.

Incompatible changes

The numbers of 0.4.0 are kept, except for the changes below; the public classes and functions keep their names and signatures.

• Names that left esTS. They live in the core; import them from anyts.
  • ests.constants no longer has the defaults of the diversity metrics (MATTR_WINDOW_LEN, MTLD_*, HDD_SAMPLE_SIZE, DIVERSITY_LOG_BASE, BRUNET_W_EXPONENT, DIVERSITY_STATS_DESC) nor VALENCY_IGNORED_DEPS and the names of the corpus measures (KEYNESS_MEASURES, G2_CRITICAL_VALUES, COLLOCATION_MEASURES, DISPERSION_STATS_DESC, DELTA_VARIANTS): they are in anyts.constants.
  • Undocumented helpers are gone from the esTS modules: is_punctuation, count_letters, safe_divide, has_words, check_sequence, iter_doc_tokens of ests.utils; Overlap, calc_overlap, calc_proportional_overlap, dice, dominant of ests.cohesion_stats; Extractor, Tokenizer of ests.extractors; Calculator of ests.diversity_stats; MEASURES, ZERO_ADJUSTMENT, ZERO_SEGMENTS, Values, compare_values of the corpus modules; the private helpers of the moved code.
  • No longer re-exported, though still importable at runtime (type checking with --no-implicit-reexport fails on them): iter_doc_words of ests.utils; calc_overlaps, count_given, calc_repetition of ests.cohesion_stats; the twelve tree helpers of ests.syntax_stats (base_dep, get_words, calc_tree_depth, ...).
• Exceptions are the classes of the core under the same names, and EstsError is AnyTSError: except EstsError also catches the errors raised by the code of the core, and tracebacks show anyts.exceptions.SourceTypeError.
• Numbers.
  • Coordination chains are the connected components over conj: coordination_chains_per_sent and mean_coordination_chain_len change for texts with nested coordination.
  • The last digits of content_overlap_prop_all, summed with math.fsum, no longer depend on the hash seed.
  • DTTR of a text or a window without repetitions is inf (was 0); STTR of a text of one lexeme or of no more words than the logarithm base is nan; zipf_alpha of a text without repetitions is 0.0 (was -0.0).
  • ELL of rare words is a number; it is nan only when the least expected frequency is at most 1 (was below e).
  • Tied most frequent words of frequency_table and delta are ordered by their means rounded to 12 digits, then alphabetically: the selection at its border and Eder's Delta, weighted by rank, can change.
  • Negative keywords against FreqDict no longer include the dictionary keys the target never counts (cd-rom, r.p.m.).
  • A byte order mark at the start of a text is split off: the words are those of the text without it. In a parsed Doc it stays a token of its own, so the parse of the first sentence can still differ.
  • is_punctuation also takes combining marks and invisible format characters (Unicode M and Cf): a lone zero-width space is no word.
  • The sentences of a custom tokenizer of SentsExtractor are stripped, and WordsExtractor with a custom tokenizer drops the tokens made only of whitespace.
• Stricter checks. Parameters and sources of the wrong type or value raise ParameterError or SourceTypeError instead of being accepted or failing with a bare TypeError, ValueError or AttributeError:
  • a float or a bool for a count, window, width, number of items, limit, length, factor or number of bootstrap samples; a negative length bound;
  • a string, an iterator or a list of non-strings (spaCy tokens) for a list of words; a non-string text or keyword;
  • an extractor, a pipeline or a frequency dictionary of another type; a non-string preset or dataset filter;
  • calc_zipf_naturalness(words, 0) and the other module functions check top_n and window_len as their classes do; sentence_rhythm takes integer lengths only.
  • The classes check their parameters before the source, so a wrong parameter is reported before an empty text and before any parsing.

The anyTS core

• Exceptions, utilities, the extractors, the lexical diversity metrics, the cohesion and tree helpers and the corpus measures (collocations, dispersion, keyness, stylometry, the statistics of the comparison) come from anyts 0.1.
• esTS keeps the Spanish layer:
  • the extractors are subclasses of those of the core with the Spanish hooks: the sentence splitter, the spaCy tokenizer, simplemma and the pattern of Spanish numbers;
  • DiversityStats takes a string or a Doc, as before;
  • keyness also takes the frequency dictionary FreqDict; the reference by frequencies of the core, FrequencyReference, is in ests.corpus;
  • function_words_profile, compare_corpora and the features of a text stay in esTS, like all the Spanish statistics.

Documentation

• The English pages of the moved parts include the text of the anyTS pages, so every formula and parameter is described in one place; the Spanish pages translate it, and a test fails when a section of the core changes until its translation is updated.
• The pages of the extractors describe the Spanish hooks; the comparison page describes the functions of its statistics.

Demo

• https://huggingface.co/spaces/SergeyShk/esTS: paste a Spanish text and get its readability with the school stage, every class of statistics, the keywords and collocations against the frequency dictionary, the plots of Zipf's law and of sentence lengths, and the highlighting.
• The code is in demo/: make demo runs it locally, and every release updates the Space.

Quality

• 1785 tests with 100% line and branch coverage.
• ruff and mypy are clean.
• The numbers of all statistics were compared with 0.4.0 on a fixed set of texts: everything else is bitwise equal.

Links

• Documentation: https://sergeyshk.github.io/esTS/
• Demo: https://huggingface.co/spaces/SergeyShk/esTS
• PyPI: https://pypi.org/project/pyests/0.5.0/
• anyTS: https://github.com/SergeyShk/anyTS

0.4.0

The fourth release of esTS adds style and sound: style metrics, phonostatistics, the scansion and rhyme of Spanish verse, highlighting of a text, and a dataset of Spanish sonnets.

Requires Python 3.11 or newer. Install with pip install pyests and import with import ests. The new statistics need no spaCy model. The one exception is the verbal nouns of the style metrics, which, like the syntactic layers of the highlighting, need a parsed text.

Incompatible changes

The API of 0.3 is unchanged. Two changes affect the numbers:

• Syllabification of the diaeresis.
  • A ü outside gü and qü, and any ï, now mark a hiatus: sü-a-ve, crü-el, glo-rï-o-sa.
  • ï no longer takes the stress.
  • HIATUS_VOWELS and ACCENTED_VOWELS change to match.
  • count_syllables, word_stress and every measure built on them move for such words: the readability formulas, n_complex_words, the syllables of verse. In the 150 texts of the corpus this touches 131 occurrences of 72 word forms.
• Negative zero. entropy, heaps_beta and cv_entropy of a degenerate text (a single lexeme, a single CV pattern) return 0.0 instead of -0.0.

No new dependencies.

Style metrics

• StyleStats computes nine metrics.
  • The SEO indicators, as in the Advego and Text.ru services: classic and academic nausea, water, spam and Zipf naturalness, plus the keyword_density method.
  • The markers of the officialese style that Spanish plain-language guides warn about: verbal nouns, compound prepositions, parenthetical expressions and clichés. The lists behind them hold 42 compound prepositions, 55 parenthetical expressions and 74 clichés.
  • A cliché that starts with an infinitive also matches the forms of the verb: proceder a finds se procedió a.
• The component is ests_style.

Phonostatistics

• PhonStats computes 13 statistics on a rule-based transcription (transcribe, the standard of Spain with yeísmo and distinción):
  • the shares of vowels, sonorants, voiced and voiceless sounds;
  • the consonant-vowel ratio;
  • consonant clusters;
  • hiatus;
  • the entropy of the CV patterns;
  • hardness;
  • the alliteration and assonance indices;
  • open syllables and the mean syllable length.
• The component is ests_phon.

Verse

• VerseStats scans a poem in metrical syllables.
  • It counts synalepha and applies the law of the final stress.
  • Each line is fitted to the meter by hiatus, dieresis and synaeresis.
  • Compound verse is read by its hemistichs.
• From the scansion it gives:
  • the meter;
  • the rhythmic stresses and the types of the endecasílabo;
  • the endings of the lines;
  • the rhyme, consonante and asonante;
  • the rhyme schemes in the Spanish notation (ABBA ABBA CDC DCD, -a-a);
  • 19 strophes, from the pareado to the décima;
  • the form of the poem: soneto, romance, silva, or the strophe shared by all stanzas.
• seseo=True reads the rhyme with the seseo of Andalusia and America.
• Accuracy on the sonnets of DISCO:
  • the length of a line agrees with DISCO in 97.0% of the lines and with rantanplan in 97.7%;
  • 99.1% of the rhyming pairs found are pairs of RhymeTagger, and 97.9% of its pairs are found.
• The helper functions are detect_meter, rhyme_scheme, split_stanzas and accentuate. The component is ests_verse.

Highlighting

• highlight marks the fragments behind the statistics in 15 layers of five groups:
  • readability: long sentences, complex and rare words;
  • syntax: passive, participial and gerund clauses, chains of de, split predicates;
  • officialese: verbal nouns, compound prepositions, clichés;
  • style: stopwords, parenthetical expressions, connectors;
  • phonics: alliteration.
• HighlightedText.to_html() gives the markup with a legend, and a notebook shows it directly.

Dataset

• SpanishSonnets holds 4,259 sonnets by 1,167 authors from the 15th to the 20th century.
  • They come from DISCO 5.0 (CC BY 4.0), with its automatic scansion and rhyme labels on every line.
  • Only the public-domain sonnets are kept.
  • Corrections to DISCO's metadata are listed in the archive.
• The records can be filtered by period, author, country and gender.

Quality

• 1876 tests with 100% line and branch coverage.
• ruff and mypy are clean.
• CI runs on Python 3.11 to 3.14 and on Linux, Windows and macOS.
• The release went through its own QA pass, and all its findings are fixed.
• The docstrings and the bilingual documentation were trimmed down to what a reader needs.

Documentation: https://sergeyshk.github.io/esTS/ · Commits: 0.3.0...0.4.0

0.3.0

The third release of esTS moves from a single text to corpora. It adds the measures of corpus linguistics and stylometry, visualizations, two datasets (a corpus of Spanish-language literature and a frequency dictionary), and lexical sophistication statistics based on that dictionary.

Requires Python 3.11 or newer. The distribution on PyPI is pyests and the package it installs is ests: pip install pyests, import ests. The spaCy model es_core_news_sm is needed by the morphological, syntactic, cohesion and lexical sophistication statistics of a string, by the function-word profile and by the comparison of corpora. The datasets are downloaded once with their download() method.

Incompatible changes

The API of 0.2 is unchanged, and all new functionality is additive. Three things may still affect existing code:

• Dialogue dashes. The tokenizer now splits off a dialogue dash stuck to a word, whether typed as hyphens in plain text (--No, -dijo, sí--dijo) or as a dash between words (reírse—me). The string API and the components share these rules. Numbers therefore change on texts with dialogue. On 13 novels and plays of the corpus, compared with 0.2.0:

  • n_words changes by up to 1% (median 0.2%);
  • n_unique_words changes by up to 7.7% (median 3.9%), because a word with a dash stuck to it used to count as a separate word;
  • dialogue hyphens in c_punctuations move from hyphen to dash;
  • n_sents does not change.

  Every readability and diversity measure built on these counts moves with them.

• New required dependencies: pandas, matplotlib and graphviz. The word tree and the network of collocations are rendered by the executables of Graphviz, which are installed separately.

• simplemma: the minimum goes from 1.0 to 2.0, with a ceiling below 3. The keys of the frequency dictionary are the lemmas of simplemma 2.0.0, and FreqDict warns once if another version is installed.

Corpus measures

• keyness compares a target corpus with a reference: a list of words, a table of frequencies, or the frequency dictionary. It offers seven measures: log-likelihood G², chi-square, %DIFF, Log Ratio, BIC, ELL and the odds ratio. Every keyword also carries G², Log Ratio and the p-value of G². Against the dictionary, the words of the target are looked up by the keys the dictionary was built with. A word missing from the dictionary gets the dictionary's lowest frequency rather than zero, so it can only be a keyword of the target.
• collocations finds ordered pairs within a window, with eight association measures: MI, MI³, t-score, Dice, logDice, log-likelihood, NPMI and minimum sensitivity. It can also return the collocations of a single word.
• dispersion measures how a word spreads over the parts of a text, with Gries's DP and its normalized form, Juilland's D, Carroll's D2, Rosengren's S and the Kullback-Leibler divergence.
• kwic builds a concordance. format_kwic and print_kwic align it on the keyword.
• Stylometry: the frequency table and z-scores of the most frequent words; Delta in four variants (Burrows, Argamon's quadratic, Eder, cosine), both between texts and from texts to reference texts for attribution; Zeta after Burrows and Craig, with its logarithmic form; Kilgarriff's chi-square; Mendenhall's curves and the distance between them; and the profile of function words by part of speech.
• Comparison of corpora: texts are cut into windows and 132 features are computed for every window. compare_corpora tests every feature and reports effect sizes (Cohen's d, Cliff's delta), a bootstrap interval of the difference of medians and Holm's correction.

Visualizations

• ests.visualizers returns matplotlib Axes or a Graphviz graph:
  • Zipf's law with a fit of Zipf-Mandelbrot;
  • literature fingerprinting;
  • a word tree;
  • dispersion, keyness and a network of collocations;
  • a dendrogram, PCA and MDS of stylometric distances, and Mendenhall's curves;
  • Heaps' law and the frequency spectrum;
  • sentence lengths.

Datasets

• SpanishLiterature holds 150 public-domain works by 33 authors from Spain, Latin America and the Philippines, from Cervantes to the 1920s: about 11 million words in four genres (prose, poems, drama, publicism). Works can be filtered by author, genre, country, years and length.
• FreqDict is a frequency dictionary of 83,785 Spanish lemmas from the Spanish books of Google Books Ngram for 1980-2019 (63 billion words). Each lemma has its frequency per million words, range, Juilland's D and number of books. It is distributed under CC BY 3.0.
• Both are downloaded once and verified against their SHA-256 checksum. They are extracted atomically, so an interrupted extraction never leaves half a dataset behind. They are stored in ests_data next to the package, or in the directory set by the environment variable ESTS_DATA_DIR or by data_dir.

Lexical sophistication

• LexicalStats, in the manner of TAALES, measures how rare the words of a text are:
  • the coverage of the dictionary;
  • the mean frequency and log frequency of the lemmas, over all words and over content words, and their mean range and dispersion;
  • the shares of the frequency bands top-1000 to top-10000;
  • surprisal and perplexity under the dictionary's unigram model;
  • lexical density.
• The component ests_lexical is the seventh factory of the spaCy pipeline.

Quality

• 1496 tests at 100% coverage of lines and branches; ruff and mypy are clean. CI runs on Python 3.11 to 3.14 and on Linux, Windows and macOS. The network tests run in a job of their own after a merge.
• The release went through a QA pass of its own, and all 22 of its findings are fixed. The documentation is bilingual, and its examples take their texts from the corpus of literature.

Documentation: https://sergeyshk.github.io/esTS/ · Commits: 0.2.0...0.3.0

0.2.0

The second release of esTS adds the statistics that read the annotation of Universal Dependencies - morphology, syntax and cohesion - and puts every class of the library on a spaCy pipeline as a component.

Requires Python 3.11 or newer. The distribution on PyPI is pyests, the package it installs is ests: pip install pyests, import ests. The statistics of 0.1 still need no trained model; the three new ones do, and python -m spacy download es_core_news_sm is what they ask for.

No incompatible changes. The API of 0.1 is untouched and everything here is an addition: the six classes it shipped behave as they did, and the helpers that grew (iter_doc_tokens, get_nlp, has_words, is_verbal_noun) are new names beside the old ones.

Morphology

• MorphStats - parts of speech and fifteen grammatical features of Universal Dependencies as the Spanish models annotate them: case, definiteness, degree, gender, mood, numeral type, number, person, polarity, politeness, possessive, pronoun type, reflexive, tense and verb form, with the whole annotation kept in tags in the CoNLL-U form. get_stats counts them, explain_text parses a text word by word, print_stats prints them with their descriptions.
• get_markers and print_markers - the markers of Spanish as shares of their own base, so texts of different lengths compare directly: the moods among the finite forms, the non-finite forms among the verb forms, ser among the two copulas, the adverbs in -mente among the adverbs. The base of the verb forms is the verbs and the auxiliaries, so a participle the model reads as an adjective stays out of it, and the copula of a passive - fue escrito, es financiado - is no choice between the copulas.
• What the models can and cannot do is measured and written down: es_core_news_sm, es_core_news_md and es_core_news_lg find the imperative in 2, 3 and 4 of 24 imperative sentences, so p_imperative under-reports whichever one is used, and the enclitics (dámelo, vámonos) are mis-analysed by all three.

Syntax

• SyntaxStats - 30 statistics on the dependency tree: the dependency distances of Liu (2008) with their mean, deviation and share of adjacent links, the depth of the tree, leaves and subtrees, the valency of the finite verbs, the coordination chains, the clauses and the subordinate clauses, the modifiers of a noun and the ratio of nouns to verbs, in the terms of Ivanov, Solnyshkina and Solovyev (2018).
• The constructions the Spanish guides to clear language warn about: the passive with ser and with se and the agentless passive, the participial and the gerund clauses with their lengths, the chains of de (el aumento de la eficiencia del uso de los recursos), the impersonal se, the infinitives, the words of negation and the split predicates (hacer una revisión instead of revisar), which split_predicates lists as it finds them.
• Every rule is written against what the models actually annotate: they use almost no subtypes of the relations, so a passive is recognized by the lemma of its auxiliary and an agent by its preposition; a participle that modifies a noun is an adjective for them; the periphrases with a gerund hang from their verb as xcomp or advcl.

Cohesion

• CohesionStats - referential cohesion in the manner of Coh-Metrix and of Coh-Metrix-Esp: the overlap of nouns, of arguments and of content words between adjacent sentences and between all pairs of sentences, binary and proportional, the share of pronouns and of demonstratives, p_given, and temporal cohesion, where the mood takes the place of the aspect Spanish has no feature for.
• 255 discourse markers in the classification of Martín Zorraquino and Portolés, shipped with the package: causal, adversative, concessive, temporal, additive, conditional and reformulative, each primary or secondary, with the density per 1000 words and the occurrences in connector_spans. A marker that heads a prepositional phrase (antes de la reunión, sobre todo el texto) is not counted, and a one-word marker is checked against its part of speech.
• The overlaps are computed over bit masks and histograms instead of every pair of sentences, so the time grows with the number of occurrences and not with the square of the number of sentences.

Components of a spaCy pipeline

• Six factories - ests_basic, ests_readability, ests_diversity, ests_morph, ests_syntax, ests_cohesion - each putting its object of statistics into doc._.<name>, so a text is annotated and measured in one pass. The name of the pipe is free and is what the extension is called: nlp.add_pipe("ests_basic", name="basic") gives doc._.basic.
• They are entry points of spacy_factories, so a pipeline saved with them loads with spacy.load(path) in a process that never imports the library. A document with no words passes through untouched, so one empty text does not stop nlp.pipe over a corpus.
• ests_readability takes its preset and ests_diversity its five parameters, validated when the component is added; ests_readability can also take the basic statistics of another component instead of computing them a second time.
• The objects they attach are not serializable: Doc.to_bytes(), DocBin(store_user_data=True) and nlp.pipe(n_process > 1) need the user data left out, or get_stats() kept apart.

Quality

• 1051 tests, coverage 100 % by lines and branches, ruff and mypy clean; the CI runs on Python 3.11 to 3.14 and on Linux, Windows and macOS.
• The release went through a QA pass of its own, and the ten findings of it are fixed or decided and written down; the documentation is bilingual, and every example in it is executed and compared with its printed output.

Documentation: https://sergeyshk.github.io/esTS/ · Commits: 0.1.0...0.2.0

0.1.0

The first release of esTS, a library for statistics extraction from texts in Spanish: object extraction, syllabification and stress, basic statistics, readability and lexical diversity - by published formulas with the coefficients and the scales of their authors, deterministically and without a neural network inside.

Requires Python 3.11 or newer. The distribution on PyPI is pyests, the package it installs is ests: pip install pyests, import ests. Dependencies: numpy, scipy, simplemma, spacy; no trained spaCy model is needed, one is required only to parse a text yourself and pass the Doc instead of a string.

Object extraction

• SentsExtractor - sentence splitting by rules that know Spanish: the inverted marks ¿ and ¡, quotes and brackets of three levels, the dash of a line of dialogue with the remark of the narrator (—¿Vienes? —preguntó María is one sentence), abbreviations (Sr., Dra., p. ej., EE. UU., a. m.), capital initials, list markers, a lower-case continuation after an ellipsis, a blank line as a boundary and a single line break that does not split a sentence. The text is scanned once, sentences are yielded as they are found.
• WordsExtractor - words by the rule-based tokenizer of the blank Spanish pipeline of spaCy, which needs no model: clitics (dámelo), abbreviations and Spanish numbers (1.500,50, 3.º, 1990-1995) stay whole. Filters for punctuation, numbers (signed numbers, ranges, dates, times, percentages and ordinals), lemmas via simplemma, lower case, stop words compared case-insensitively, word length and N-grams.
• CharNgramsExtractor - character N-grams over the string or inside words, the feature stylometry is built on.

Syllables and stress

• syllabify by the orthographic rules of the Ortografía de la lengua española (RAE, 2010): diphthongs, hiatuses and triphthongs, an accented weak vowel, two close vowels of the same letter as a hiatus, the h that does not break a diphthong, the silent u of qu and gu, the vocalic ü, y as a vowel word-finally, digraphs, onset clusters and the distribution of three or more consonants.
• word_stress, word_stresses and stress_type - the stressed syllable from the written accent or from the ending of the word, two stresses for adverbs in -mente and one per part of a hyphenated compound, the classes aguda, llana, esdrújula and sobresdrújula. No dictionary and no download.
• Checked word by word against silabeador on the 520 stop words of spaCy and by hand on about 150 words, including foreign names and decomposed (NFD) input.

Basic statistics

• BasicStats - sentences, words, unique, long, complex, simple, monosyllabic and polysyllabic words, characters, letters, spaces, syllables, punctuation marks with the distribution by type, the distributions of words by letters and by syllables, and the normalized shares of all of them. A complex word has three or more syllables and a long word seven or more letters, as the Spanish formulas count them.
• count_punctuations and punctuation_profile - thirteen types of marks, with the inverted ¿ and ¡ counted as question and exclamation marks, the ellipsis of …, ... and ?.. as one mark, the dash of a dialogue told from a hyphen and from an end-of-line hyphenation, and inverted_share, the share of inverted marks among all question and exclamation marks.

Readability

• ReadabilityStats - the Flesch reading ease with the coefficients of Szigriszt-Pazos (1993) or of Fernández Huerta (1959), the comprehensibility formula of Gutiérrez de Polini (1972), the grade of Crawford (1989), Legibilidad µ of Muñoz Baquedano and Muñoz Urra (2006), the SOL grade of Contreras et al. (1999) and the language-independent LIX and RIX.
• The interpretation layer: the INFLESZ scale of Barrio-Cantalejo et al. (2008) and the bands of Szigriszt-Pazos and of Fernández Huerta, the µ scale, a consensus grade over the grade formulas, the school stages of Spain with the reader age, and reading time by the speeds of Spanish-speaking readers (Brysbaert 2019, Ripoll, Tapia and Aguado 2020).
• The scale follows the preset: general is read on INFLESZ, classic on the bands of Fernández Huerta, in describe_level and in the conversion that feeds the consensus grade.
• Every coefficient and every band was taken from its source rather than from another library: the last term of Fernández Huerta is read as the mean sentence length, as Law (2011) showed it must be; Legibilidad µ reproduces the worked example of the authors' manual exactly; the SOL conversion comes from the table of the original article.

Lexical diversity

• DiversityStats - 32 metrics: the TTR family (TTR, RTTR, CTTR, Herdan, Summer, Maas, Dugast), MATTR and MSTTR, MTLD with its moving-window variants MA-MTLD and MTLD-W, HD-D, the Simpson family, the hapax index (Honoré's R), the measures of Yule, Herdan, Sichel, Michéa, Brunet, Dugast and Baayen, the hapax ratio and α₂, Shannon entropy, evenness and perplexity, the slope of Zipf's law, the Zipf-Mandelbrot fit and the exponent of Heaps' law.
• windowed - any metric over windows of equal length with the mean, the standard deviation and a confidence interval, the standard way to compare texts of different lengths.
• The conventions of koRpus and of Kyle's lexical-diversity, all of them parameters of the class: logarithm base 10, window and segment of 50 words, MTLD threshold 0.72 with a minimum factor of 10 words, HD-D sample of 42 words.

Quality

• 676 tests, coverage 100 %, ruff and mypy clean; the CI runs on Python 3.11 to 3.14 and on Linux, Windows and macOS.
• The values of the metrics are pinned to worked examples computed by hand and to reference implementations; the documentation is bilingual, English and Spanish, and every example in it is executed and compared with its printed output.

Documentation: https://sergeyshk.github.io/esTS/ · Commits: https://github.com/SergeyShk/esTS/commits/0.1.0