0.1.0

The first release of esTS, a library for statistics extraction from texts in Spanish: object extraction, syllabification and stress, basic statistics, readability and lexical diversity - by published formulas with the coefficients and the scales of their authors, deterministically and without a neural network inside.

Requires Python 3.11 or newer. The distribution on PyPI is pyests, the package it installs is ests: pip install pyests, import ests. Dependencies: numpy, scipy, simplemma, spacy; no trained spaCy model is needed, one is required only to parse a text yourself and pass the Doc instead of a string.

Object extraction

• SentsExtractor - sentence splitting by rules that know Spanish: the inverted marks ¿ and ¡, quotes and brackets of three levels, the dash of a line of dialogue with the remark of the narrator (—¿Vienes? —preguntó María is one sentence), abbreviations (Sr., Dra., p. ej., EE. UU., a. m.), capital initials, list markers, a lower-case continuation after an ellipsis, a blank line as a boundary and a single line break that does not split a sentence. The text is scanned once, sentences are yielded as they are found.
• WordsExtractor - words by the rule-based tokenizer of the blank Spanish pipeline of spaCy, which needs no model: clitics (dámelo), abbreviations and Spanish numbers (1.500,50, 3.º, 1990-1995) stay whole. Filters for punctuation, numbers (signed numbers, ranges, dates, times, percentages and ordinals), lemmas via simplemma, lower case, stop words compared case-insensitively, word length and N-grams.
• CharNgramsExtractor - character N-grams over the string or inside words, the feature stylometry is built on.

Syllables and stress

• syllabify by the orthographic rules of the Ortografía de la lengua española (RAE, 2010): diphthongs, hiatuses and triphthongs, an accented weak vowel, two close vowels of the same letter as a hiatus, the h that does not break a diphthong, the silent u of qu and gu, the vocalic ü, y as a vowel word-finally, digraphs, onset clusters and the distribution of three or more consonants.
• word_stress, word_stresses and stress_type - the stressed syllable from the written accent or from the ending of the word, two stresses for adverbs in -mente and one per part of a hyphenated compound, the classes aguda, llana, esdrújula and sobresdrújula. No dictionary and no download.
• Checked word by word against silabeador on the 520 stop words of spaCy and by hand on about 150 words, including foreign names and decomposed (NFD) input.

Basic statistics

• BasicStats - sentences, words, unique, long, complex, simple, monosyllabic and polysyllabic words, characters, letters, spaces, syllables, punctuation marks with the distribution by type, the distributions of words by letters and by syllables, and the normalized shares of all of them. A complex word has three or more syllables and a long word seven or more letters, as the Spanish formulas count them.
• count_punctuations and punctuation_profile - thirteen types of marks, with the inverted ¿ and ¡ counted as question and exclamation marks, the ellipsis of …, ... and ?.. as one mark, the dash of a dialogue told from a hyphen and from an end-of-line hyphenation, and inverted_share, the share of inverted marks among all question and exclamation marks.

Readability

• ReadabilityStats - the Flesch reading ease with the coefficients of Szigriszt-Pazos (1993) or of Fernández Huerta (1959), the comprehensibility formula of Gutiérrez de Polini (1972), the grade of Crawford (1989), Legibilidad µ of Muñoz Baquedano and Muñoz Urra (2006), the SOL grade of Contreras et al. (1999) and the language-independent LIX and RIX.
• The interpretation layer: the INFLESZ scale of Barrio-Cantalejo et al. (2008) and the bands of Szigriszt-Pazos and of Fernández Huerta, the µ scale, a consensus grade over the grade formulas, the school stages of Spain with the reader age, and reading time by the speeds of Spanish-speaking readers (Brysbaert 2019, Ripoll, Tapia and Aguado 2020).
• The scale follows the preset: general is read on INFLESZ, classic on the bands of Fernández Huerta, in describe_level and in the conversion that feeds the consensus grade.
• Every coefficient and every band was taken from its source rather than from another library: the last term of Fernández Huerta is read as the mean sentence length, as Law (2011) showed it must be; Legibilidad µ reproduces the worked example of the authors' manual exactly; the SOL conversion comes from the table of the original article.

Lexical diversity

• DiversityStats - 32 metrics: the TTR family (TTR, RTTR, CTTR, Herdan, Summer, Maas, Dugast), MATTR and MSTTR, MTLD with its moving-window variants MA-MTLD and MTLD-W, HD-D, the Simpson family, the hapax index (Honoré's R), the measures of Yule, Herdan, Sichel, Michéa, Brunet, Dugast and Baayen, the hapax ratio and α₂, Shannon entropy, evenness and perplexity, the slope of Zipf's law, the Zipf-Mandelbrot fit and the exponent of Heaps' law.
• windowed - any metric over windows of equal length with the mean, the standard deviation and a confidence interval, the standard way to compare texts of different lengths.
• The conventions of koRpus and of Kyle's lexical-diversity, all of them parameters of the class: logarithm base 10, window and segment of 50 words, MTLD threshold 0.72 with a minimum factor of 10 words, HD-D sample of 42 words.

Quality

• 676 tests, coverage 100 %, ruff and mypy clean; the CI runs on Python 3.11 to 3.14 and on Linux, Windows and macOS.
• The values of the metrics are pinned to worked examples computed by hand and to reference implementations; the documentation is bilingual, English and Spanish, and every example in it is executed and compared with its printed output.

Documentation: https://sergeyshk.github.io/esTS/ · Commits: https://github.com/SergeyShk/esTS/commits/0.1.0