0.3.0

The third release of esTS moves from a single text to corpora. It adds the measures of corpus linguistics and stylometry, visualizations, two datasets (a corpus of Spanish-language literature and a frequency dictionary), and lexical sophistication statistics based on that dictionary.

Requires Python 3.11 or newer. The distribution on PyPI is pyests and the package it installs is ests: pip install pyests, import ests. The spaCy model es_core_news_sm is needed by the morphological, syntactic, cohesion and lexical sophistication statistics of a string, by the function-word profile and by the comparison of corpora. The datasets are downloaded once with their download() method.

Incompatible changes

The API of 0.2 is unchanged, and all new functionality is additive. Three things may still affect existing code:

• Dialogue dashes. The tokenizer now splits off a dialogue dash stuck to a word, whether typed as hyphens in plain text (--No, -dijo, sí--dijo) or as a dash between words (reírse—me). The string API and the components share these rules. Numbers therefore change on texts with dialogue. On 13 novels and plays of the corpus, compared with 0.2.0:

  • n_words changes by up to 1% (median 0.2%);
  • n_unique_words changes by up to 7.7% (median 3.9%), because a word with a dash stuck to it used to count as a separate word;
  • dialogue hyphens in c_punctuations move from hyphen to dash;
  • n_sents does not change.

  Every readability and diversity measure built on these counts moves with them.

• New required dependencies: pandas, matplotlib and graphviz. The word tree and the network of collocations are rendered by the executables of Graphviz, which are installed separately.

• simplemma: the minimum goes from 1.0 to 2.0, with a ceiling below 3. The keys of the frequency dictionary are the lemmas of simplemma 2.0.0, and FreqDict warns once if another version is installed.

Corpus measures

• keyness compares a target corpus with a reference: a list of words, a table of frequencies, or the frequency dictionary. It offers seven measures: log-likelihood G², chi-square, %DIFF, Log Ratio, BIC, ELL and the odds ratio. Every keyword also carries G², Log Ratio and the p-value of G². Against the dictionary, the words of the target are looked up by the keys the dictionary was built with. A word missing from the dictionary gets the dictionary's lowest frequency rather than zero, so it can only be a keyword of the target.
• collocations finds ordered pairs within a window, with eight association measures: MI, MI³, t-score, Dice, logDice, log-likelihood, NPMI and minimum sensitivity. It can also return the collocations of a single word.
• dispersion measures how a word spreads over the parts of a text, with Gries's DP and its normalized form, Juilland's D, Carroll's D2, Rosengren's S and the Kullback-Leibler divergence.
• kwic builds a concordance. format_kwic and print_kwic align it on the keyword.
• Stylometry: the frequency table and z-scores of the most frequent words; Delta in four variants (Burrows, Argamon's quadratic, Eder, cosine), both between texts and from texts to reference texts for attribution; Zeta after Burrows and Craig, with its logarithmic form; Kilgarriff's chi-square; Mendenhall's curves and the distance between them; and the profile of function words by part of speech.
• Comparison of corpora: texts are cut into windows and 132 features are computed for every window. compare_corpora tests every feature and reports effect sizes (Cohen's d, Cliff's delta), a bootstrap interval of the difference of medians and Holm's correction.

Visualizations

• ests.visualizers returns matplotlib Axes or a Graphviz graph:
  • Zipf's law with a fit of Zipf-Mandelbrot;
  • literature fingerprinting;
  • a word tree;
  • dispersion, keyness and a network of collocations;
  • a dendrogram, PCA and MDS of stylometric distances, and Mendenhall's curves;
  • Heaps' law and the frequency spectrum;
  • sentence lengths.

Datasets

• SpanishLiterature holds 150 public-domain works by 33 authors from Spain, Latin America and the Philippines, from Cervantes to the 1920s: about 11 million words in four genres (prose, poems, drama, publicism). Works can be filtered by author, genre, country, years and length.
• FreqDict is a frequency dictionary of 83,785 Spanish lemmas from the Spanish books of Google Books Ngram for 1980-2019 (63 billion words). Each lemma has its frequency per million words, range, Juilland's D and number of books. It is distributed under CC BY 3.0.
• Both are downloaded once and verified against their SHA-256 checksum. They are extracted atomically, so an interrupted extraction never leaves half a dataset behind. They are stored in ests_data next to the package, or in the directory set by the environment variable ESTS_DATA_DIR or by data_dir.

Lexical sophistication

• LexicalStats, in the manner of TAALES, measures how rare the words of a text are:
  • the coverage of the dictionary;
  • the mean frequency and log frequency of the lemmas, over all words and over content words, and their mean range and dispersion;
  • the shares of the frequency bands top-1000 to top-10000;
  • surprisal and perplexity under the dictionary's unigram model;
  • lexical density.
• The component ests_lexical is the seventh factory of the spaCy pipeline.

Quality

• 1496 tests at 100% coverage of lines and branches; ruff and mypy are clean. CI runs on Python 3.11 to 3.14 and on Linux, Windows and macOS. The network tests run in a job of their own after a merge.
• The release went through a QA pass of its own, and all 22 of its findings are fixed. The documentation is bilingual, and its examples take their texts from the corpus of literature.

Documentation: https://sergeyshk.github.io/esTS/ · Commits: 0.2.0...0.3.0