0.5.0

The fifth release of esTS moves the library onto anyTS, the language-independent core of text statistics, and adds a demo on Hugging Face Spaces.

Requires Python 3.11 or newer. Install with pip install pyests and import with import ests. The new dependency is anyts>=0.1,<0.2, installed with the package.

Incompatible changes

The numbers of 0.4.0 are kept, except for the changes below; the public classes and functions keep their names and signatures.

• Names that left esTS. They live in the core; import them from anyts.
  • ests.constants no longer has the defaults of the diversity metrics (MATTR_WINDOW_LEN, MTLD_*, HDD_SAMPLE_SIZE, DIVERSITY_LOG_BASE, BRUNET_W_EXPONENT, DIVERSITY_STATS_DESC) nor VALENCY_IGNORED_DEPS and the names of the corpus measures (KEYNESS_MEASURES, G2_CRITICAL_VALUES, COLLOCATION_MEASURES, DISPERSION_STATS_DESC, DELTA_VARIANTS): they are in anyts.constants.
  • Undocumented helpers are gone from the esTS modules: is_punctuation, count_letters, safe_divide, has_words, check_sequence, iter_doc_tokens of ests.utils; Overlap, calc_overlap, calc_proportional_overlap, dice, dominant of ests.cohesion_stats; Extractor, Tokenizer of ests.extractors; Calculator of ests.diversity_stats; MEASURES, ZERO_ADJUSTMENT, ZERO_SEGMENTS, Values, compare_values of the corpus modules; the private helpers of the moved code.
  • No longer re-exported, though still importable at runtime (type checking with --no-implicit-reexport fails on them): iter_doc_words of ests.utils; calc_overlaps, count_given, calc_repetition of ests.cohesion_stats; the twelve tree helpers of ests.syntax_stats (base_dep, get_words, calc_tree_depth, ...).
• Exceptions are the classes of the core under the same names, and EstsError is AnyTSError: except EstsError also catches the errors raised by the code of the core, and tracebacks show anyts.exceptions.SourceTypeError.
• Numbers.
  • Coordination chains are the connected components over conj: coordination_chains_per_sent and mean_coordination_chain_len change for texts with nested coordination.
  • The last digits of content_overlap_prop_all, summed with math.fsum, no longer depend on the hash seed.
  • DTTR of a text or a window without repetitions is inf (was 0); STTR of a text of one lexeme or of no more words than the logarithm base is nan; zipf_alpha of a text without repetitions is 0.0 (was -0.0).
  • ELL of rare words is a number; it is nan only when the least expected frequency is at most 1 (was below e).
  • Tied most frequent words of frequency_table and delta are ordered by their means rounded to 12 digits, then alphabetically: the selection at its border and Eder's Delta, weighted by rank, can change.
  • Negative keywords against FreqDict no longer include the dictionary keys the target never counts (cd-rom, r.p.m.).
  • A byte order mark at the start of a text is split off: the words are those of the text without it. In a parsed Doc it stays a token of its own, so the parse of the first sentence can still differ.
  • is_punctuation also takes combining marks and invisible format characters (Unicode M and Cf): a lone zero-width space is no word.
  • The sentences of a custom tokenizer of SentsExtractor are stripped, and WordsExtractor with a custom tokenizer drops the tokens made only of whitespace.
• Stricter checks. Parameters and sources of the wrong type or value raise ParameterError or SourceTypeError instead of being accepted or failing with a bare TypeError, ValueError or AttributeError:
  • a float or a bool for a count, window, width, number of items, limit, length, factor or number of bootstrap samples; a negative length bound;
  • a string, an iterator or a list of non-strings (spaCy tokens) for a list of words; a non-string text or keyword;
  • an extractor, a pipeline or a frequency dictionary of another type; a non-string preset or dataset filter;
  • calc_zipf_naturalness(words, 0) and the other module functions check top_n and window_len as their classes do; sentence_rhythm takes integer lengths only.
  • The classes check their parameters before the source, so a wrong parameter is reported before an empty text and before any parsing.

The anyTS core

• Exceptions, utilities, the extractors, the lexical diversity metrics, the cohesion and tree helpers and the corpus measures (collocations, dispersion, keyness, stylometry, the statistics of the comparison) come from anyts 0.1.
• esTS keeps the Spanish layer:
  • the extractors are subclasses of those of the core with the Spanish hooks: the sentence splitter, the spaCy tokenizer, simplemma and the pattern of Spanish numbers;
  • DiversityStats takes a string or a Doc, as before;
  • keyness also takes the frequency dictionary FreqDict; the reference by frequencies of the core, FrequencyReference, is in ests.corpus;
  • function_words_profile, compare_corpora and the features of a text stay in esTS, like all the Spanish statistics.

Documentation

• The English pages of the moved parts include the text of the anyTS pages, so every formula and parameter is described in one place; the Spanish pages translate it, and a test fails when a section of the core changes until its translation is updated.
• The pages of the extractors describe the Spanish hooks; the comparison page describes the functions of its statistics.

Demo

• https://huggingface.co/spaces/SergeyShk/esTS: paste a Spanish text and get its readability with the school stage, every class of statistics, the keywords and collocations against the frequency dictionary, the plots of Zipf's law and of sentence lengths, and the highlighting.
• The code is in demo/: make demo runs it locally, and every release updates the Space.

Quality

• 1785 tests with 100% line and branch coverage.
• ruff and mypy are clean.
• The numbers of all statistics were compared with 0.4.0 on a fixed set of texts: everything else is bitwise equal.

Links

• Documentation: https://sergeyshk.github.io/esTS/
• Demo: https://huggingface.co/spaces/SergeyShk/esTS
• PyPI: https://pypi.org/project/pyests/0.5.0/
• anyTS: https://github.com/SergeyShk/anyTS