Skip to content

0.6.0

Choose a tag to compare

View all tags
@SergeyShk SergeyShk released this 29 Sep 19:50
· 166 commits to master since this release
0.6.0
eac003e
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

The sixth release of esTS moves the library onto the second slice of anyTS, the language-independent core of text statistics. It also:

  • revisits how sentences, clauses and connectors are counted;
  • computes alliteration and assonance against the words in random order;
  • adds a gallery of notebooks.

Requires Python 3.11 or newer. Install with pip install pyests and import with import ests. The dependencies are now anyts>=0.2.3,<0.3 and spacy>=3.8, both installed with the package.

Incompatible changes

The public classes and functions keep their names, and their signatures change only where noted below.

  • Dependencies. spaCy 3.8 or newer is required: on spaCy 3.7 import ests fails without click, which typer no longer installs. The model es_core_news_sm 3.8 and the demo were on 3.8 already. anyTS 0.2.3 or newer within 0.2 is also required.
  • calc_smog_index(n_complex, n_sents, a, b, c). b is now the number of sentences of the sample (30), and the constant is c. A positional b changes meaning: calc_smog_index(10, 30, 1.043, 3.1291) gave 6.427 and now gives 4.194.
  • Names that left esTS. They live in the core; import them from anyts.
    • ests.constants no longer has COMPLEX_SYL_FACTOR, LONG_WORD_LETTER_FACTOR, SPACES, PUNCTUATION_TYPES, BASIC_STATS_DESC, SMOG_COMPLEX_SYL_FACTOR and LIX_LONG_WORD_LETTER_FACTOR (they are in anyts.constants), nor PUNCTUATIONS, HIGHLIGHT_SYNTAX_LAYERS and HIGHLIGHT_TAGGED_LAYERS (replaced by HIGHLIGHT_LAYER_ANNOTATIONS).
    • ests.utils no longer has to_path, download_file, extract_archive, sha256 (in anyts.datasets), DOWNLOAD_TIMEOUT or TAR_DATA_FILTER. The module ests.datasets.dataset is gone.
    • Undocumented names are gone:
      • DASH_PATTERN, ELLIPSIS_PATTERN and PUNCTUATION_CHARS of ests.basic_stats;
      • WordTree, TreeDrawer, FreqNode and Direction of ests.visualizers.word_tree;
      • MARGIN, MISSING_COLOR and SQUARE of ests.visualizers.fingerprinting;
      • LINE_BREAK, CSS, select_layers, get_text_words, get_text_sents, get_doc_words, get_doc_sents and split_segments of the highlighting;
      • SOUNDS, SOUNDS_ORDER, CONSONANT_COLUMNS and the private helpers of the repetition index in ests.phon_stats.
    • Word, Sent and Highlight are the classes of the core.
  • Stricter checks. These raise ParameterError or SourceTypeError instead of being accepted or failing with a bare Python error:
    • Readability.
      • calc_consensus_grade checks the preset even without a reading ease.
      • A grade or a reading ease that is not a finite number raises ParameterError when it is converted or averaged: flesch_reading_easy_to_grade, ReadabilityStats.reading_ease_to_grade, calc_consensus_grade and grade_to_age. flesch_reading_easy_to_grade(nan) gave 13.
      • A reading speed that is not a positive number also raises: calc_reading_time(10, nan) gave nan.
      • ReadabilityStats checks the extractors given with ready basic statistics and refuses basic statistics of another class.
    • Types of words and features.
      • A word that is not a string in the syllables (syllabify, count_syllables, word_stress, word_stresses, stress_type), in transcribe and in is_stopword.
      • Words or parts of speech of find_connectors that are not a list: generators are refused now. Parts of speech must be as many as the words.
      • Features of compare_corpora and corpus_features that are not a function: features=0 no longer falls back to the default.
    • Plots.
      • The axes of every plot (ax) must be a matplotlib Axes.
      • The keywords of keyness_plot, the collocations of collocation_network and the concordances of format_kwic are checked; iterators are still accepted.
      • highlight refuses a bool or nan as alliteration_threshold, and sentence_lengths refuses sets, mappings and negative lengths.
      • A component name taken by an extension of another package raises ParameterError.
  • The formulas of counts return nan when the number of words or sentences they divide by is zero, instead of raising ZeroDivisionError. This covers Flesch, Flesch-Kincaid, Coleman-Liau, ARI, SMOG, fog, LIX, RIX, Gutiérrez de Polini, Crawford and SOL. On other inputs the values are unchanged.
  • kwic compares the word forms in NFC without soft hyphens, so a decomposed accent matches. A phrase no longer runs across the end of a sentence (by the Spanish rules) or of a paragraph.
  • expand_phrases: a phrase written as such wins over the forms of another phrase (a efectos del next to a efectos de). The built-in lists do not change.
  • fingerprinting makes every segment segment_len words long (anyTS 0.2).

Numbers

The numbers of 0.5.0 are kept except for the changes below. On a fixed set of texts 156 of 4620 results changed, all in these categories.

  • Sentences.
    • A dash glued to the terminal mark, as the editions of Project Gutenberg type it, now ends the sentence: baja.--Tiene opens a new line of dialogue.
    • A glued dash before whitespace closes its line of dialogue and stays with it: ¡Traidores!--, viuda.-- El niño.
    • A remark after a spaced dash no longer splits the sentence: Sí. --dijo él.
    • Abbreviations, initials and list markers still end nothing.
    • The corpus of literature has 634,712 sentences instead of 624,200 (+1.7%).
    • n_sents and the readability of a string move with the sentence count. A parsed Doc keeps the sentences of the parser.
  • Clauses (SyntaxStats). On 450 windows of the corpus of literature, clauses_per_sent rises by 11.7%, subordinate_clauses_per_sent by 21.2% and p_complex_sents by 8.6%, while mean_clause_len falls by 10.2%. The models es_core_news_lg and es_dep_news_trf shift by about as much. The rules behind this:
    • An infinitive under xcomp heads a subordinate clause (quiere salir), unless it makes a periphrasis with its verb (suele salir, vuelve a salir: INFINITIVE_PERIPHRASES, NGLE ch. 28) or the verb is parecer.
    • A periphrasis whose infinitive the models attach as conj (tiene que salir) is one predicate.
    • The form of a predicate is read from its first auxiliary or copula (predicate_form):
      • compound tenses, passives and progressives head their clauses (había llegado, estaba cantando), and so do finite predicates under acl and xcomp;
      • an infinitive form under acl is no clause (el miedo de ser robado);
      • compound and copular gerunds are gerund clauses (habiendo llegado tarde, siendo elegido presidente).
  • Connectors (CohesionStats).
    • The dictionary has 250 markers. apenas, nada más, lo mismo que, esto es and es más are removed. después de todo is now reformulative; en primer/segundo/tercer/último lugar and primeramente are now additive.
    • Rules of place:
      • pues opening a sentence is no connector;
      • encima counts only at the start of a sentence or after y, e, ni or pero;
      • en cuanto a and algo así como count nothing.
    • On the 450 windows, connectors per 1000 words fall by 2.3%: causal −16.1%, reformulative −10.1%, temporal −5.9%.
    • The documentation now credits the markers correctly (Martín Zorraquino & Portolés and the NGLE, grouped in the seven classes of the library), with the corrected Coh-Metrix-Esp link.
  • Alliteration and assonance (PhonStats, calc_alliteration, calc_assonance).
    • The expected number of repetitions is the exact one for the words in random order (hypergeometric), through anyts.phonetics.calc_repetition_index.
    • Before, it was binomial and read below 1 on random text: 0.966 on shuffled sonnets, now 0.996.
    • Books change in the third digit.
    • A text where no sound is shared by two words gives nan instead of 0.0.

Notebooks

A gallery of four notebooks in examples/ shows the library on its datasets, and a Colab button opens each:

  • a short story of Quiroga through every tool;
  • ten prose authors compared by stylometry;
  • the Spanish sonnet across four centuries;
  • the corpus measures, Spain against Spanish America.

The outputs are kept, and a weekly workflow runs the notebooks and fails when a number their text quotes has moved.

Fixes

  • Sentence splitting is linear on long runs of spaces after a dash and on runs of terminal marks, which took seconds.
  • Highlighting a Doc is linear again: 400,000 characters take 0.16 s instead of 42 s (anyTS 0.2.3).
  • zipf(show_theory=True) accepts an integer alpha.
  • SyntaxStats finds the clause heads once, so it is about as fast as in 0.5.0 despite the new rules.
  • The documentation examples, the rules on the pages, the Spanish of the translations and the notes on the models (the recall of the imperative per model) are corrected.

Quality

  • 1811 tests pass with 100% line and branch coverage.
  • ruff and mypy are clean.
  • The notebooks are checked weekly.
  • The numbers of all statistics were compared with 0.5.0 on a fixed set of texts; everything outside the changes above is bitwise equal.

Links

Assets 2
Loading