Skip to content

0.7.0

Choose a tag to compare

View all tags
@SergeyShk SergeyShk released this 16 Oct 00:54
· 82 commits to master since this release
0.7.0
ac4098f
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

The seventh release of esTS moves the library onto anyTS 0.2.13, the language-independent core of text statistics. It also:

  • reads every readability statistic by a scale;
  • starts a sentence after a blank line in a parsed Doc;
  • shows the words behind the officialese markers and the academic nausea, which now leaves out the stopwords;
  • keeps abbreviations whole in every case.

Requires Python 3.11 or newer. Install with pip install -U pyests and import with import ests. The dependencies are now anyts>=0.2.13,<0.3 and spacy>=3.8.15, both installed with the package.

New

  • Readability readings.
    • ReadabilityStats.describe(stat) reads every statistic that has a scale. The reading ease is read by the scale of the preset, INFLESZ for general and Fernández Huerta for classic; level_scale(stat) returns that scale. Legibilidad µ is read by its bands, and LIX by the text types of Björnsson (the English labels of the core).
    • The grades of Crawford, SOL, RIX and the consensus grade are read as a stage of the Spanish school system with its ages.
    • describe gives None for a statistic without a scale or with an undefined value.
    • scale_level, the reading of a value by any scale, is exported from ests.readability_stats.
  • Blank lines.
    • The ests_blank_lines component starts a sentence after a blank line, so the parser no longer glues a paragraph to the next sentence.
      • get_nlp puts it before the parser (or the senter or sentencizer).
      • It runs in linear time (0.025 s on 4000 paragraphs), so get_nlp parses as fast as in 0.6.1.
      • It has an entry point, so a saved pipeline loads without importing esTS.
    • SentsExtractor splits at blank lines with a tokenizer of your own too.
    • A Doc with only partial sentence starts, such as one run without the parser, is read by its text in BasicStats, CohesionStats, kwic, sentence_lengths and the highlighting.
  • Style.
    • StyleStats.markers() returns the words and phrases each officialese marker counts: the verbal nouns, the compound prepositions, the parenthetical expressions and the clichés, in the order of the text.
    • find_parentheticals(words) returns the bounds of the parenthetical expressions in a list of words, as pairs of start and end.
    • StyleStats.nausea_words() and find_nausea_words return the words of the academic nausea with their occurrences. The nausea_stopwords parameter of StyleStats and of the ests_style component sets its stopwords; it is the last parameter, so positional calls keep working.
  • Abbreviations. ests.constants.TOKENIZER_ABBREVIATIONS lists the 46 abbreviations that spaCy keeps whole in some cases only (Sr., pág., etc., a.m.). Every tokenizer of esTS keeps them whole in lower case, capitalized and in upper case, so kwic(text, "sr. pérez") finds Sr. Pérez too.
  • Corpus. ests.corpus.check_keyness_params checks the parameters of keyness and is documented.
  • Demo. The readability table has a "Reading" column.

Incompatible changes

The public classes and functions keep their names. Their signatures change only by new optional parameters:

  • nausea_stopwords of StyleStats and ests_style, and stopwords of calc_academic_nausea, both last;
  • the keyword-only messages of calc_windowed and check_params in ests.diversity_stats.
  • Dependencies.
    • anyTS 0.2.13 or newer within 0.2 is required.
    • spaCy 3.8.15 or newer is required: spaCy 3.8.2-3.8.14 import click without requiring it, so import ests failed there without it.
  • Readability levels.
    • describe_level, flesch_reading_easy_to_level and mu_to_level raise ParameterError for a value that is not a finite number. Before:
      • for nan they returned "";
      • for inf and -inf they returned the top and the bottom band;
      • for a value that is not a number they raised TypeError.
    • describe_level and flesch_reading_easy_to_level raise ParameterError for a scale that is not a known name. A list or a set raised TypeError, and a false scale ("", [], 0) in describe_level fell back to the scale of the preset.
    • describe_grade raises ParameterError for an array of names, where it raised TypeError.
    • describe_level("lix") returns a band instead of raising.
  • Academic nausea. It sums the top_n most frequent words without the stopwords, divided by all words. Before, the stopwords were counted, so the top ten of almost every text was de, la, que, el, and none of 300 excerpts of SpanishLiterature was within the norm of Advego (5-15%). On these excerpts the median falls from 29.2% to 6.3%, and 245 of them are within the norm.
  • Abbreviations as words. The tokenizer of get_tokenizer and get_nlp keeps the 46 abbreviations of TOKENIZER_ABBREVIATIONS whole in every case, so sr., ud., Pág. and SR. are one word with the period, where 0.6.1 split the period off. The words of the extractors and the frequency counts change with them, and a keyword without the period no longer finds such a word: kwic("Habló el sr. Pérez ayer.", "sr") finds nothing, "sr." finds it.
  • Pipeline.
    • get_nlp().pipe_names has ests_blank_lines before the parser, so the sentences of a parsed Doc start after blank lines.
    • Every component of esTS applies the tokenizer rules to the whole pipeline it is added to, abbreviations included.
  • LexicalStats.band_coverage raises ParameterError for bounds that are not an iterable (100, None), where it raised TypeError. An iterator is read once.
  • Characters without a UTF-8 form. A text with a lone surrogate raises SourceError in every class that parses it, where spaCy raised UnicodeEncodeError.

Numbers

On a fixed set of texts 534 of 4620 results changed against 0.6.1, all in these categories; everything else is bitwise equal.

  • Sentences of a Doc. A string or a Doc parsed through get_nlp now has a sentence boundary at every blank line. A paragraph opened by a dash or a quote keeps the mark with its first word.
    • Everything built on the sentences of a Doc moves for the texts with blank lines: syntax, cohesion, and BasicStats and ReadabilityStats of a Doc.
    • SyntaxStats and CohesionStats of a string parse it with get_nlp, so they move with the Doc.
    • BasicStats and ReadabilityStats of a string split it with sentenize, so these changes do not touch them.
  • Sentences of a string.
    • A sentence goes on after an abbreviation written without spaces (a.c., p.ej., EE.UU.), as it does after the spaced forms, and after esq. and apdo..
    • A line of Unicode spaces (a no-break space, U+3000) between two line breaks is now blank and ends a sentence.
  • Abbreviations.
    • Texts with SR., Pág., ud., ee. uu. or a.m. change their words and their parse.
    • A dash of a dialogue glued to a dotted acronym (S.A.—dijo, »P.D.--El) no longer splits its last period off.
    • LexicalStats looks an abbreviation up without its period, so Sr., SRA., pág. and etc. find sr, sra, página and etcétera in the frequency dictionary. An initial (D., J.) keeps its period and stays out of the dictionary.
  • Academic nausea, as described above.
  • PhonStats of a text without sounds. Every statistic is nan; the shares were 0 while the ratios were nan. p_heavy_clusters is nan for a text without consonants.
  • zipf_naturalness is computed in integers, so only its last bits change: frequencies 11, 8, 8, 2, 2 give 50.0 instead of 49.999999999999986, just under the norm.
  • n_spaces and p_spaces count every Unicode space (Zs), so they rise in texts with no-break spaces.

Fixes

  • SyntaxStats follows a long chain of coordinated clauses without recursion: 600 of them raised RecursionError.
  • dispersion_plot and frequency_spectrum_plot accept a numpy array or a pandas Series of words, and fingerprinting and wordtree one of texts, instead of failing on their truth value; an empty one raises SourceError (anyTS 0.2.10).
  • ests.utils.count_words_by_spans reads a pandas Series by position, not by its index labels, and returns Python integers.
  • A component of esTS added to a pipeline that has its tokenizer rules already does not reload the tokenizer. Loading es_core_news_sm with the ten components takes about half the time of 0.6.1.
  • The demo shows an infinite value as a dash.

Quality

  • 1995 tests pass with 100% line and branch coverage, which is now required.

  • ruff and mypy are clean.

  • A QA pass before the release covered:

    • a fuzzing of 2.6 million calls on 3504 inputs;
    • the lowest versions of the dependencies and Python 3.14;
    • the notebooks and the demo;
    • the documentation in English and Spanish, with its 132 examples run on each language.

    Its findings are fixed in anyTS 0.2.13 and in this release.

Links

Assets 2
Loading