Skip to content

0.8.0

Choose a tag to compare

View all tags
@SergeyShk SergeyShk released this 23 Oct 23:49
· 39 commits to master since this release
0.8.0
8c63cd7
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

The eighth release of esTS moves every class of statistics onto the frames of anyTS 0.3, the language-independent core of text statistics. esTS keeps the Spanish layer: the tokenizer and the sentences, the transcription and the syllables, the clause rules, the lists of words and the dictionary of connectors. It also:

  • counts the infinitive clauses apart from the subordinate ones;
  • counts the connectors by their place in the sentence;
  • imports about ten times faster, without loading spaCy;
  • gives every class of statistics the same interface: get_stats(flat=True), to_dataframe, describe and print_stats.

Requires Python 3.11 or newer. Install with pip install -U pyests and import with import ests. The dependencies are now anyts>=0.3.9,<0.4 and spacy>=3.8.15,<4, both installed with the package.

New

  • The frames of anyTS 0.3. StyleStats, LexicalStats, PhonStats, SyntaxStats and CohesionStats are subclasses of the frames of the core. So are the spaCy components and the finders of the highlighting.
  • One interface for all classes. MorphStats and VerseStats are now Stats of the core too.
    • Every class has get_stats(flat=True), to_dataframe(), describe(stat) and the table of print_stats.
    • MorphStats.get_stats gives the counts by keys such as mood_Sub with flat=True.
    • describe gives None for a statistic without a scale.
  • Infinitive clauses.
    • The new key infinitive_clauses_per_sent counts the infinitive clauses under a verb (quiere salir). They no longer count as subordinate clauses.
    • is_infinitive_clause_head is new in ests.syntax_stats.
    • is_infinitive_periphrasis also covers the infinitive of a raising verb under xcomp (parece dormir).
  • Connectors by place.
    • The new keys connectors_inter and connectors_intra count, per 1000 words, the connectors that link a sentence to the ones before and those that link the parts of a sentence.
    • A subordinator that opens a sentence begins its own clause or phrase, so it is intra-sentential.
    • SpanishConnectorMatcher is public. It carries the Spanish rules as class attributes, with three new lists of ests.constants:
      • CONNECTOR_SUBORDINATORS: 66 conjunctive and prepositional phrases (ya que, a pesar de, siempre que);
      • CONNECTOR_PARCELLATION: porque and conque, which link a sentence they open to the one before;
      • CONNECTOR_SPLIT_BY_MARKS: 51 phrases ending in que, como or si, which a punctuation mark inside splits (No tire así, que se romperá).
    • The dictionary gains 23 anaphoric forms (a pesar de esto, gracias a ello) and now has 273 connectors.
    • find_connectors(..., breaks=) takes the positions of the marks, and find_connector_highlights(..., text=) the text, so the layer of the highlighting follows the same rules as CohesionStats.
  • Fast import. import ests takes 0.08 s instead of 0.89 s and loads no spaCy, pandas, matplotlib or SciPy. ests.visualizers and ests.corpus load their modules on first use, and the spaCy components load when they are first used or found by their entry points.
  • Clichés of the component. nlp.add_pipe("ests_style", config={"cliches": [...]}) counts the clichés of a list.
  • Presets of a subclass. A subclass of ReadabilityStats may narrow the presets, and a preset it adds is read by the scales of general (preset_level_scales, preset_reading_ease_grades).
  • Documentation. Every page has metric cards and signature blocks, with the errors of every class and function. The examples are run in English and in Spanish.

Incompatible changes

The public classes and functions keep their names. Their signatures change only by new optional parameters, except the name that a component class built directly needs.

  • Dependencies.
    • anyTS 0.3.9 or newer within 0.3 is required.
    • spaCy is bounded below 4: the model es_core_news_sm 3.8.0 bounds it only in its meta.json.
  • Names removed from ests.constants. Their values live in anyts.constants:
    • ALLITERATION_MIN_WORD_LEN, ALLITERATION_THRESHOLD;
    • CLAUSE_DEPS, SUBORDINATE_CLAUSE_DEPS, SUBJECT_DEPS, NOUN_MODIFIER_DEPS;
    • PHON_STATS_DESC, PHON_WINDOW_LEN;
    • CONTENT_UD_POS, NAUSEA_TOP_N;
    • FREQUENCY_BANDS, which ests.lexical_stats still re-exports.
  • Other names.
    • ests.phon_stats.PHON_STATS_DESC is gone: use PhonStats.stats_desc.
    • ests.cohesion_stats has no ConnectorIndex and split_doc_sents (the core has split_doc_units). Import CONNECTOR_CLASSES and CONNECTOR_TYPES from ests.constants.
    • is_finite_verb and is_infinitive come from anyts.utils, not from ests.syntax_stats. AUXILIARY_DEPS there is now a frozenset.
    • ests.utils.BLANK_LINE is gone.
    • ests.lexical_stats.is_number is the function of the core, is_number(word, pattern). The Spanish pattern is ests.extractors.NUMBER_PATTERN.
  • Components.
    • The component classes leave ests.__all__, so from ests import * no longer brings them. ests.StyleStatsComponent and the factories work as before.
    • A component class built directly needs its name: LexicalStatsComponent(nlp, "lexical"). add_pipe is unaffected.
    • The preset of ests_readability is None by default, the preset of the class; a saved config has null.
    • A wrong parameter raises the error of the core at add_pipe. A wrong seseo or data_dir or an unknown key still raises the ConfigValidationError of spaCy.
  • Syntax.
    • An infinitive clause under a verb no longer counts in subordinate_clauses_per_sent and p_complex_sents; it has its own key.
    • An infinitive in a gerund or participial phrase belongs to that phrase and heads no clause.
    • An infinitive that the model tags as csubj of its periphrasis (llega a convertirse) heads no clause.
  • Connectors.
    • A rule of a connector reads no word across a punctuation mark.
    • A phrase of CONNECTOR_SPLIT_BY_MARKS with a mark inside is no connector.
    • An anaphoric form counts as a connector of its own text in c_connectors.
  • Windows. split_windows cuts consecutive windows of exactly window words. The rest is a window only with at least min_words words, a whole window by default; min_words above window raises ParameterError. Before, the windows were equal parts of about window words.
  • Messages and tables.
    • The errors have the texts of the core, with hints (The data source has no parts of speech: parse the text with a model…).
    • print_stats prints the table of the core, with the columns fitted to the text, in every class but MorphStats.
    • ests.constants.CORE_MESSAGES is gone.
  • Checks.
    • seseo of VerseStats must be a bool.
    • So must the switches of BasicStats, WordsExtractor, CharNgramsExtractor and kwic.
    • The finders of the highlighting, find_phrases, sentence_rhythm, is_parenthetical and noun_lemmas check their arguments. SourceTypeError or ParameterError replaces a bare TypeError or AttributeError.
  • Readability attributes. ReadabilityStats.level_scales has the key flesch_reading_easy, and reading_ease_grades is the scale of general. The results of the methods are the same.
  • StyleStats.verbal_nouns is a property instead of a cached_property.

Numbers

On a fixed set of texts, 137 of 4620 results changed against 0.7.0, all in these categories. Everything else is bitwise equal.

  • New keys. infinitive_clauses_per_sent in every SyntaxStats, and connectors_inter and connectors_intra in every CohesionStats, with their components.
  • Clauses. The clause keys of SyntaxStats move in the texts with infinitive clauses: the subordinate clauses only fall. On 1050 fragments of SpanishLiterature:
    • the subordinate clauses fall from 43,989 to 39,875;
    • 3,766 infinitive clauses are counted apart;
    • the clauses go from 89,293 to 89,223. The infinitives in gerund and participial phrases no longer count, while a coordinated nominal predicate (y estaba cansado) and an infinitive under a periphrasis attached as conj now do.
  • Connectors. A few texts gain a connector after a comma (sin gracia alguna, y el amor). On the same fragments the connectors go from 36,050 to 36,092: 64 more by the marks, 22 fewer split by a mark. 11.8% of them link a sentence to the ones before.
  • Chains of de. A punctuation mark is no link of a chain: one text goes from 11 chains to 10.
  • Windows, as above: split_windows, corpus_features, compare_features and compare_corpora.
  • Error texts, of the core.

Fixes

  • Pickles.
    • SyntaxStats keeps no Doc of the user: a pickle of 30k characters takes 141 KB instead of 885 KB.
    • LexicalStats, StyleStats, SyntaxStats and CohesionStats built with nlp= leave the pipeline out of a pickle and a copy. A LexicalStats took 17 MB.
  • Speed. VerseStats is linear in the length of a line: 0.03 s instead of 0.41 s on a line of 8,000 words.
  • Extractors. StyleStats and PhonStats of a Doc with a words_extractor read the words with the extractor, as the other classes do.
  • expand_phrases gives its keys in one order whatever the hash seed.
  • is_agentless of a word that is no passive form gives False.
  • Demo.
    • It started again.
    • It cuts a long run of punctuation marks, on which the tokenizer of spaCy hung for minutes.

Quality

  • 2073 tests pass with 100% line and branch coverage.

  • ruff and mypy are clean.

  • CI runs on Python 3.11–3.14, on Linux, Windows and macOS, and with the lowest versions of the dependencies.

  • A QA pass before the release covered:

    • the rules of 0.8 on 1050 fragments of SpanishLiterature and 1500 sonnets;
    • 2218 outcomes of the ten classes on 29 sources, and a fuzzing of 17 entry points;
    • the import time, the lowest and the newest dependencies, Python 3.14, the notebooks and the demo;
    • the documentation in English and in Spanish, with all their examples run.

    Its findings are fixed in anyTS 0.3.9 and in this release, except one: the non-conditional si (¡Si ya te lo dije!) still counts as a conditional connector, until a hook of the core in 0.9.

Links

Assets 2
Loading