Skip to content

0.4.0

Choose a tag to compare

View all tags
@SergeyShk SergeyShk released this 25 Oct 15:42
· 32 commits to master since this release
0.4.0
ff57357
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

The fourth release of anyTS, the language-independent core of the text statistics libraries ruTS (Russian) and esTS (Spanish).

New in this release:

  • morphological statistics and readability formulas with their original coefficients;
  • new measures of syntax, cohesion and lexical diversity;
  • keywords by n-grams, rolling Delta, Zeta segments and three stylometric plots.

Requirements

  • Python 3.11 or newer.
  • spaCy 3.8.15 or newer within 3.
  • preshed 3.0.11 or newer, except 3.0.14 (new). The wheels of 3.0.14 break the tokenizer of spaCy (explosion/spaCy#14058).
  • numpy 1.26, pandas 2.1, scipy 1.11, matplotlib 3.8 and graphviz 0.20.3 or newer. No trained spaCy model is needed.
  • Rendering the word tree and the network of collocations also needs the executables of Graphviz.
pip install anyts

ruTS 0.18 and esTS 0.8 require anyTS 0.3 (anyts<0.4).

Incompatible changes

  • Messages (74 → 90 keys).
    • hook_result is removed, and check_messages refuses a dictionary that has it. The result of each hook has a key of its own instead: hook_doc, hook_lexical_words, hook_cohesion_words, hook_syllables, hook_sounds, hook_highlights and hook_morph_words.
    • New keys:
      • source_span;
      • connector_breaks, connector_breaks_short, connector_blocked, connector_blocked_empty and connector_type;
      • morph_features, morph_markers, morph_stats_desc and morph_values_desc.
    • source_type names the type, "The data source cannot be of type {type}". A Span gets source_span, "The data source cannot be a Span: pass span.as_doc()". anyts.utils.source_type_error gives either error for a class to raise.
    • layer_annotations lists the annotations after a colon, and the default HighlightedText.annotation_names are bare nouns ("parts of speech" instead of "the parts of speech").
    • The errors of ConnectorMatcher.find and is_subordinator take the messages of the class.
  • New keys of the statistics.
    • BASIC_STATS_DESC gets 2 keys, DIVERSITY_STATS_DESC 2, COHESION_STATS_DESC 6 and SYNTAX_STATS_DESC 14.
    • A subclass with the stats_desc of the core gets them in get_stats, print_stats and to_dataframe, and the widths of print_stats change.
    • The units of L2SCA, the prepositional chains and the similarity of sentences, and the overlaps with the next two sentences and of the paragraphs, are computed on first use. A subclass without these keys costs nearly as before.
  • CohesionWord has two more fields at the end, connector_blocked and paragraph_start. Unpacking it by position breaks.
  • Behaviour.
    • SyntaxStats raises SourceError (no_pos) for a Doc with a parse but without parts of speech, as CohesionStats and MorphStats do.
    • PhonStats.mean_syllable_len counts the sounds of the classes of a syllable, by its CV pattern.
    • Quotes and the apostrophe are no longer marks between words in find_marks, find_unit_marks and find_connector_highlights.
    • BLANK_LINE takes every line break of Unicode: \r\n, \r, U+0085 and U+2028 besides \n, and U+2029 alone. The zero-width space and the byte order mark count as whitespace on a blank line. This bounds the sentences of SentsExtractor, start_sents_after_blank_lines, kwic and the paragraphs. The pattern moves to anyts.utils, and anyts.extractors still exports it.
    • positive of keyness must be a bool.
    • Equal scores of keyness and collocations are ordered by frequency, the scores compared to 12 significant digits.
    • In compare_features, every feature draws its bootstrap from a generator of its own. The intervals of a feature no longer depend on the other columns, and they change once, for the same seed.
    • Juilland's D of dispersion and calc_juilland_d is cut at 0.
    • A SyntaxStats or CohesionStats of the core pickled by 0.3.9 loads, but its get_stats, print_stats and to_dataframe raise AttributeError: it lacks the new keys. Such objects need to be computed again. A subclass that names only the keys of 0.3.9 is not affected, nor are the other classes.

Features

  • Morphology: MorphStats with the analyze hook and MorphWord. It counts the parts of speech and the features of Universal Dependencies, word by word or by value, and gives the markers of a language library, explain_text and print_markers. The component is MorphStatsComponent.

  • Unadapted readability: UnadaptedReadabilityStats and eleven formulas with their original coefficients:

    • FORCAST, and Linsear Write in the form of textstat;
    • Powers-Sumner-Kearl and McAlpine EFLAW;
    • the Wiener Sachtextformel in its four variants, and Dickes-Steiwer;
    • Gulpease, Fucks, Strain, ELF and Wheeler-Smith.

    The component is UnadaptedReadabilityStatsComponent.

  • Syntax: 14 keys:

    • the words before the root;
    • the T-units and phrases of L2SCA: T/S, MLT, C/T, CT/T, VP/T, CP/C and CN/C;
    • the prepositional chains, with the is_prep_complement hook;
    • the share of nominal subjects and objects before the verb;
    • the similarity of adjacent sentences, by the edit distance of their parts of speech and by their dependency triples.

    Also calc_edit_distance, calc_multiset_similarity and dependency_triples.

  • Cohesion: the overlaps of nouns, arguments and content words with the next two sentences and between paragraphs, in the manner of TAACO, by calc_forward_overlap. Paragraphs start after a blank line, by CohesionWord.paragraph_start, and the class gives n_paragraphs.

  • Diversity: Orlov's Z (calc_orlov_z), burstiness (calc_burstiness, Kim and Jo 2016), and calc_mamtld(step=), faster by the step. BasicStats adds letters_entropy and syllables_entropy.

  • Corpus:

    • keyness(n=) gives keywords by n-grams of a list of texts;
    • collocations(side=) gives directed windows, and a node no longer slows it down;
    • rolling_delta with statistics=, and zeta_segments;
    • split_paragraphs in anyts.extractors: the numbers of words of its paragraphs give the sizes of the parts of dispersion.

  • Plots: rolling_delta_plot, zeta_scatter_plot and delta_histogram_plot.

  • Helpers:

    • source_type_error, find_paragraph_starts and find_unit_paragraph_starts in anyts.utils;
    • ConnectorMatcher.find(blocked=), CohesionWord.connector_blocked and find_connector_highlights(blocked=): the positions where no connector starts, by a rule of a language that reads more than the words and their parts of speech, such as the parse.

For the libraries

  • New sections of the docs to include (50).

    • components.md: MorphStatsComponent, UnadaptedReadabilityStatsComponent.
    • corpus/stylometry.md: rolling_delta, zeta_segments.
    • extractors/sentences.md: split_paragraphs.
    • stats/cohesion_stats.md: CohesionStats-forward_overlap, calc_forward_overlap.
    • stats/diversity_stats.md: calc_orlov_z, calc_burstiness.
    • stats/morph_stats.md: MorphStats, MorphStats-attributes, MorphStats-parameters, MorphStats-hooks, MorphStats-metrics, MorphStats-get_stats, MorphStats-print_stats, MorphStats-markers, MorphStats-explain_text, MorphWord.
    • stats/syntax_stats.md: SyntaxStats-words_before_root, SyntaxStats-l2sca, SyntaxStats-prep_chains, SyntaxStats-word_order, SyntaxStats-similarity, is_prep_complement, calc_edit_distance, calc_multiset_similarity, dependency_triples.
    • stats/unadapted_readability_stats.md: UnadaptedReadabilityStats, UnadaptedReadabilityStats-attributes, UnadaptedReadabilityStats-parameters, UnadaptedReadabilityStats-hooks, UnadaptedReadabilityStats-metrics, and the eleven calc_* of the formulas.
    • utils/messages.md: source_type_error.
    • utils/spacy.md: find_unit_paragraph_starts.
    • utils/words.md: find_paragraph_starts.
    • visualizers/stylometry.md: rolling_delta_plot, zeta_scatter_plot, delta_histogram_plot.

  • Anchors the tables of metrics link to. The sections DiversityStats-metrics, CohesionStats-metrics and SyntaxStats-metrics link to the headings of new cards, which stand above their sections, outside them. Without these anchors, mkdocs build --strict fails. Above each of these sections, a library writes a heading with the anchor:

    • #forward_overlap above CohesionStats-forward_overlap;
    • #orlov_z above calc_orlov_z, and #burstiness above calc_burstiness;
    • #words_before_root, #l2sca, #prep_chains, #word_order and #similarity above SyntaxStats-words_before_root, -l2sca, -prep_chains, -word_order and -similarity.

    In the same way, UnadaptedReadabilityStats-metrics links to the cards of its eleven formulas: each anchor is the name of the calc_* section below it without calc_ (#forcast above calc_forcast).

  • Paragraphs. An analyze of CohesionStats that builds its own words sets paragraph_start by find_paragraph_starts or find_unit_paragraph_starts. Without it, the keys *_para are nan.

  • Corpus. keyness of several texts takes the list of texts, not their words joined. A mapping of the frequencies of n-grams goes with n=1.

Quality

  • Tests: 2,390 tests with 100% line and branch coverage. Two tests of the docs check every parameter table against its signature and run the examples of 39 pages against their results.

  • CI: Python 3.11-3.14 on Linux, Windows and macOS, a job on the lowest versions of the dependencies, and a job for the examples of the docs.

  • Numbers of the libraries: through the classes of esTS 0.8 and ruTS 0.18, the keys of 0.3.9 give the same numbers bitwise. The exception is mean_syllable_len of ruTS, 2.369 → 2.333 on average over its texts. The snapshots of the libraries, 4,620 and 4,822 results, change only by the changes named above.

  • QA: a quality check before the release covered:

    • syntax, cohesion and morphology;
    • diversity, the basic statistics and readability;
    • the corpus, stylometry and the plots;
    • the move of the libraries;
    • the docs, the public API, the packaging and the speed.

    It fixed what it found (#95-#100).

Links

Assets 2
Loading