Skip to content

RKorAPClient v1.4.0 (CRAN-Release)

Latest

Choose a tag to compare

@github-actions github-actions released this 09 Sep 15:35
· 3 commits to master since this release
  • cacheAs now consistently covers all query functionsfrequencyQuery(), corpusStats(), collocationScoreQuery() and textMetadata(). The cache file records its relevant production context: parameters, KorAP instance, index revision and package version. cacheAsInfo() prints that record, blessCacheAs() vouches for a file that is sound regardless, and withCachedResults() takes the files as they are for one expression, mode = "offline" refusing to compute anything that is not in one already

  • the names of a named vc vector are now consistently used as labels by frequencyQuery(), corpusStats() and collocationScoreQuery(), as collocationAnalysis() already did.
    Where a vector carries no names, nothing changes: no label column appears that was not there before

  • collocationAnalysis() now discards collocates that occur less often than expected by chance. logDice, by which it ranks and thresholds, expresses how salient a pair is rather than how surprising, so a frequent word could appear among the top collocates although the node does not attract it at all: for Grund in a 5+5 window, Berlin reaches a logDice of 3.84, close to triftiger at 3.96, while co-occurring 1.74 bits less often than expected. The new minObservedExpectedRatio parameter defaults to 1 and keeps such pairs out. Raise it to demand a stronger contrast, e.g. 2 for collocates occurring at least twice as often as expected, or set it to 0 for the unfiltered result of earlier versions, e.g. to study repulsion. collocationScoreQuery() is unaffected, as there the pairs to score are given explicitly

  • changed logDice values: logDice() is now computed as defined by Rychlý (2008), 14 + log2(2 * O / (O1 + O2)), so that its values are comparable to those of Sketch Engine and other tools. It previously multiplied the node frequency by the window size, 14 + log2(2 * O / (w * O1 + O2)), which added a count of window positions to a count of word tokens and made the coefficient asymmetric, so that swapping node and collocate changed the score. Because w * O1 dominated the denominator for a frequent node, rare collocates were penalised: for triftiger as a collocate of Grund in a 5+5 window, logDice was 0.64, below Berlin at 2.03, although Berlin co-occurs with Grund less often than chance predicts. The values are now 3.96 and 3.84. Scores rise by up to log2(w), that is by up to 3.32 for the default context of 5 left and 5 right, so collocationAnalysis() with the default thresholdScore = "logDice" and threshold = 2 is now somewhat more permissive when recursing. Results computed with a total window size of 1, as in the light verb construction example of the Readme, are unaffected. The other association scores are unchanged: they take the window size into account through the expected frequency E, which is correct

  • changed ll() values: the contingency table of ll() scaled only its row total by the window size, an inconsistency spotted by Tim Feldmüller, leaving cells that do not add up to one sample. Following Evert (2004), the sample consists of co-occurrence tokens, so the sample size and both marginals scale with the window: an occurrence of either word takes part in window_size pairs. The expected co-occurrence frequency is unchanged at window_size * O1 * O2 / N, which is why pmi, mi2 and mi3 are unaffected, but log-likelihood values differ, by 0.1% to well over 100% depending on the frequencies and the window. This also removes the case where N - window_size * O1 turned negative and the score became NaN, since window_size * (N - O1) cannot: the warning added earlier in this development version is therefore gone again, having treated a symptom of this

  • improved coverage of the doc-prompting tests, which now range from a frequency query over time to comparing collocates across virtual corpora, keeping a result in a cacheAs file and labelling corpora by name. They prompt current LLMs with the Readme and check the code written from it, so that a gap in the documentation shows up as a failing test. This guards the quality of the Readme and improves vibe coding results. The approach is briefly described in Kupietz et al. (2026)

  • fixed collocation analysis dropping snippets whose markup did not have one particular shape, which cost about 15% of the hits of a contains(<base/s=s>, ...) query: those are cut at the sentence boundary and carry a <span class="cutted"> inside the match, and a match filling the whole sentence leaves an empty context span. The two context spans are now read one by one and stripped of their markup, whatever it contains (#14). This only concerns servers that do not deliver tokenized matches, where collocation analysis falls back to parsing the KWIC markup

  • findExample() no longer aborts with "replacement has length zero" when a failed request leaves the query without any snippet to take an example from (#14)

  • snippet2FreqTable() and matches2FreqTable() work with their own default of an empty stopword list again, which used to leave the table without the column the stopwords are joined on

  • dropped the PTXQC dependency, which was imported for two small string functions (lcpCount() and lcsCount(), used by queryStringToLabel()) but pulled in rmzqc, jsonvalidate and V8, and with them the only dependency requiring a libv8 installation.