Functions for calculating Open Vocabulary lexical statistics.
See lexFrequencyPipeline API documentation below. Also see
index.test.js
for a recreation of the
WWBP example.
// import your lexicon data as a JSON object (i.e., Record<string, number>)
import lexicon from "./lexicon.json" with { type: "json" }; // example
// import the lexFrequencyPipeline function from this module
import { lexFrequencyPipeline } from "lex-helpers";
// define your intercept
const intercept = 10.523; // example
// create your custom pipeline
const pipeline = lexFrequencyPipeline(lexicon, intercept);
// get some tokens
const doc1Tokens = ["the", "cat", "sat", "on", "the", "mat"]; // example
const doc2Tokens = ["the", "dog", "sat", "on", "the", "hat"]; // example
// run the pipeline
const doc1result = pipeline(doc1tokens); // {number}
const doc2result = pipeline(doc2tokens); // {number}
Corrects IEEE 754 floating point errors using toFixed
.
Example:
import { correctFloat } from "lex-helpers";
const initial = 0.1 + 0.2; // 0.30000000000000004
const corrected = correctFloat(initial); // 0.3
Get the frequencies of tokens in a corpus.
Example:
import { getFrequencies } from "lex-helpers";
const tokens = ["the", "cat", "sat", "on", "the", "mat"];
const frequencies = getFrequencies(tokens); // Map<{ the: 2, cat: 1, sat: 1, on: 1, mat: 1 }>
Get the weighted relative frequencies of tokens in a corpus.
Example:
import { getWeightedRelativeFrequencies } from "lex-helpers";
const lexicon = { the: -93, cat: 100, sat: 50, on: -10, mat: 5 };
const frequencies = { the: 2, cat: 1, sat: 1, on: 1, mat: 1 };
const weightedRelativeFrequencies = getWeightedRelativeFrequencies(
lexicon,
frequencies,
); // IterableIterator<[string, number]>
Get the final value of a token in a corpus.
Example:
import { getLexiconValue } from "lex-helpers";
const weightedRelativeFrequencies = {
the: 0.4,
cat: 0.1,
sat: 0.1,
on: 0.1,
mat: 0.1,
};
const intercept = 10.523;
const lexiconValue = getLexiconValue(weightedRelativeFrequencies, intercept); // {number}
Create a custom pipeline for calculating lexical statistics.
N.B. This is provided for simple use cases. For larger datasets it is recommended that you create your own pipeline using the functions provided in this module.
The pipeline is
getFrequencies -> getWeightedRelativeFrequencies -> getLexiconValue -> correctFloat
Example:
import { lexFrequencyPipeline } from "lex-helpers";
const lexicon = { the: 0.2, cat: 0.1, sat: 0.1, on: 0.1, mat: 0.1 };
const intercept = 10.523;
const pipeline = lexFrequencyPipeline(lexicon, intercept);
const tokens = ["the", "cat", "sat", "on", "the", "mat"];
const result = pipeline(tokens); // 12.523
The same as lexFrequencyPipeline
but getFrequencies simply counts a token as present (1) or absent (0), instead of the token's frequency.
Sum and correct the values of an object.
Example:
import { sumValues } from "lex-helpers";
const values = [{ value: 0.1 }, { value: 0.2 }, { value: 0.3 }];
const sum = sumValues(values); // 0.6
(C) 2017-24 P. Hughes. All rights reserved.
Released under the MIT licence.