Skip to content

v1.5.0 - Saturation Trainer

Choose a tag to compare

View all tags
@leafo leafo released this 28 May 06:31
· 7 commits to master since this release
v1.5.0
This tag was signed with the committer’s verified signature.
leafo leaf corcoran
GPG key ID: 542601E3888790A7
Verified
Learn about vigilant mode.
d8a7f18

https://luarocks.org/modules/leafo/lapis-bayes

Added

  • SaturationTrainer (lapis.bayes.trainers.saturation): The first trainer class that selects candidate tokens based on the categories they will be trained for. Saturation trainer attempts to avoid over-training tokens by skipping tokens that pass the confidence threshold. Can be configured with saturation_threshold, min_observations, train_novel, and train_opposite.
  • lapis.bayes.tokenizers.util: shared helpers for cleaning up tokens, including sanitize_token, sanitize_tokens, and normalize_url_text.
  • stopwords list added to SpamTokenizer, unigram tokens that are in the stopwords list are now dropped by default.
  • lapis.bayes.text.utf8 now also exposes the variation_selector, zero_width_joiner, and flag_tag_sequence patterns.

Changed

  • Word selection in candidate_words now ranks words by how strongly each word is tied to a category using a chi-squared scoring system. Ties are broken alphabetically for deterministic output.
  • Removing invisible characters no longer breaks emoji and other multi part characters. Emoji built from several joined parts, flag emoji, keycap characters, and accented or combined characters are kept whole, while stray invisible characters are still removed.
  • SpamTokenizer, PostgresTextTokenizer, and UrlDomainsTokenizer now clean up their tokens at the end, removing invisible characters and dropping anything left empty. Tokens should never have mixed in invisible Unicode characters now.
  • URLs are now decoded before they are read, so links that hide their address with percent encoding are still picked up.

Full Changelog: v1.4.0...v1.5.0

Assets 4
Loading