v1.4.0 - Spam Tokenizer

https://luarocks.org/modules/leafo/lapis-bayes

This release adds a multiclass classifier, a spam-oriented tokenizer, a character n-gram tokenizer for language detection, and a collection of text utility modules. The two-class BayesClassifier has new options for per-token weighting and uncertainty weighting, and its log-mode probability calculation has been rewritten.

Added

• BayesMultiClassifier (lapis.bayes.classifiers.bayes_multi): a multiclass naive Bayes classifier supporting more than two categories at once. Uses Laplace smoothing with softmax-normalized scores.
• SpamTokenizer (lapis.bayes.tokenizers.spam): a tokenizer aimed at spam classification. See the dedicated section below.
• NgramTokenizer (lapis.bayes.tokenizers.ngram): character n-gram tokenizer, useful for tasks like language detection.
• BaseTokenizer (lapis.bayes.tokenizers.base): shared base class for tokenizers.
• Text utility modules under lapis.bayes.text.*:
  • unaccent: strip accents and normalize confusable Unicode characters using a large mapping table
  • stem: Porter stemmer (port of the Snowball English algorithm)
  • punycode: RFC 3492 Punycode encoding for internationalized domain names
  • utf8: UTF-8 codepoint helpers and Lua patterns for ranges like zero-width characters
• Added train_text method to BaseClassifier that bundles tokenization and training. The top-level lapis.bayes.train_text now delegates to this method and accepts a filter_tokens option to post-process tokens before they are written.
• Added token_weight_patterns option to BayesClassifier: a pattern → weight table that boosts or de-emphasizes matching tokens during scoring.
• Added uncertainty_weight / unclassified_counts options to BayesClassifier: down-weight tokens that also appear frequently in an unlabeled reference corpus, so common-but-uninformative tokens contribute less to the classification.
• Added ignore_domains option to UrlDomainsTokenizer.

SpamTokenizer

Spam content typically tries to evade naive bag-of-words detection: tokens are obfuscated with look-alike Unicode and zero-width characters, URLs are hidden behind userinfo or homoglyph hostnames, the same words and links repeat throughout the document, and the surrounding language may not be English. SpamTokenizer is built specifically for this domain.

At a high level, it processes text in these stages:

1. HTML extraction. Markup is stripped via web_sanitize so HTML-embedded text is not broken into garbage tokens.
2. Unicode normalization. Zero-width characters are stripped globally, and (by default) accented and confusable characters are folded to ASCII using the bundled unaccent table. Optional Porter stemming and optional CJK character splitting are available.
3. Structured extraction. URLs, emails, currencies, percentages, repeated punctuation, and invalid UTF-8 bytes are emitted as their own tagged tokens (domain:example.com, email_user:bob, currency:$, punct:!3, etc.). Hostnames are punycoded and emitted both as the full domain and as each .tld / .sub.tld suffix, so an unseen subdomain still matches a known parent.
4. Filtering. Words are lowercased and length-bounded (with truncation rather than dropping for over-length tokens), with ignore_words, ignore_tokens, and ignore_domains knobs to suppress noise.
5. Sampling. Optional bigrams can be emitted, and sample_at_most caps the token count with dithered ordering so the same input does not always produce the same N tokens.

Changed

• Rewrote the log-mode probability calculation in BayesClassifier. The math now matches the multiplicative form exactly, and a max_log_sum shift is applied so log-mode stays numerically stable when scoring many tokens.
• lapis.bayes.text_probabilities and lapis.bayes.classify_text now forward opts through to the underlying classifier, so per-call options work without instantiating the classifier manually.
• UrlDomainsTokenizer now handles user@host phishing patterns (extracts the real host rather than the userinfo), and lifts domain extraction earlier in the pipeline so domains are not lost when sampling.
• Categories:increment_words sorts inserts by word before the upsert, giving a deterministic per-row lock order and preventing deadlocks under concurrent training.
• The top-level lapis.bayes module no longer eagerly requires the default classifier (or lapis.db). The default classifier is loaded lazily inside each entrypoint.
• Bumped minimum lapis dependency to >= 1.16.0.

Removed

• Removed BayesModClassifier (lapis.bayes.classifiers.bayes_mod) and WeightedClassifier (lapis.bayes.classifiers.weighted). Neither was usable in practice.

Full Changelog: v1.3.0...v1.4.0