Skip to content

bitermplus: v1.0.0

Latest

Choose a tag to compare

@maximtrp maximtrp released this 27 Aug 19:12

Results are identical to 0.12.0 for a fixed seed. Models pickled with earlier versions still load.

Breaking changes

BTM.transform() no longer stores its result. It returns the matrix and leaves the model unchanged. These four properties are removed from BTM; accessing them raises AttributeError naming the replacement:

Removed from BTM Replacement
matrix_docs_topics_ p_zd = model.transform(docs)
matrix_topics_docs_ model.transform(docs).T
labels_ model.transform(docs).argmax(axis=1)
perplexity_ btm.perplexity(model.matrix_topics_words_, p_zd, X, model.topics_num_)
# Before                          # After
model.transform(docs_vec)         p_zd = model.transform(docs_vec)
p = model.perplexity_             p = btm.perplexity(
                                      model.matrix_topics_words_, p_zd, X, 8)

BTMClassifier keeps all four — see New features. coherence_ is unchanged on both classes.

BTMClassifier.__init__ no longer validates parameters. Invalid values now raise from fit(). Required by scikit-learn's estimator contract; fixes clone, get_params, set_params, GridSearchCV and pipelines.

entropy() raises ValueError when T == 1. Rényi entropy is undefined there: q = 1/T = 1 zeroes the F/(q-1) denominator. Previously returned an arbitrary value.

Deprecated

  • entropy(max_probs=False) now warns. It returns 0.0 for any row-normalised p_wz, so it carries no information.

New features

  • BTMClassifier.perplexity(X=None) — scores any document set; defaults to the training documents. Mirrors LatentDirichletAllocation.perplexity(X).
  • BTMClassifier.labels_ and .matrix_docs_topics_ — fitted attributes describing the training documents, set in fit and stable across later transform calls. perplexity_ is now training perplexity rather than "last transform".
  • get_biterms(..., as_array=True) — returns (n, 2) int32 arrays instead of nested Python lists. BTM.fit accepts either form; BTMClassifier uses it automatically. Default return type unchanged.
  • benchmarks/bench.py — timing, peak-memory and result-fingerprint harness (--label, --compare).

Bug fixes

  • transform([]) raised ValueError: Invalid shape in axis 0: 0; now returns an empty (0, n_topics) array.
  • BTM accepted SciPy sparse arrays (csr_array) via issparse() then failed with BufferError — sparse arrays sum to 1-D, sparse matrices to a 2-D np.matrix.
  • get_top_topic_words() / get_top_topic_docs() raised ValueError: No objects to concatenate for an empty topics_idx; now return an empty DataFrame. Non-positive words_num / docs_num are rejected.
  • Reading BTMClassifier.perplexity_ re-ran transform and overwrote model state, changing what labels_ and matrix_docs_topics_ returned afterwards.

Performance

12,295 documents, 4,703 words, 20 topics, 1.26M biterms, 100 iterations (benchmarks/bench.py):

Before After
get_biterms(as_array=True) 0.73 s / 179 MB 0.17 s / 12 MB 4.3x, 15x less memory
Biterm-to-array conversion 0.199 s 0.016 s 12x
coherence 0.014 s 0.007 s 1.9x
transform (sum_b) 0.122 s 0.098 s 1.25x
fit (Gibbs sampling) 16.2 s 15.1 s 1.08x

fit is memory- and FP-latency-bound; caching division reciprocals and a word-major count matrix were both measured and reverted for showing no gain. The remainder comes from hoisting loop invariants and building with -O3 (no -ffast-math).

Documentation

  • entropy() now states the Koltcov (2018) formulas it implements and is tested against an independent implementation of them.
  • Regression tests added for every bug above.