Repository navigation
Results are identical to 0.12.0 for a fixed seed. Models pickled with earlier versions still load.
Breaking changes
BTM.transform() no longer stores its result. It returns the matrix and leaves the model unchanged. These four properties are removed from BTM; accessing them raises AttributeError naming the replacement:
Removed from BTM |
Replacement |
|---|---|
matrix_docs_topics_ |
p_zd = model.transform(docs) |
matrix_topics_docs_ |
model.transform(docs).T |
labels_ |
model.transform(docs).argmax(axis=1) |
perplexity_ |
btm.perplexity(model.matrix_topics_words_, p_zd, X, model.topics_num_) |
# Before # After
model.transform(docs_vec) p_zd = model.transform(docs_vec)
p = model.perplexity_ p = btm.perplexity(
model.matrix_topics_words_, p_zd, X, 8)BTMClassifier keeps all four — see New features. coherence_ is unchanged on both classes.
BTMClassifier.__init__ no longer validates parameters. Invalid values now raise from fit(). Required by scikit-learn's estimator contract; fixes clone, get_params, set_params, GridSearchCV and pipelines.
entropy() raises ValueError when T == 1. Rényi entropy is undefined there: q = 1/T = 1 zeroes the F/(q-1) denominator. Previously returned an arbitrary value.
Deprecated
entropy(max_probs=False)now warns. It returns0.0for any row-normalisedp_wz, so it carries no information.
New features
BTMClassifier.perplexity(X=None)— scores any document set; defaults to the training documents. MirrorsLatentDirichletAllocation.perplexity(X).BTMClassifier.labels_and.matrix_docs_topics_— fitted attributes describing the training documents, set infitand stable across latertransformcalls.perplexity_is now training perplexity rather than "last transform".get_biterms(..., as_array=True)— returns(n, 2)int32 arrays instead of nested Python lists.BTM.fitaccepts either form;BTMClassifieruses it automatically. Default return type unchanged.benchmarks/bench.py— timing, peak-memory and result-fingerprint harness (--label,--compare).
Bug fixes
transform([])raisedValueError: Invalid shape in axis 0: 0; now returns an empty(0, n_topics)array.BTMaccepted SciPy sparse arrays (csr_array) viaissparse()then failed withBufferError— sparse arrays sum to 1-D, sparse matrices to a 2-Dnp.matrix.get_top_topic_words()/get_top_topic_docs()raisedValueError: No objects to concatenatefor an emptytopics_idx; now return an emptyDataFrame. Non-positivewords_num/docs_numare rejected.- Reading
BTMClassifier.perplexity_re-rantransformand overwrote model state, changing whatlabels_andmatrix_docs_topics_returned afterwards.
Performance
12,295 documents, 4,703 words, 20 topics, 1.26M biterms, 100 iterations (benchmarks/bench.py):
| Before | After | ||
|---|---|---|---|
get_biterms(as_array=True) |
0.73 s / 179 MB | 0.17 s / 12 MB | 4.3x, 15x less memory |
| Biterm-to-array conversion | 0.199 s | 0.016 s | 12x |
coherence |
0.014 s | 0.007 s | 1.9x |
transform (sum_b) |
0.122 s | 0.098 s | 1.25x |
fit (Gibbs sampling) |
16.2 s | 15.1 s | 1.08x |
fit is memory- and FP-latency-bound; caching division reciprocals and a word-major count matrix were both measured and reverted for showing no gain. The remainder comes from hoisting loop invariants and building with -O3 (no -ffast-math).
Documentation
entropy()now states the Koltcov (2018) formulas it implements and is tested against an independent implementation of them.- Regression tests added for every bug above.