Skip to content

mallette 0.2.0

Choose a tag to compare

@dmberry dmberry released this 02 Nov 15:31
· 2 commits to main since this release
  • Diagnostics, as MALLET's --diagnostics-file (TopicModelDiagnostics): model.diagnostics(n) gives, for each topic, its tokens, document entropy, word length, coherence, distance from uniform and from the corpus, effective number of words, token–document difference, rank-1 documents, allocation ratio and count, and exclusivity, and for each of its top words the measures defined word by word; model.diagnosticsXML(n) writes MALLET's file. The same file, byte for byte, on every corpus and setting checked.
  • Inference for new documents (TopicInferencer, infer-topics): model.inferencer(seed).infer(docs, { iterations, thinning, burnin }), and docTopics(r) for MALLET's output. The documents are imported as the model's were; words the model does not hold are passed over. The same proportions as MALLET's for the same seed, on MALLET's sample and on 500 documents of Webster's 1913 the model was not trained on.
  • Documents as tokens: importDocs takes { name, tokens } as well as text, for a bench that makes its own tokens (lower-cased and stopwords removed as text's are).
  • A model as data: model.toJSON() and mallette.inferencerFrom(data, seed).
  • A Web Worker: mallette.trainInWorker(docs, k, o), with progress and cancel, loading src/mallette-worker.js; it returns the result, the model as data, the topic keys and the diagnostics.
  • A test suite, npm test, with no Java: MALLET's own outputs for its sample (test/expected/) compared byte for byte, and the rest checked; test/worker.html for the Worker. test/compare.js now compares the diagnostics file and inferred topics too (--infer for other documents), and refuses a seed that MALLET's command line cannot take (a negative one is read as missing).
  • The README's count of word types in MALLET's sample corrected: 893, not 891.