Skip to content
Hierarchical Dirichlet processes. Topic models where the data determine the number of topics. This implements Gibbs sampling.
Branch: master
Clone or download
Latest commit 94bdf82 Feb 22, 2017
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
hdp-faster Minor cleanup Feb 1, 2017
hdp add include statement Feb 20, 2017
.gitignore Minor cleanup Feb 1, 2017
LICENSE Initial commit Mar 29, 2015
Makefile Minor cleanup Feb 1, 2017
README.md initial commit Mar 29, 2015

README.md

Hierarchical Dirichlet Process (with Split-Merge Operations)


(C) Copyright 2010, Chong Wang and David Blei. Written by Chong Wang.

This is a C++ implementation of hierarchical Dirichlet process for topic modeling.

README

NB: The split-merge algorithm is preliminary. Note that this code requires the Gnu Scientific Library, http://www.gnu.org/software/gsl/


TABLE OF CONTENTS

A. COMPILING

B. POSTERIOR INFERENCE

C. INFERENCE ON NEW DATA

D. PARAMETER SETTINGS

E. PRINTING TOPICS


A. COMPILING

Type "make" in a shell. Make sure the GSL is installed. You may need to change the Makefile a bit.

B. POSTERIOR INFERENCE

The following shows an example of performing posterior inference on a set of documents,

hdp --algorithm train --data data --directory train_dir

Data format

--data points to a file where each line is of the form (the LDA-C format):

 [M] [term_1]:[count] [term_2]:[count] ...  [term_N]:[count]

where [M] is the number of unique terms in the document, and the [count] associated with each term is how many times that term appeared in the document.

The sampler will produce some files in the --directory,

*-topics.dat: the word counts for each topic, with each line as a topic

*-word-assignments.dat: print each word's assignment to the topic and the table, which is in R-friendly format, d w z t

d: document id w: word id z: topic index t: table index (only for document level. If you only analyze the topics, this is irrelevant.)

*.bin: the binary model file used for inference on new data.

state.log: various information to monitor the Markov chain.

More parameter settings, run: hdp --help

Note: some parameters for split-merge are hand coded at the beginning of hdp.cpp file.


C. INFERENCE ON NEW DATA

To perform inference on a different set of data (in the same format as before), run:

hdp --algorithm test --data data --saved_model saved_model --directory test_dir

where --saved_model is the binary file from the posterior inference on training data.

The sampler will produce some files in the --directory,

test-*-topics.dat: the word counts for each topic, with each line as a topic

test*-word-assignments.dat: print each word's assignment to the topic and the table, which is in R-friendly format.

test.log: various information to monitor the Markov chain.

test-*.bin: the binary model file used for inference on newer data.

More parameter settings, run: hdp --help


D. PARAMETER SETTINGS

The meaning of the parameters is the same as in the in the following paper

Y. Teh, M. Jordan, M. Beal, and D. Blei. Hierarchical Dirichlet processes. Journal of the American Statistical Association, 2006. 101[476]:1566-1581


E. PRINTING TOPICS

A R script (print.topics.R) is included to print topics. Make sure it is executable. (chmod +x print.topics.R) For example,

print.topics.R mode-topics.dat vocab.dat topics.dat 10

will produce a topic list with top 10 words selected. For help, run,

print.topics.R

You can’t perform that action at this time.