lda.Rmd

```{r, include=FALSE}
library(knitr)
opts_chunk$set(fig.path = "figures_lda/")
```

Latent Dirichlet Allocation
========================================================

Topic modelling techniques such as Latent Dirichlet Allocation (LDA) can be a usefull tool for social scientists to analyze large amounts of natural language data. Algorithms for LDA are available in R, for instance in the `topicmodels` package. In this howto we demonstrate several function in the `corpustools` package that facilitate the use of LDA using the `topicmodels` package.

As a starting point we use a Document Term Matrix (dtm) in the `DocumentTermMatrix` format offered in the `tm` package. Note that we also offer a howto for creating the dtm. 

```{r}
library(corpustools)
data(sotu) # state of the union speeches by Barack Obama and George H. Bush.
head(sotu.tokens)
dtm = dtm.create(documents=sotu.tokens$aid, terms=sotu.tokens$lemma, filter=sotu.tokens$pos1 %in%  c('N','M','A'))
dtm
```

Not all terms are equally informative of the underlying semantic structures of texts, and some terms are rather useless for this purpose. For interpretation and computational purposes it is worthwhile to delete some of the less usefull words from the dtm before fitting the LDA model.  As seen from the red message lines above, the dtm.create automatically uses some filtering of terms, but it can be good to customize this for your research.

Now we are ready to fit the model! We made a wrapper called `lda.fit` for the `LDA` function in the `topicmodels` package. This wrapper doesn't do anything interesting, except for deleting empty columns/rows from the dtm, which can occur after filtering out words. 

The main input for `topmod.lda.fit` is:
- the document term matrix
- K: the number of topics (this has to be defined a priori)
- Optionally, it can be usefull to increase the number of iterations. This takes more time, but increases performance.

```{r}
set.seed(12345)
m = lda.fit(dtm, K=20, num.iterations=1000)
terms(m, 10)[,1:5] # show first 5 topics, with ten top words per topic
```

We now have a fitted lda model. The `terms` function shows the most prominent words for each topic (we only selected the first 5 topics for convenience). 

One of the thing we can do with the LDA topics, is analyze how much attention they get over time, and how much they are used by different sources (e.g., people, newspapers, organizations). To do so, we need to match this article metadata. We can order the metadata to the documents in the LDA model by matching it to the documents slot.

```{r}
head(sotu.meta)
meta = sotu.meta[match(m@documents, sotu.meta$id),]
```

We can now do some plotting. First, we can make a wordcloud for a more fancy (and actually quite informative and intuitive) representation of the top words of a topic.

```{r, message=F, warning=FALSE, fig.width=6, fig.height=6}
lda.plot.wordcloud(m, topic_nr=1)
lda.plot.wordcloud(m, topic_nr=2)
```

With `lda.plot.time` and `lda.plot.category`, we can plot the salience of the topic over time and for a given categorical variable.

```{r}
lda.plot.time(m, 1, meta$date, date_interval='year')
lda.plot.category(m, 1, meta$headline)
```

It can be usefull to print all this information together. That is what the following function does.

```{r, message=F, warning=FALSE, fig.width=10, fig.height=10}
lda.plot.topic(m, 1, meta$date, meta$headline, date_interval='year')
lda.plot.topic(m, 2, meta$date, meta$headline, date_interval='year')
```

For further substantive analysis, we can also create a data frame containing the topic proportion for each document:

```{r}
docs = topics.per.document(m)
docs = merge(meta, docs)
head(docs)
```