Skip to content
master
Switch branches/tags
Go to file
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
fig
 
 
 
 
 
 
 
 
 
 

README.md

A topic model for the debates of the 18th German Bundestag as a showcase example

Markus Konrad markus.konrad@wzb.eu, May 2018

Important note: If you want to git clone this project, you need to install git lfs first.

Overview

For a workshop on practical topic modeling, I created this topic model as a showcase example that demostrates the steps that are necessary to take in order to arrive at a usable, informative model:

  1. Preprocessing the raw data (preproc_raw.py)
  2. Generating the document-term-matrix from the data (generate_tokens.py)
  3. Evaluating topic models for a set of hyperparameters (tm_eval.py and tm_eval_plot.py)
  4. Generating the final model using the best combination of hyperparameters (generate_model.py)
  5. Visualizing, interpreting and analysing the model (report1.ipynb, report2.ipynb and example_analyses.py) – note that this was not the focus of the workshop and hence only exemplary analyses are given

Used software packages

This example uses Python 2.7 because of some dependency issues (namely the pattern package for better lemmatization of German texts does not support Python 3).

These are the main software packages in use:

  • tmtoolkit for evaluating models in parallel, calculating some model statistics and visualizations
  • lda for topic modeling with LDA using Gibbs sampling
  • PyLDAVis and Jupyter Notebooks for interactive visualizations

All software dependencies can be installed via pip install -r requirements.txt.

Data

The data for the debates comes from offenesparlament.de.

License

Licensed under Apache License 2.0 (except for the data from offenesparlament.de which have their own license). See LICENSE file.

About

An example topic model for debates from the 18th German Bundestag

Topics

Resources

License

Releases

No releases published

Packages

No packages published