A toolkit for producing n-gram language models. The highlights are the implementation of Kneser-Ney growing and revised Kneser pruning methods.
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
app
cmake
python-wrapper
src
stored_away
tests
.gitignore
CMakeLists.txt
acknowledgements.txt
commands.html
install
licence.txt
possible_optimizations.txt
readme

readme

VariKN language modeling toolkit provides tools for training n-gram
language models. Amongst the supported methods are:

- Absolute discounting
- Kneser-Ney smoothing
- Revised Kneser pruning
- Kneser-Ney growing

The descriptions of Revised Kneser pruning and Kneser-Ney growing can
be found in the paper Vesa Siivola, Teemu Hirsimäki and Sami Virpioja,
"On Growing and Pruning Kneser-Ney Smoothed N-Gram Models", IEEE
Transactions on Speech, Audio and Language Processing,
15(5):1617-1624, 2007 (available at
https://sites.google.com/site/vesassiivola/publications/TASLP2007.pdf).

The package provides a accurate pruning for Kneser-Ney smoothed
models. Also, it is possible to train a very high-order n-gram models
with the growing algorithm. The models can be output to arpa lm
format, which is compatible with most common other tools in the
field. You can look at the paper Vesa Siivola, Mathias Creutz and
Mikko Kurimo: "Morfessor and VariKN machine learning tools for speech
and language technology", Proceedings of the 8th International
Conference on Speech Communication and Technology (INTERSPEECH'07),
2007 (available at
https://sites.google.com/site/vesassiivola/publications/is2007less.pdf)
for guidelines on typical use.

For help on installing the toolkit, read the file named "install".

For help on using the built commands, read the file named "commands.html"

If it looks like there is interest in using the toolkit but the
documentation is inadequate, let me know the main sticking points and
I'll try to do something about it. If you are interested in a well
supported toolkit that provides a wide range of functionality, you
might want to look at the srilm toolkit
(http://www-speech.sri.com/projects/srilm/).