README.md

Obtaining LM data

./install-examples.sh

This downloads the data used in this tutorial.

Training example

Train an LSTM LM using a class-factor softmax:

./train_rnnlm -s -t ../rnnlm/ptb-mikolov/train.txt -d ../rnnlm/ptb-mikolov/valid.txt \
     -c ../rnnlm/ptb-mikolov/clusters-mkcls.txt -D 0.3 --hidden_size 256 --eta_decay_onset_epoch 10 --eta_decay_rate 0.5

Train an LSTM LM with a standard softmax:

./train_rnnlm -s -t ../rnnlm/ptb-mikolov/train.txt -d ../rnnlm/ptb-mikolov/valid.txt \
     -D 0.3 --hidden_size 256 --eta_decay_onset_epoch 10 --eta_decay_rate 0.5

Evaluation example

Evaluate a trained model:

./train_rnnlm -t ../rnnlm/ptb-mikolov/train.txt -c ../rnnlm/ptb-mikolov/clusters-mkcls.txt \
     -m lm_0.3_2_128_256-pid7865.params --hidden_size 256 -p ../rnnlm/ptb-mikolov/test.txt

PTB Baselines

Model dev test
5-gram KN 188.0 178.9
2x128, dropout=0.3, class-factored softmax 164.4 157.7
2x256, dropout=0.3, CFSM, decay 0.5@>10 129.7 125.4