Why this tool?
Parallel RNN trainer implementes Two stage class RNNs and parallel RNNs proposed in the following paper
Z. H. Huang, G. Zweig, M. Levit, B. Dumoulin, B. Oguz and S. Chang, Accelerating Recurrent Neural Network Training via Two Stage Classes and Parallelization, in Automatic Speech Recognition and Understanding (ASRU), 2013.
Two stage class RNNs uses two stage classes (super classes and classes) as opposed to one class. Parallel RNN trainer splits the training data into batches and then dispatchs jobs to multiple CPUs/nodes for slave models training. Two stage class RNNs and parallel RNNs not only result in equal or lower WERs compared to original RNNs but also accelerate training by 2 and 10 times respectively. Code is developed based on RNNLM 0.3e (Tomas Mikolov). The following changes are made
Separate Vocab part to a class
Add two stage class (super class and class) to speed up training. Two options to generate super classes: even or frequency based.
new Maxent feature hash function
Explicit RNN constructors from random initialization or from model file
Submit HPC jobs to train slave RNN models
Master model update after done with HPC slave RNN model training
To build, run
build.sh to generate the binary at
RNNParaExp/readme.txt for experiments.
Please send your questions/comments to email@example.com