Dynamic evaluation of Transformer-XL
This code applies dynamic evaluation to pretrained Transformer-XL models from this paper. Our codebase is a modified version of their codebase. We used this code to obtain state of the art results on WikiText-103 (perplexity: 16.4), enwik8 (bits/char: 0.94), and text8 (bits/char: 1.04).
- Python 2.7
- Tensorflow, these results used version 1.13.1, but similar versions should work as well
Obtain and dynamically evaluate pretrained SoTA models
1. Download preprocessed data (vocab) & pretrained models
(a) cd to the
(b) Download the models with
2. Run dynamic evaluation scripts to replicate SoTA results on GPUs
Wiki-Text 103: run
bash sota/wt103.shwith the default parameters. Takes < 20 minutes. Should get a perplexity of 16.4.
bash sota/enwik8.shwith the default parameters. Takes a few hours. Should get a bpc of 0.940.
bash sota/text8.shwith the default parameters. Takes a few hours. Should get a bpc of 1.038.
By default, scripts set CUDA_VISABLE_DEVICES=0 . This contains run to a single gpu.
Not currently set up to run on multiple threads. This easiest way to do this would probably be to divide the test set up into chunks and dynamically evaluate separate runs of
dynamiceval_tf.py, and combine results.
SGD dynamic evaluation can be applied by setting
--rms=Falsein the arguments of dynamiceval_tf.py . This will require a higher learning rate to get best results.
If you are doing hyper parameter tuning for dynamic evaluation, be sure to do this on the validation set. Use the validation set by setting
--eval_split=validin the arguments of dynamiceval_tf.py.
ratioof dynamiceval_tf.py argument allows you to only use a subset of (1/ratio) of the evaluation set. This can speed up hyper-parameter tuning.
Setting the learning rate to 0 is equivalent to static/normal evaluation.