Dynamic evaluation of Transformer-XL
Introduction
This code applies dynamic evaluation to pretrained Transformer-XL models from this paper. Our codebase is a modified version of their codebase. We used this code to obtain state of the art results on WikiText-103 (perplexity: 16.4), enwik8 (bits/char: 0.94), and text8 (bits/char: 1.04).
Requirements
- Python 2.7
- Tensorflow, these results used version 1.13.1, but similar versions should work as well
Obtain and dynamically evaluate pretrained SoTA models
1. Download preprocessed data (vocab) & pretrained models
(a) cd to the tf
folder
(b) Download the models with bash sota/download.sh
2. Run dynamic evaluation scripts to replicate SoTA results on GPUs
-
Wiki-Text 103: run
bash sota/wt103.sh
with the default parameters. Takes < 20 minutes. Should get a perplexity of 16.4. -
enwik8: run
bash sota/enwik8.sh
with the default parameters. Takes a few hours. Should get a bpc of 0.940. -
text8: run
bash sota/text8.sh
with the default parameters. Takes a few hours. Should get a bpc of 1.038.
Other Notes:
-
By default, scripts set CUDA_VISABLE_DEVICES=0 . This contains run to a single gpu.
-
Not currently set up to run on multiple threads. This easiest way to do this would probably be to divide the test set up into chunks and dynamically evaluate separate runs of
dynamiceval_tf.py
, and combine results. -
SGD dynamic evaluation can be applied by setting
--rms=False
in the arguments of dynamiceval_tf.py . This will require a higher learning rate to get best results. -
If you are doing hyper parameter tuning for dynamic evaluation, be sure to do this on the validation set. Use the validation set by setting
--eval_split=valid
in the arguments of dynamiceval_tf.py. -
The
ratio
of dynamiceval_tf.py argument allows you to only use a subset of (1/ratio) of the evaluation set. This can speed up hyper-parameter tuning. -
Setting the learning rate to 0 is equivalent to static/normal evaluation.