Skip to content
No description, website, or topics provided.
Branch: master
Clone or download
ben
Latest commit 9a57df5 Apr 12, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
tf first commit Apr 12, 2019
LICENSE first commit Apr 12, 2019
README.md first commit Apr 12, 2019
getdata.sh first commit Apr 12, 2019
prep_text8.py first commit Apr 12, 2019

README.md

Dynamic evaluation of Transformer-XL

Introduction

This code applies dynamic evaluation to pretrained Transformer-XL models from this paper. Our codebase is a modified version of their codebase. We used this code to obtain state of the art results on WikiText-103 (perplexity: 16.4), enwik8 (bits/char: 0.94), and text8 (bits/char: 1.04).

Requirements

  • Python 2.7
  • Tensorflow, these results used version 1.13.1, but similar versions should work as well

Obtain and dynamically evaluate pretrained SoTA models

1. Download preprocessed data (vocab) & pretrained models

(a) cd to the tf folder

(b) Download the models with bash sota/download.sh

2. Run dynamic evaluation scripts to replicate SoTA results on GPUs

  • Wiki-Text 103: run bash sota/wt103.sh with the default parameters. Takes < 20 minutes. Should get a perplexity of 16.4.

  • enwik8: run bash sota/enwik8.sh with the default parameters. Takes a few hours. Should get a bpc of 0.940.

  • text8: run bash sota/text8.sh with the default parameters. Takes a few hours. Should get a bpc of 1.038.

Other Notes:

  • By default, scripts set CUDA_VISABLE_DEVICES=0 . This contains run to a single gpu.

  • Not currently set up to run on multiple threads. This easiest way to do this would probably be to divide the test set up into chunks and dynamically evaluate separate runs of dynamiceval_tf.py, and combine results.

  • SGD dynamic evaluation can be applied by setting --rms=False in the arguments of dynamiceval_tf.py . This will require a higher learning rate to get best results.

  • If you are doing hyper parameter tuning for dynamic evaluation, be sure to do this on the validation set. Use the validation set by setting --eval_split=valid in the arguments of dynamiceval_tf.py.

  • The ratio of dynamiceval_tf.py argument allows you to only use a subset of (1/ratio) of the evaluation set. This can speed up hyper-parameter tuning.

  • Setting the learning rate to 0 is equivalent to static/normal evaluation.

You can’t perform that action at this time.