Skip to content
master
Go to file
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
bwb
 
 
ptb
 
 
 
 
 
 
 
 
 
 

README.md

Introduction

This codebase supports replication of the language modeling results in Recurrent Additive Networks (Kenton Lee, Omer Levy, and Luke Zettlemoyer).

Recurrent Additive Networks

The TensorFlow implementation of Recurrent Additive Networks (RAN) is found in ran.py and is used by the experiments in the subdirectories.

Experiments

Penn Treebank:

The word-level language modeling for Penn Treebank is found under the ptb directory. This code is derived from https://github.com/tensorflow/models/tree/master/tutorials/rnn/ptb.

Data preparation

  • curl -O http://www.fit.vutbr.cz/~imikolov/rnnlm/simple-examples.tgz
  • mkdir data
  • tar -xzvf simple-examples.tgz -C data

Train and Evaluate

  • python -m ptb.ptb_word_lm --data_path=data/simple-examples/data --model=tanh_medium

Replace tanh_medium with the desired setting.

Billion-word Benchmark:

The word-level language modeling for the billion-word benchmark is found under the bwb directory. This code is derived from https://github.com/rafaljozefowicz/lm.

Data preparation

  • curl -O http://www.statmt.org/lm-benchmark/1-billion-word-language-modeling-benchmark-r13output.tar.gz
  • mkdir data
  • tar -xzvf 1-billion-word-language-modeling-benchmark-r13output.tar.gz -C data
  • curl -o data/1-billion-word-language-modeling-benchmark-r13output/1b_word_vocab.txt https://raw.githubusercontent.com/rafaljozefowicz/lm/master/1b_word_vocab.txt

Train

  • CUDA_VISIBLE_DEVICES=0,1 python -m bwb.single_lm_train --logdir logs --num_gpus 2 --hpconfig num_shards=2 --datadir data/1-billion-word-language-modeling-benchmark-r13output

Evaluate

  • CUDA_VISIBLE_DEVICES= python -m bwb.single_lm_train --logdir logs --mode eval_test_ave --hpconfig num_shards=2 --datadir data/1-billion-word-language-modeling-benchmark-r13output

Text8:

The character-level language modeling for Text8 is found under the text directory. This code is derived from https://github.com/julian121266/RecurrentHighwayNetworks

Data Preparation

  • curl -O http://mattmahoney.net/dc/text8.zip
  • mkdir data
  • unzip text8.zip -d data

Train and Evaluate

  • python -m text8.char_train

About

No description, website, or topics provided.

Resources

Releases

No releases published

Packages

No packages published

Languages

You can’t perform that action at this time.