Status: Archive (code is provided as-is, no updates expected)

Code for the Neural GPU model originally described in [[http://arxiv.org/abs/1511.08228]].

Running experiments

Running one instance

The following would use 256 filters to train on binary multiplication, then 4-ary, then decimal:

python neural_gpu_trainer.py --nmaps=256 --task=bmul,qmul,mul --progressive_curriculum=5

My typical invocation is something like

  CUDA_VISIBLE_DEVICES=0 python neural_gpu_trainer.py --random_seed=0 --max_steps=200000 --forward_max=201 --nmaps=256 --task=bmul,qmul,mul --time_till_eval=4 --progressive_curriculum=5 --train_dir=../logs/August-12-curriculum/forward_max=201-nmaps=256-task=bmul,qmul,mul-progressive_curriculum=5-random_seed=0

The tests on decimal carry were done using invocations like the following:

  CUDA_VISIBLE_DEVICES=0 neural_gpu_trainer.py --train_dir=../logs/run1 --random_seed=1 --max_steps=100000 --forward_max=201 --nmaps=128 --task=add --time_till_eval=4 --time_till_ckpt=1

You can find a list of options, and their default values, in neuralgpu/trainer.py.

Examining results

Loading and examining a model

examples/examples_for_loading_model.py gives a simple instance of loading a model and running it on an instance.

Plotting results

Something like python plots/get_pretty_score.py cachedlogs/*/*task=bmul,qmul,mul-* works. There are a lot of options to make it prettier (renaming stuff, removing some runs, changing titles, reordering, etc.). For example, one of my plots was made with

python get_pretty_score.py cachedlogs/A*/*256*[=,]mul-* --titles '256 filters|' --title 'Decimal multiplication is easier with curriculum' --task mul --remove_strings='|-progressive_curriculum=5' --exclude='layer|progressive' --order '4,2,1,3' --global-legend=1

Requirements

• TensorFlow (see tensorflow.org for how to install)
• Matplotlib for Python (sudo apt-get install python-matplotlib)
• joblib

Credits

Original code by Lukasz Kaiser (lukaszkaiser). Modified by Eric Price (ecprice)