Alternative approach for Adaptive Computation Time in TensorFlow
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
data
hparams
models
results
train_utils
.gitignore
LICENSE
README.md
__init__.py
requirements.txt
train.py

README.md

Adaptive Computation Time

Alternative approach on Adaptive Computation Time for Recurrent Neural Networks by Alex Graves in TensorFlow.

Contributor: Berkay Antmen, Divyam Madaan, Bryan M. Li, Aidan N. Gomez

Idea

We use a small trick to help with the training of ACT modules. The idea is to penalize pondering less when the model is incorrect and more when it is correct.

In effect, relaxing the objective constraints when the model is struggling and then asking for computation efficiency only once the model has solved the problem. We implement this simply by scaling the ponder cost term by the inverse of the task loss. See function calculate_ponder_cost()

To use new ponder cost, set flag --use_new_ponder_cost=True

Running this Code

Call pip install -r requirements.txt to install all dependencies.

Training Data

Sample Call

The settings for the reader used is passed as flag. For example, to run the parity task, call:

python -m ACT.train --model=act --hparam_sets=act_parity  --output_dir=runs/act_parity --data=parity  --train_steps=100000

Results

Task Accuracy Sequence error rate Pondering
Parity parity_acc parity_ser parity_ponder
Sort sort_acc sort_ser sort_ponder
Addition add_acc add_ser add_ponder
  • #0d6306 Our ponder cost
  • #00e5ff LSTM cell
  • #001289 Original ponder cost

Results for parity, sort and addition tasks are displayed. Light blue color represents LSTM cell, dark blue represents ACT cell with old ponder cost and green represents ACT cell with our new ponder cost.

The model with the new ponder cost ponders comparatively more than the model with the original ponder cost. The ACT cell with new ponder cost performed better than the old ponder cost for parity task, though performed similarly for sort and addition tasks.

Contributing

We'd love to accept your contributions to this project. Please feel free to open an issue, or submit a pull request as necessary. If you have implementations of this repository in other ML frameworks, please reach out so we may highlight them here.

Acknowledgment

The ACT code is build upon DeNeutoy/act-tensorflow