Skip to content


Switch branches/tags

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?

Latest commit


Git stats


Failed to load latest commit information.
Latest commit message
Commit time

Adaptive Computation Time

Alternative approach on Adaptive Computation Time for Recurrent Neural Networks by Alex Graves in TensorFlow.

Contributor: Berkay Antmen, Divyam Madaan, Bryan M. Li, Aidan N. Gomez


We use a small trick to help with the training of ACT modules. The idea is to penalize pondering less when the model is incorrect and more when it is correct.

In effect, relaxing the objective constraints when the model is struggling and then asking for computation efficiency only once the model has solved the problem. We implement this simply by scaling the ponder cost term by the inverse of the task loss. See function calculate_ponder_cost()

To use new ponder cost, set flag --use_new_ponder_cost=True

Running this Code

Call pip install -r requirements.txt to install all dependencies.

Training Data

Sample Call

The settings for the reader used is passed as flag. For example, to run the parity task, call:

python -m ACT.train --model=act --hparam_sets=act_parity  --output_dir=runs/act_parity --data=parity  --train_steps=100000


Task Accuracy Sequence error rate Pondering
Parity parity_acc parity_ser parity_ponder
Sort sort_acc sort_ser sort_ponder
Addition add_acc add_ser add_ponder
  • #0d6306 Our ponder cost
  • #00e5ff LSTM cell
  • #001289 Original ponder cost

Results for parity, sort and addition tasks are displayed. Light blue color represents LSTM cell, dark blue represents ACT cell with old ponder cost and green represents ACT cell with our new ponder cost.

The model with the new ponder cost ponders comparatively more than the model with the original ponder cost. The ACT cell with new ponder cost performed better than the old ponder cost for parity task, though performed similarly for sort and addition tasks.


We'd love to accept your contributions to this project. Please feel free to open an issue, or submit a pull request as necessary. If you have implementations of this repository in other ML frameworks, please reach out so we may highlight them here.


The ACT code is build upon DeNeutoy/act-tensorflow


No releases published


No packages published

Contributors 4