[WIP]
This is a PyTorch implementation of Neural Arithmetic Logic Units by Andrew Trask, Felix Hill, Scott Reed, Jack Rae, Chris Dyer and Phil Blunsom.
from models import *
# single layer modules
NAC(in_features, out_features)
NALU(in_features, out_features)
# stacked layers
MultiLayerNAC(num_layers, in_dim, hidden_dim, out_dim)
MultiLayerNALU(num_layers, in_dim, hidden_dim, out_dim)
To reproduce "Numerical Extrapolation Failures in Neural Networks" (Section 1.1), run:
python failures.py
This should generate the following plot:
To reproduce "Simple Function Learning Tasks" (Section 4.1), run:
python function_learning.py
This should generate a text file called interpolation.txt
with the following results. (Currently only supports interpolation, I'm working on the rest)
Relu6 | None | NAC | NALU | |
---|---|---|---|---|
a + b | 0.002 | 0.004 | 0.001 | 0.017 |
a - b | 0.046 | 0.005 | 0.000 | 0.003 |
a * b | 83.012 | 0.444 | 5.218 | 5.218 |
a / b | 106.441 | 0.338 | 2.096 | 2.096 |
a ^ 2 | 94.103 | 0.630 | 3.871 | 0.196 |