Unofficial implementation of Adaptive Input in PyTorch
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
AdapInp.py Add files via upload Nov 10, 2018
LICENSE Initial commit Nov 10, 2018
README.md

README.md

Adaptive Input

Unofficial implementation of Adaptive Input in PyTorch

Combined with Adaptive Softmax, Adaptive Input drastically decreases the number of parameters of your model.

It is also known that Adaptive I/O results in lower ppl in rare words, since the traditional I/O layers underfit to infrequent tokens.

I/O tying is not supported yet, though it's quite simple to do that.

For more details, please refer to the paper: Adaptive Input Representations for Neural Language Modeling by Alexei Baevski, Michael Auli https://arxiv.org/abs/1809.10853.

Your can import Adaptive Softmax as AdaptiveLogSoftmaxWithLoss from torch.nn.modules.adaptive.

For using adaptive I/O, you need to preprocess your text (or more generally sequence) dataset, so that the number assigned to each token is in the order such that the more frequently occuring token is assigned lower number.