Gated-Convolutional-Networks

Language Modeling with Gated Convolutional Networks Yann N. Dauphin, Angela Fan, Michael Auli, David Grangier https://arxiv.org/abs/1612.08083

This is a PyTorch implementation of Facebook AI Research Lab's paper: Language Modeling with Gated Convolutional Networks. This paper applies a convolutional approach to language modelling with a novel Gated-CNN model.

Architecture

Requirements

• Download Google 1 Billion Word dataset.
• PyTorch 0.2.0_3

Reference

• https://github.com/anantzoid/Language-Modeling-GatedCNN

TODO

• impl adaptive softmax
• bug? accuracy didn't increase from epoch#1
• support variable sentence length using padding
• Understand padding at the beginning of sentence
• impl checkpoints