Hybrid Self-Attention Network for Machine Translation

[zomux]

Main Authors / Organization

Kaitao Song, Xu Tan, Furong Peng, Jianfeng Lu

Nanjing University of Science and Technology
Microsoft Research
Institute of Big Data Science and Industry, Shanxi University

PDF link

https://arxiv.org/pdf/1811.00253.pdf

Hypothesis

Adding relative positional information to self-attention improves Transformer.

Approach

Using directional mask and local mask then fuse the representations together in the self-attention mechanism.

Main Experimental Result

WMT14 De-En Translation task: + 0.4 BLEU
WMT17 Ch-En Translation task: +1.0 BLEU