Transformer-tensorflow

Yet another tensorflow implementation of Attention Is All You Need

Usage

First you need to download IWSLT 2016 German–English parallel corpus dataset as below

./data/IWSLT16
├── IWSLT16.TED.dev2010.de-en.de.xml
├── IWSLT16.TED.dev2010.de-en.en.xml
├── IWSLT16.TED.tst2010.de-en.de.xml
├── IWSLT16.TED.tst2010.de-en.en.xml
├── IWSLT16.TED.tst2011.de-en.de.xml
├── IWSLT16.TED.tst2011.de-en.en.xml
├── IWSLT16.TED.tst2012.de-en.de.xml
├── IWSLT16.TED.tst2012.de-en.en.xml
├── IWSLT16.TED.tst2013.de-en.de.xml
├── IWSLT16.TED.tst2013.de-en.en.xml
├── IWSLT16.TED.tst2014.de-en.de.xml
├── IWSLT16.TED.tst2014.de-en.en.xml
├── README
├── train.tags.de-en.de
└── train.tags.de-en.en

And then,

# Pre-process data
$ python preprocess.py --dataset IWSLT16
# Train IWSLT16
$ python train.py --dataset IWSLT16

TO-DO

• BLEU Score (priority: high)
• Test Accuracy (priority: high)
• update learning rate as paper (priority: low)

Training Loss

References

• Kyubyong's transformer
• DongjunLee's transformer