Neural Machine Translation with Fairseq

This repository contains workflows for training Neural Machine Translation (NMT) models using Fairseq. It covers data preparation, preprocessing, model training, evaluation, and backtranslation to improve translation quality.

---

Setup

• Clone and install Fairseq
• Ensure your environment has GPU support
• Install dependencies (PyTorch, SacreBLEU, etc.)
• Configure Google Drive mounting if working in Colab

---

Data Preparation

1. Parallel Corpora

   • Use official WMT parallel datasets (e.g., WMT18 English–German).
   • Preprocess with fairseq-preprocess using a shared dictionary.
   • Save artifacts into data-bin/.

2. Scripts Management

   • Custom scripts (e.g., prepare-wmt18en2de.sh, sacrebleu.sh) are copied between Google Drive and the working directory to ensure persistence.

---

Training

• Train Transformer-based models (transformer_wmt_en_de) using parallel corpora.

• Key hyperparameters:

  • Label smoothing: 0.1
  • Adam optimizer (betas=(0.9, 0.98))
  • Inverse square root learning rate scheduler
  • Max tokens: 3584
  • Gradient accumulation (--update-freq 16)

• Mixed precision training (--fp16) for efficiency.

• Training progress and checkpoints are stored in checkpoints_* directories.

---

Evaluation

• Translation quality is evaluated with SacreBLEU.
• Example evaluation datasets: WMT17, WMT08/Europarl.
• BLEU scores are logged for model comparison.

---

Backtranslation

Backtranslation is used to leverage monolingual data to improve translation quality.

1. Prepare Parallel Data

• Preprocess WMT18 En–De dataset.
• Train an intermediate De→En model on the parallel corpus.
• Evaluate the intermediate model with SacreBLEU.

2. Prepare Monolingual Data

• Collect large-scale German-only monolingual data.
• Preprocess the data in shards for efficiency (shard00–shard24).
• Align dictionaries with the parallel dataset.

3. Generate Synthetic Parallel Data

• Use the trained De→En model to backtranslate German monolingual data into English.

• Two strategies:

  • Beam search (--beam 5) → high-quality translations.
  • Top-k sampling (--sampling-topk 10) → diverse translations.

• Save results in backtranslation_output/.

4. Extract and Combine Data

• Use extract_bt_data.py to filter synthetic sentence pairs.

• Combine synthetic parallel data with original WMT18 parallel corpus:

  • wmt18_en_de_para_plus_bt_beam
  • wmt18_en_de_para_plus_bt_top10

5. Train with Backtranslated Data

• Train En→De models on the combined dataset.

• Two training settings:

  • Beam backtranslation model (checkpoints_en_de_parallel_bt_beam)
  • Top-10 sampling backtranslation model (checkpoints_en_de_parallel_bt_top10)

• Upsampling ensures the parallel dataset is not overwhelmed by synthetic data (--upsample-primary 1).

6. Evaluate

• Evaluate trained models with SacreBLEU.
• Compare parallel-only vs parallel+BT models.
• Expect improved BLEU scores due to the additional synthetic training data.

---

Checkpoints & Outputs

• checkpoints_de_en_parallel → Intermediate De→En model
• backtranslation_output_beam / backtranslation_output_top10 → Synthetic translations
• checkpoints_en_de_parallel_bt_* → Final En→De models trained with backtranslation

---

References

• Fairseq Documentation
• SacreBLEU
• Edunov et al. (2018). Understanding Back-Translation at Scale.