Skip to content

Code for the paper "Principled Paraphrase Generation with Parallel Corpora"

Notifications You must be signed in to change notification settings

aitorormazabal/paraphrasing-from-parallel

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

5 Commits
 
 
 
 
 
 

Repository files navigation

Principled Paraphrasing from Parallel Corpora

This repository contains the implementation of our paraphrasing model, as described in our paper.

Training

It is important that you use the version of fairseq included in this repo, as it contains minor changes to the optimizers that allow to pass the retain_graph flag to the backward function, which is necessary for the proper propagation of gradients as described in the paper.

To train a model, you first need to binarize your bilingual training data, using the mBART sentencepiece tokenizer as described in the mBART page. Instead of using the full pre-trained dictionary of 250K words for fairseq-preprocess, we create a new one of the most common 40K words. That is, your fairseq-preprocess call should look like this:

fairseq-preprocess \
  --source-lang en_XX \
  --target-lang fr_XX \
  --trainpref train.spm \
  --validpref valid.spm \
  --testpref test.spm \
  --destdir ${data_bin_dir} \
  --thresholdtgt 0 \
  --thresholdsrc 0 \
  --nwordssrc 40000 \
  --nwordstgt 40000 \
  --joined-dictionary \
  --workers 70 

Once you have binarized your data, you can call fairseq-trained, passing the fs_modules subdirectory as the user-dir. Here is how a call might look like:

user_dir=/path/to/fairseq/fs_modules/
mbart_dir=/path/to/mBART/mbart.cc25.v2/
fairseq-train  ${data_bin_dir} \
--arch transformer-dis-single \
--criterion label_smoothed_shared_disentanglement_loss \
--task translation-dis \
--load-mbart ${mbart_dir} \
--optimizer adam \
--user-dir ${user_dir} \
--encoder-embed-dim 1024 \
--encoder-ffn-embed-dim 4096 \
--decoder-embed-dim 1024 \
--decoder-ffn-embed-dim 4096 \
--encoder-embed-dim 1024 \
--encoder-ffn-embed-dim 4096 \
--ddp-backend=no_c10d \
--max-tokens 2048 \
--adam-betas '(0.9, 0.98)' \
--update-freq 8 \
--log-format json \
--log-interval 1 \
--lr 3e-5 \
--lr-scheduler polynomial_decay \
--total-num-update ${total_num_update} \
--save-interval-updates 10000 \
--warmup-updates 2500  \
--share-all-embeddings \
--fp16 \
--label-smoothing 0.2 \
--dropout 0.3 \
--attention-dropout 0.1 \
--skip-invalid-size-inputs-valid-test  \
--src-langtok '[en_XX]' \
--tgt-langtok '[fr_XX]' \
-s en_XX -t fr_XX \
--disc-chance ${disc_chance} \
--lmbda ${lambda} \
--save-dir ${save_dir} \
--max-update 100005

Where ${lambda} and ${disc_chance} correspond to $\lambda$ and $K$ in the paper, respectively.

Pretrained model

Alternatively, you can download our pretrained model for $\lambda=0.73$ and $K=0.7$ here.

Evaluation

Once the model is trained, you can run inference on it using the same task as for training, with an additional --inference-mode 1 parameter. An inference command might look like this:

fairseq-interactive ${data_bin_dir} \
--path $checkpoint \
--task translation-dis \
--source-lang en_XX \
--target-lang fr_XX \
--bpe sentencepiece \
--sentencepiece-model ${mbart_dir}/sentence.bpe.model \
--user-dir ${user_dir} \
--nbest 1 \
--beam 5 \
--inference-mode 1 \

Citing

If you use the code in this repository for your research, please cite our paper:

@inproceedings{artetxe2018acl,
author    = {Ormazabal, Aitor and Artetxe, Mikel and Soroa, Aitor and Labaka, Gorka and Agirre, Eneko},
title     = {Principled Paraphrase Generation with Parallel Corpora},
booktitle = {Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
year      = {2022},
pages     = {1621–1638}
}

About

Code for the paper "Principled Paraphrase Generation with Parallel Corpora"

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages