Skip to content
master
Switch branches/tags
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Integrating Transformer and Paraphrase Rules for Sentence Simplification

Paper Link: http://www.aclweb.org/anthology/D18-1355

Note that some improvement from original EMNLP paper:

  • we modified the code to allow supporting subword and the model performs well.
  • we found replacing name entities might not be a good idea (i.e. replace John to person0) since it lose some information. Instead, subword is helpful for reducing the huge vocabulary coming from name entities.
  • we found the context(memory) addressing is probably redundant. Without it, the model can achieve same(even better) performance.

Data Download:

https://drive.google.com/open?id=132Jlza-16Ws1DJ7h4O89TyxJiFSFAPw7

Pretrained Model Download:

https://drive.google.com/open?id=16gO8cLXttGR64_xvLHgMwgJeB1DzT93N

Command to run the model:

python model/train.py -ngpus 1 -bsize 64 -fw transformer -out bertal_wkori_direct -op adagrad -lr 0.01 --mode transbert_ori -nh 8 -nhl 6 -nel 6 -ndl 6 -lc True -eval_freq 0 --fetch_mode tf_example_dataset --subword_vocab_size 0 --dmode wk --tie_embedding all --bert_mode bert_token:bertbase:init --environment aws --memory direct python model/eval.py -ngpus 1 -bsize 256 -fw transformer -out bertal_wkori_direct -op adagrad -lr 0.01 --mode transbert_ori -nh 8 -nhl 6 -nel 6 -ndl 6 -lc True -eval_freq 0 --subword_vocab_size 0 --dmode wk --tie_embedding all --bert_mode bert_token:bertbase:init --environment aws

Arugument instruction

  • bsize: batch size
  • out: the output folder will contains log, best model and result report
  • tie_embedding: all means tie the encoder/decoder/projection w embedding, we found it can speed up the training
  • bert_mode: the mode of using BERT bert_token indicates we use the subtoken vocabulary from BERT; bertbase indicates we use BERT base version (due to the memory issue, we did not try BERT large version yet)
  • environment: the path config of the experiment. Please change it in model/model_config.py to fit to your system

More config you can check them in util/arguments.py

Citation

Zhao, Sanqiang, et al. "Integrating Transformer and Paraphrase Rules for Sentence Simplification." Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. 2018.

@article{zhao2018integrating,
  title={Integrating Transformer and Paraphrase Rules for Sentence Simplification},
  author={Zhao, Sanqiang and Meng, Rui and He, Daqing and Andi, Saptono and Bambang, Parmanto},
  journal={arXiv preprint arXiv:1810.11193},
  year={2018}
}

About

Text Simplification Model based on Encoder-Decoder (includes Transformer and Seq2Seq) model.

Topics

Resources

Releases

No releases published

Packages

No packages published