Skip to content
/ sundae Public

Unofficial PyTorch implementation of "Step-unrolled Denoising Autoencoders for Text Generation"

License

Notifications You must be signed in to change notification settings

vvvm23/sundae

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

16 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Step-unrolled Denoising Autoencoders for Text Generation

Unofficial PyTorch implementation of "Step-unrolled Denoising Autoencoders for Text Generation"

Work-in-progress, more experiments soon.

Currently implements unconditional character-level text generation on the text8 dataset.

Requires x-transformers and ptpt to be installed.

Usage

Run SUNDAE training using config file cfg.toml:

python main.py --cfg-path cfg.toml

Sample text from trained model saved at ckpt.pt:

python main.py --cfg-path cfg.toml --resume ckpt.pt --sample --nb-samples 16

Other useful flags:

--seed          # set RNG seed 
--no-save       # disable saving of checkpoints [False]
--no-cuda       # disable the use of CUDA device [False]
--no-amp        # disable the use of automatic mixed precision [False]
--nb-workers    # set number of dataloader workers. [4]

TODO:

  • Add argmax-unrolled sampling
  • Encoder-decoder translation experiments
  • Pre-trained checkpoints

Citation

Step-unrolled Denoising Autoencoders for Text Generation

Nikolay Savinov, Junyoung Chung, Mikolaj Binkowski, Erich Elsen, Aaron van den Oord

@misc{savinov2021stepunrolled,
      title={Step-unrolled Denoising Autoencoders for Text Generation}, 
      author={Nikolay Savinov and Junyoung Chung and Mikolaj Binkowski and Erich Elsen and Aaron van den Oord},
      year={2021},
      eprint={2112.06749},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

About

Unofficial PyTorch implementation of "Step-unrolled Denoising Autoencoders for Text Generation"

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages