Name		Name	Last commit message	Last commit date
parent directory ..
README.md		README.md
__init__.py		__init__.py
edits.py		edits.py
errorify.py		errorify.py
helpers.py		helpers.py
preprocess_lang8.py		preprocess_lang8.py
preprocess_output_vocab.py		preprocess_output_vocab.py
preprocess_reading_lookup.py		preprocess_reading_lookup.py
preprocess_transformations.py		preprocess_transformations.py
preprocess_wiki.py		preprocess_wiki.py

README.md

Dataset Preprocessing Modules

utils.preprocess_wiki: Generates a TFRecordDataset from a Wikipedia dump extracted by WikiExtractor.
utils.preprocess_lang8: Generates a TFRecordDataset from the Lang8 corpus.
utils.edits: Module for edit-tagging parallel sentences.
utils.errorify: Module for generating synthetic errors in sentences.
utils.helpers: General common helper functions.