Corpus preprocessing
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Type Name Latest commit message Commit time
Failed to load latest commit information.
CMakeLists.txt Fix minimum CMake version per FindICU comments Sep 10, 2018
FindICU.cmake Added support for CMake 2.6 adding FindICU.cmake file to the project Sep 10, 2018


Pipelines for preprocessing corpora.

Compile with cmake:

mkdir build
cd build
cmake ..
make -j4

Paths are relative to the build directory.

bin/ $language $lower

does all the tokenization and normalization for normal text that has already been extracted and sentence split.


takes Gigaword XML files on stdin and outputs text with P tags intended to be used as input to the sentence splitter. Also removes or normalizes many ad-hoc parenthesized expressions like (UNDERLINE) and consecutive duplicate lines.

moses/ems/support/split-sentences.perl -l $language

is the Moses/Europarl sentence splitter with a bugfix to also split sentences separated by two spaces.

bin/ $language

preserves existing line breaks and introduces additional breaks when multiple sentences appear in the same line. This is useful when you want to use the target side of parallel corpora for language modeling.

bin/ $language

combines the unwrap and sentence split steps.


deduplicates text at the line level.

bin/shard $prefix $shard_count

Shards stdin into multiple files named prefix0 prefix1 prefix2 etc. This is useful when the deduper above runs out of memory.

bin/remove_long_lines $length_limit

removes lines longer than the specified length in bytes. The default is 2000 bytes.


removes lines with invalid UTF-8


Removes lines that contain bad UTF8; contain control characters; have less than 90% Latin, Common, or Inherited characters (except angle brackets); or have less than 50% Latin characters. I used this for giga-fren.

bin/process_unicode -l $language [--flatten] [--normalize] [--lower]

Processes UTF8.

  • --lower lowercases
  • --normalize applies the ICU normalization function
  • --flatten applies a bunch of substitutions for punctuation
bin/heuristics.perl -l $language

A collection of substitution heuristics from various people.

moses/tokenizer/tokenizer.perl -l $language

The Moses tokenizer.

bin/truecase --model $model

is a fast reimplementation of the Moses truecase.perl script. It does not support factors.

xzcat $language.*.raw.xz |commoncrawl_dedupe /dev/null |xz >$language.deduped.xz

Process the CommonCrawl n-grams raw files into the deduped files:

  • Remove lines beginning with df6fa1abb58549287111ba8d776733e9 (these mark document boundaries)
  • Strip leading and trailing whitespace
  • Deduplicate, preserving the first instance of the line
  • Remove any lines with invalid UTF-8