Flattening the Gigaword Datset
The scripts in this repository dump the text of the Gigaword dataset into a single file, for use with language modeling (and other!) toolkits.
See my blog post on flattening the Gigaword corpus for more information about how the code in this repo works.
Table of Contents
To run this code, you must have GNU Parallel. This can be installed on Ubuntu with:
sudo apt-get install parallel
This project was developed in Python 3.6, but should work with Python 3.x and 2.x. Please raise an issue if you find that this is not the case.
Conda will set up a virtual environment with the exact version of Python used for development along with all the dependencies needed to run the code in this package.
Create a conda environment with Python 3.6.
conda create -n flat python=3.6
Now activate the conda environment.
source activate flat
Install the required dependencies with
pip install -r requirements.txt
Install the required SpaCy data pack.
python -m spacy download en
flatten_one_gigaword.py takes in the path of a Gigaword data file
and an output directory to write a flattened version to. The bash script at
flatten_all_gigaword.sh is a thin wrapper that feeds the paths of all the
Gigaword data files to
flatten_one_gigaword.py and combines the final output.
flatten_all_gigaword.sh takes in three positional arguments:
The path to the Gigaword directory, with all of the data files unzipped.
A directory to write the flattened files to and the final combined output. It will be created if it does not exist.
The number of files to process at once.
For example, you can run:
./flatten_all_gigaword.sh ./data/gigaword_eng_5/ tmp/ 24
to extract data (in parallel, processing 24 files at a time) from the Gigaword corpus
./data/gigaword_eng_5/ and write the flattened files + combined output to