Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Python R Shell
Fetching latest commit…
Cannot retrieve the latest commit at this time.
|Failed to load latest commit information.|
ngrampy is a python class for manipulating google (or similarly formatted) n-gram data. It provides a python class for very basic table manipulations such that operations on tables are mimiced by operations on the hard drive, with huge n-gram files that cannot be read into RAM. This takes a lot of hard drive time, but can handle arbitrary file sizes (5~20gb is typical). This is *not* optimized for speed, since these things take a long time anyways and are typically run once. Usually, it makes more sense to process the google files once, concatinging and collapsing by some dates into a large file with all the ngrams (since this may take a few days). For this, the process-google.py script is fastest (much faster than LineFile). In collapsing dates, it makes a much smaller file (~10GB for eng-us 2grams) gzip -dc /home/piantado/Desktop/GoogleBooks/eng-us-all/2/* | python process-google.py /tmp/G-eng-us-all Or, unpigz is about 2x as fast as gzip on my computer (it multithreads fetching, file io, etc.) This perl script does not do any fancy filtering of the ngrams. To download data from google, you can use download.py NOTE: In general, you should use this library with export PYTHONIOENCODING=utf-8 so that you can handle utf-8 characters from google. NOTE: This splits columns in the text files by whitespace; if you want something else, you should merge with underscores or something NOTE: The pypy tends to run much faster than python for this! ======================================================== == LICENSE ======================================================== ngrampy is licensed under GPL 3.0 ======================================================== == INSTALLATION: ======================================================== Put this library somewhere--mine lives in /home/piantado/mit/Libraries/ngrampy/ Set the PYTHONPATH environment variable to point to ngrampy/: export PYTHONPATH=$PYTHONPATH:/home/piantado/Desktop/mit/Libraries/ngrampy You can put this into your .bashrc file to make it loaded automatically when you open a terminal. On ubuntu and most linux, this is: echo 'export PYTHONPATH=$PYTHONPATH:/home/piantado/Desktop/mit/Libraries/ngrampy' >> ~/.bashrc You can also do echo 'export PYTHONIOENCODING=utf-8' >> ~/.bashrc although this will change your default python encoding. And you should be ready to use the library