nGram-Python

nGram: New Word Recognition (Chinese news texts).

This repo contains dataCollector (which crawls data from Sina news website), dataCleaner (which tidies the crawled data and aggregates them), nGram (which counts n-gram words and produce nGram data), and wordRecognition (which uses the nGram data to recognize chinese words).

Code Constitution Specifics

dataCollector

Spiders to crawl Sina news texts (with titles), using Python's Requests + BeautifulSoup.

Data source: From daily news list "http://news.sina.com.cn/old1000/news1000_YYYYMMDD.shtml", where YYYY, MM, DD are year, month, and day respectively.

dataCleaner

Python script to aggregate the crawled texts in a year to a single file, e.g. 2006_all.

nGram

Scan texts to generate statistical nGram files.

wordRecognition

Use nGram data to recognize words.

Data Specifics

The raw data is acquired by dataCollector and classified by date. The data contain Sina news texts with labels like <p></p>, and the Cinese-text-only result is acquired by dataCleaner as data.txt (using texts of 2006 January - April, totally 1.9GB approximately, 1,990,660,672 bytes precisely).

nGram.py utility

nGram.py is a command line utility. Usage:

$ src/nGram/nGram.py
nGram.py: count nGram words in Chinese Texts
Usage:
         ./nGram.py inputFileName nMin nMax [outputFileName]
Explanation:
         inputFileName -- Name of the input data text
         nMin, nMax -- the range of N for n-gram
         outputFileName -- (optional) Name of the output nGram text
Example:
         ./nGram.py data.txt 2 5
$