This is a python reimplementation of the maximum-likelihood estimation approach proposed by Anand Venkataraman for unsupervised word segmentation.

The paper describing the approach can be found at the link here

As of 10/17/18, this repo only implements backoffs for unigrams and phonemes. To run, clone the repo:

git clone https://github.com/yczeng/venkataraman-approach.git

cd into the directory, and run:

python search.py

The results of the segmentation will be stored in results/result.txt.

Performance is measured using precision, recall, and F-score.

Precision is the number of correct words found out of all words found, recall is the number of correct words found out of all correct words, and F-scores it he geometric average of pricision and recall (2 * precision * recall) / (precision + recall).

To score a segmented lexicon, run score.py [segmented lexicon] [model lexicon] where segmented lexicon and model lexicon are text files containing a dictionary of words in the lexicon. An example command is:

python score.py results/lexicon.txt data/Bernstein-Ratner87-lexicon

It looks like results are:

Precision: 0.526275115919629
Recall: 0.5155185465556397
f1 Score: 0.5208413001912047