Skip to content
Simple word-level language ID using Viterbi based on unigram frequencies and character n-grams.
Python
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
word-level-language-id Add example scripts and documentation. Jun 22, 2017
.gitignore Initial commit Jun 22, 2017
LICENSE Initial commit Jun 22, 2017
README.md Add example scripts and documentation. Jun 22, 2017

README.md

Word-level language ID

Simple word-level language identification using the Viterbi algorithm based on unigram frequencies and character n-grams.

Usage

I recommend using Python 3 for better Unicode support.

To quickly try out the system, corpora and language models are already included for British English and Irish. See below how to add new ones. You might want to do some post-processing on the lexicons because e.g. the Irish one contains some English as well and vice versa.

Run word-level language ID on some example sentences:

python word-level-language-id/identify.py

Train new language models

Create or download a unigram frequency lexicon, e.g. from the Crúbadán Project which has those readily available for over 2000 languages.

For example, download and unzip British English and Irish:

wget http://crubadan.org/files/en-GB.zip 
wget http://crubadan.org/files/ga.zip

unzip '*.zip' -d word-level-language-id/corpora

Train the language models.

python word-level-language-id/train.py
You can’t perform that action at this time.