GitHub - nick-b/lidruby: Language Identification with Ruby: probabilistic language identification with ruby1.9

Branches Tags

Name		Name	Last commit message	Last commit date
Latest commit History 8 Commits
models		models
README		README
generate_models.rb		generate_models.rb
lid.rb		lid.rb

Repository files navigation

NB: Requires ruby1.9.

A module to identify which of any one of a number of human languages a given text is in.

We use a simple similarity measure between frequency counts of bigrams to compare an unknown text to a set of models of known languages.  

The language models are built with samples from: 

  http://www.unicode.org/udhr/downloads.html

which is copied to models/ .