an re-implementation of rmmseg (Chinese word segmentation library for Ruby) in C++
C++ Ruby
Switch branches/tags
Nothing to show
Pull request Compare This branch is 29 commits behind pluskid:master.
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
data
ext/rmmseg
lib
tools
.gitignore
README
rmmseg-cpp.gemspec

README

= rmmseg-cpp

== Background

rmmseg (http://rmmseg.rubyforge.org) is a Chinese word segmentation library
written for and in Ruby. It features full integration with Ruby. However, 
its performance (both time and memory) is terrible in some cases, especially
when you use the complex algorithm.

So I re-implemented rmmseg in C++ and wrapped it as a Ruby extension. This
gem is aimed at high performance and thus less extensible than the pure-
Ruby rmmseg gem. There are also some differences between them:

* The dictionary format is different. For performance reason, the words
  dictionary of rmmseg-cpp included the word length information in it. 
  See rdoc of the Dictionary class for more information of the format.
  It is likely that I will upgrade rmmseg's dictionary format to make
  those two compatible, writing a Ruby script to convert the dictionary
  is almost trivial.

  While the dictionaries in rmmseg will be loaded automatically when needed,
  you'll need to load dictionaries explicitly in rmmseg-cpp.

* Only complex algorithm is provided currently. And I don't see any need
  to implement the simple algorithm here if complex algorithm has a good
  performance and a much better accuracy.

== Install

This project is hosted at github (http://github.com/pluskid/rmmseg-cpp/).
You can use rubygems to install rmmseg-cpp:

  sudo gem install pluskid-rmmseg-cpp --source=http://gems.github.com

Unfortunately, the command is long and you'll have to use the 'pluskid'
prefix currently. Or you can also check out the latest source code:

  git clone git://github.com/pluskid/rmmseg-cpp.git

== Usage

Integration with ferret is almost identical as before, to run the example for
rmmseg (http://rmmseg.rubyforge.org/#Analyzer-for-Ferret), only one minor
change is needed:

  RMMSeg::Dictionary.load_dictionaries

Add the line above to explicitly load the default dictionaries before 
using the analyzer. You can also add your own dictionaries through
<tt>Dictionary#add</tt>.