Skip to content
No description or website provided.
C++ Perl C
Latest commit eb7ec18 Mar 31, 2011 @rboulton Olly points out that xapian/unicode.h shouldn't be included by extern…
…al files.

Include via xapian.h instead

README.rst

Description

This module is a word tokenizer for CJK texts, supporting n-gram tokenization. It is designed to be used with Xapian (http://xapian.org), and uses Xapian's unicode routines.

Currently, there is no documentation other than the source code.

Authors

Features

  • N-gram tokenization on CJK texts.
  • Conversion from Traditional Chinese to Simplified Chinese, and vice versa.

History

This project was taken from http://code.google.com/p/cjk-tokenizer/ , but then modified to use Xapian's internal unicode routines.

Something went wrong with that request. Please try again.