Skip to content
Provides Japanese fulltext search for Search::Fulltext CPAN module
Perl
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
lib/Search/Fulltext/Tokenizer
share/dic
t
.gitignore
.travis.yml
Build.PL
Changes
LICENSE
META.json
README.md
cpanfile
minil.toml

README.md

NAME

Search::Fulltext::Tokenizer::MeCab - Provides Japanese fulltext search for Search::Fulltext module

SYNOPSIS

use Search::Fulltext;
use Search::Fulltext::Tokenizer::MeCab;


my $query = '猫';
my @docs = (
    '我輩は猫である',
    '犬も歩けば棒に当る',
    '実家でてんちゃんって猫を飼ってまして,ものすっごい可愛いんですよほんと',
);


my $fts = Search::Fulltext->new({
    docs      => \@docs,
    tokenizer => "perl 'Search::Fulltext::Tokenizer::MeCab::tokenizer'",
});
my $results = $fts->search($query);
is_deeply($results, [0, 2]);        # 1st & 3rd include '猫'
my $results = $fts->search('猫 AND 可愛い');
is_deeply($results, [2]);

DESCRIPTION

Search::Fulltext::Tokenizer::MeCab is a Japanse tokenizer working with fulltext search module Search::Fulltext. Only you have to do is specify perl 'Search::Fulltext::Tokenizer::MeCab::tokenizer' as a tokenizer of Search::Fulltext.

my $fts = Search::Fulltext->new({
    docs      => \@docs,
    tokenizer => "perl 'Search::Fulltext::Tokenizer::MeCab::tokenizer'",
});

You are supposed to use UTF-8 strings for docs.

Although various queries are available like "QUERIES" in Search::Fulltext, wildcard query (e.g. '我*') and phrase query (e.g. '"我輩は猫である"') are not supported.

User dictionary can be used to change the tokenizing behavior of internally-used Text::MeCab. See ENVIRONMENTAL VARIABLES section for detailes.

ENVIRONMENTAL VARIABLES

Some environmental variables are provided to customize the behavior of Search::Fulltext::Tokenizer::MeCab.

Typical usage:

$ ENV1=foobar ENV2=buz perl /path/to/your_script_using_this_module ARGS
  • MECABDIC_USERDIC

    Specify path(s) to MeCab's user dictionary.

    See MeCab's manual to learn how to create user dictionary.

    Examples:

      MECABDIC_USERDIC="/path/to/yourdic1.dic"
      MECABDIC_USERDIC="/path/to/yourdic1.dic, /path/to/yourdic2.dic"
    
  • MECABDIC_DEBUG

    When set to not 0, debug strings appear on STDERR.

    Especially, outputs below would help check how your docs are tokenized.

      string to be parsed: 我輩は猫である (7)
      token: 我輩 (2)
      token: は (1)
      token: 猫 (1)
      token: で (1)
      token: ある (2)
      ...
      string to be parsed: 猫 AND 可愛い (9)
      token: 猫 (1)
      string to be parsed:  可愛い (4)
      token: 可愛い (3)
    

    Note that not only docs but also queries are also tokenized.

SUPPORTS

Bug reports and pull requests are welcome at https://github.com/laysakura/Search-Fulltext-Tokenizer-MeCab !

To read this manual via perldoc, use -t option for correctly displaying UTF-8 caracters.

$ perldoc -t Search::Fulltext::Tokenizer::MeCab

VERSION

Version 1.05

AUTHOR

Sho Nakatani lay.sakura@gmail.com, a.k.a. @laysakura

You can’t perform that action at this time.