Using the tokenizer

[donnekgit]

Hi Dan
I'm wondering how to implement the use of the tokenizer from this toolset. You suggest:
$tokenizer = new GeneralTokenizer();
$tokens = $tokenizer->tokenize("some text")
but how do I instantiate the toolset itself on the php page? I've tried various include/require lines, but none of them work.
Thanks.

[yooper]

I am going to need some more information in order to help you.

Are you using Composer?

What type of web page are you using plain PHP, or some type of template system?

Cheers,

[donnekgit]

I have Composer installed system-wide, but I've never used it. I'm using command-line php, ie scripts that are sandwiched in .

[yooper]

The easiest route will be to use composer.
Create a file composer.json in your project directory or wherever your script is you want to use the tokenizer in. Here is the contents of the composer.json file.

{
    "require-dev": {
        "yooper/php-text-analysis": "dev-master"
    },
    "minimum-stability": "dev"
}

Afterwards run the command

composer install

That will install all the dependencies. Here is an example cli script to show how to access the library

#!/usr/bin/php
<?php
require_once('vendor/autoload.php');

$str = "PHP Text Analysis is a library for performing Information Retrieval (IR) and Natural Language Processing (NLP) tasks using the PHP language";

$tokenizer = new \TextAnalysis\Tokenizers\WhitespaceTokenizer();
$tokens = $tokenizer->tokenize($str);
var_dump($tokens);

[donnekgit]

Cool - thanks for that. It worked fine, so I can now examine the toolset.