Skip to content

Commit

Permalink
doc en: apply english corrections
Browse files Browse the repository at this point in the history
  • Loading branch information
kenhys committed May 23, 2013
1 parent 573f620 commit 7b19373
Showing 1 changed file with 4 additions and 5 deletions.
9 changes: 4 additions & 5 deletions doc/source/reference/commands/tokenize.txt
Expand Up @@ -10,16 +10,15 @@

.. Caution::

``tokenize`` command is the experimental feature.
``tokenize`` command is an experimental feature.
This command may be changed in the future.

Summary
-------

``tokenize`` command tokenizes text by the specified tokenizer.

There is no need to create table to use ``tokenize`` command.
It is usefull for you to check the results of tokenizer.
There is no need to create a table to use ``tokenize`` command.

Syntax
--------
Expand Down Expand Up @@ -56,7 +55,7 @@ There are required parameters, ``tokenizer`` and ``string``.

It specifies the name of tokenizer.

Here are the list of built-in tokenizer:
Here are the list of built-in tokenizers:

* TokenBigram
* TokenBigramSplitSymbol
Expand All @@ -71,7 +70,7 @@ Here are the list of built-in tokenizer:
* TokenTrigram
* TokenUnigram

If you want to try another tokenizer, you need to register additional tokenizer plugin by ``register`` command.
If you want to use other tokenizers, you need to register additional tokenizer plugin by ``register`` command.

``string``
""""""""""
Expand Down

0 comments on commit 7b19373

Please sign in to comment.