Skip to content

Commit

Permalink
Update example in documentation (#9)
Browse files Browse the repository at this point in the history
  • Loading branch information
vbkaisetsu committed Apr 13, 2023
1 parent ef0f5c8 commit 044ce52
Show file tree
Hide file tree
Showing 2 changed files with 16 additions and 3 deletions.
17 changes: 15 additions & 2 deletions docs/source/examples.rst
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ You can check the version number as shown below to use compatible models:
>>> import vibrato
>>> vibrato.VIBRATO_VERSION
'0.3.3'
'0.5.0'
Examples:

Expand All @@ -20,7 +20,7 @@ Examples:
>>> import vibrato
>>> with open('path/to/system.dic', 'rb') as fp:
>>> dict_data = fp.read()
... dict_data = fp.read()
>>> tokenizer = vibrato.Vibrato(dict_data)
>>> tokens = tokenizer.tokenize('社長は火星猫だ')
Expand All @@ -46,3 +46,16 @@ Examples:
>>> tokens[0].end()
2
The distributed models are compressed in zstd format. If you want to load these compressed models,
you must decompress them outside the API:

.. code-block:: python
>>> import vibrato
>>> import zstandard # zstandard package in PyPI
>>> dctx = zstandard.ZstdDecompressor()
>>> with open('path/to/system.dic.zst', 'rb') as fp:
... dict_reader = dctx.stream_reader(fp)
>>> tokenizer = vibrato.Vibrato(dict_reader.read())
2 changes: 1 addition & 1 deletion src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -152,7 +152,7 @@ pub struct TokenizerWrapper {
/// Examples:
/// >>> import vibrato
/// >>> with open('path/to/system.dic', 'rb') as fp:
/// >>> dict_data = fp.read()
/// ... dict_data = fp.read()
/// >>> tokenizer = vibrato.Vibrato(dict_data)
/// >>> tokens = tokenizer.tokenize('社長は火星猫だ')
/// >>> len(tokens)
Expand Down

0 comments on commit 044ce52

Please sign in to comment.