Add token class #17

himkt · 2019-07-10T07:22:58Z

I introduce the class Token such that we can see part-of-speech tags of parsed results.

tokenizer = WordTokenizer(tokenizer="MeCab", with_postag=True)
print(tokenizer.tokenize(sentence))  # -> `我輩 (名詞), は (助詞), 猫 (名詞), で (助動詞), ある (助動詞), 。 (記号)`

himkt · 2019-07-10T08:00:53Z

tiny_tokenizer/word_tokenizer.py

 import warnings


+class Token:


[Note] Token breaks the compatibility.

himkt added 7 commits July 10, 2019 16:22

Add Token class and BaseWordLevelTokenizer

31e806c

Revise tests

0edfacf

Define operator eq for Token

816bb7e

Use Token in tests

70f3433

Update examples

bf7af58

Update examples to add examples with postags

ed76568

Bump up version

a20515a

himkt changed the title ~~[WIP] Add token class~~ Add token class Jul 10, 2019

himkt commented Jul 10, 2019

View reviewed changes

tiny_tokenizer/word_tokenizer.py

import warnings

class Token:

Copy link

Owner Author

himkt Jul 10, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Note] Token breaks the compatibility.

himkt merged commit 1e99cb9 into master Jul 10, 2019

himkt deleted the add-token-class branch July 10, 2019 08:01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add token class #17

Add token class #17

himkt commented Jul 10, 2019 •

edited

Loading

himkt Jul 10, 2019

Add token class #17

Add token class #17

Conversation

himkt commented Jul 10, 2019 • edited Loading

himkt Jul 10, 2019

Choose a reason for hiding this comment

himkt commented Jul 10, 2019 •

edited

Loading