Hashtag, Emoji and Mention Handler not Working

[ertugrul-dmr]

While working on some preprocessing steps (#270) we noticed that arguments for tokenizers (including Text2Doc) has no effect on the outcome. Basically enable/disable options are not working.

For this issue we might need to create test cases and find the root cause of the problem.

To replicate this you can basically follow these steps:

from sadedegel.bblock.word_tokenizer import ICUTokenizer
tokenizer = ICUTokenizer(hashtag=True, mention=True, emoji=True)
text = 'bu #bir @metindir, 🍰'
print(tokenizer._tokenize(text))

Will result same with False or True options. Same goes for Text2Doc example below:

from sadedegel.extension.sklearn import Text2Doc
tokenizer = Text2Doc(hashtag=True, mention=True, emoji=True)
texts =[ 'bu', '#bir', '@metindir', '🍰']
print(tokenizer.transform(texts))

[husnusensoy]

Please refer to pr #274 to see that this is not a bug but the expected behavior