Tokenization strategy

[IlyasMoutawwakil]

Hi and thanks for the awesome repo. Did you try any other tokenization strategies (sentencepiece, wordpiece or bpe). I see you use a character level tokenization which is nice but probably dosen't make full use of the language model capabilities. I would love to get some insights.

[YongWookHa]

Hello.
Thank you for having an interest in this repository.
Unfortunately, I had used only simple tokenization methods because I used this model to recognize Korean letters.
I'm pretty sure that you would get better performance when you use more sophisticated methods like BPE or sentencepiece.

[WongVi]

@YongWookHa could you please let me know how can I make a tokenizer for the Japanese dataset?