The demo code about tokenizer.py in tools dir exists some mistakes

[kkscilife]

Describe the bug

In demo code: python tools/tokenizer.py --raw_data_name your_raw_data_file_name(without suffix) --input_file_type 'text' or 'json' or 'jsonl' --bin your_output_bin_path
'text' should be 'txt'
same to the demo code: python tools/tokenizer.py --raw_data_name raw_data --input_file_type 'text' --bin cn/output.bin
Both in chinese and engish，and other language if existed

Environment

any browser

Other information

No response

[gaoyang07]

Hello there, thanks for reporting the typo. We've updated the tools/tokenizer.py and also the usages in README.

Check pr51 for more details.