Tokenizer For Indian Languages

Tokenizer output can either be raw or in Shakti Standard Format (https://aclanthology.org/W14-5208.pdf).

Check the code for instructions

How to run the codes

- python3 tokenize_in_raw_format_without_sentence_tokenization.py --input Input --output Output
- python3 tokenize_in_SSF_format_without_sentence_tokenization.py --input Input --output Output
- python3 tokenize_in_raw_format_with_sentence_tokenization.py --input Input --output Output --lang language
- python3 tokenize_in_SSF_format_with_sentence_tokenization.py --input Input --output Output --lang language