Add Bengali to text-to-lexemes

[bodhisattwawiki]

No description provided.

[fnielsen]

Did you test it? There is sanitization in views.py too.

[fnielsen]

There may also be some issues with word tokenization.

[fnielsen]

Here is a test: https://tools.wmflabs.org/ordia/text-to-lexemes?text=%E0%A6%AD%E0%A6%BE%E0%A6%B0%E0%A6%A4+%E0%A6%9C%E0%A6%BE%E0%A6%A8%E0%A7%81%E0%A6%AF%E0%A6%BC%E0%A6%BE%E0%A6%B0%E0%A6%BF&text-language=bn

You should tell me: Is the word tokenization ok? I think there might be a problem. For an Indian language it did not initially work as apparently a more complex regular expression was needed.

[fnielsen]

Thanks for your contribution!

[bodhisattwawiki]

Yes, there is a problem with the text জানুয়ারি in your link, where it has been split into two. Here it looks ok though, with the same texts.

[fnielsen]

You second link also split the last word (there is something wrong with the way that I construct the URL.

[bodhisattwawiki]

Strange, the second link was all right, before I posted the message. Here is the screenshot, where the text didn't split.

[fnielsen]

I think it is a problem with my handling of POST and GET requests.

[fnielsen]

You can try and copy-paste the text from the edit field in this URL: https://tools.wmflabs.org/ordia/text-to-lexemes?text=<copy-paste-here>&text-language=bn