New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Malay language support #12602
Malay language support #12602
Conversation
Thanks for the PR! An initial question about the tokenizer exceptions list: would it make sense for Indonesian and Malay to use the exact same list of exceptions? This looks like it's just a copy of the Indonesian list, so Malay could import this list from Indonesian rather than copying it if there aren't any differences. |
Hi, currently the exceptions list not perfect, some of the Indonesian exceptions can be use for Malay and some might need to be remove in future. I think better separate it so that in future easy to maintain |
Thanks for the info! It's fine to maintain a separate list. Are you planning on updating the exceptions list for Malay for this PR? It's a little unexpected for users to get default exceptions that weren't customized for this language. |
ok, I will update the list within this week |
Thanks, this looks like a good initial version for all the defaults! |
Description
Add Malay language
Types of change
new feature
Checklist