Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

support for multiliungal tokenizer with hooks for malayalam, adding range support #27

Open
wants to merge 38 commits into
base: multilingual
Choose a base branch
from

Conversation

dsplog
Copy link
Contributor

@dsplog dsplog commented Nov 30, 2023

fix for #29 and support for malaylam

    text = 'hello (1200 - 1230)'
    out = normalize_text(text)
    print(out)
    hello (one thousand two hundred to one thousand two hundred thirty)

@yl4579
Copy link
Owner

yl4579 commented Dec 4, 2023

I would recommend you create a new branch for this so I can merged and refer to people to this branch in README. I don't want to mess up with the current branch.

@dsplog
Copy link
Contributor Author

dsplog commented Dec 8, 2023

I would recommend you create a new branch for this so I can merged and refer to people to this branch in README. I don't want to mess up with the current branch.

can you plz create a new branch, and will change the merge destination on PR.

@yl4579
Copy link
Owner

yl4579 commented Dec 8, 2023

@dsplog dsplog changed the base branch from main to multilingual December 10, 2023 06:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants