-
Notifications
You must be signed in to change notification settings - Fork 1.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Enhancement] Support tokenize function #45119
base: main
Are you sure you want to change the base?
[Enhancement] Support tokenize function #45119
Conversation
b368173
to
5536c53
Compare
5536c53
to
c10ab97
Compare
c10ab97
to
49e7d23
Compare
Signed-off-by: leorishdu <18771113323@163.com>
49e7d23
to
923dbc4
Compare
@dujijun007 thank you for the contribution, could you create an issue to describe this new function? |
|
Signed-off-by: leorishdu <18771113323@163.com>
f69928c
to
e4247fc
Compare
Quality Gate passedIssues Measures |
[FE Incremental Coverage Report]✅ pass : 0 / 0 (0%) |
[BE Incremental Coverage Report]✅ pass : 55 / 57 (96.49%) file detail
|
Why I'm doing:
The different results of tokenization provided by various tokenizers are too vague to users, so we need a tokenize function to allow users to figure it out easily.
What I'm doing:
Support a tokenize function, like
tokenize(<tokenizer_name>, <content>)
Fixes #45145
What type of PR is this:
Does this PR entail a change in behavior?
If yes, please specify the type of change:
Checklist:
Bugfix cherry-pick branch check: