Skip to content

Reduce memory use in case-based tokenization. #8375

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Dec 9, 2024
Merged

Reduce memory use in case-based tokenization. #8375

merged 1 commit into from
Dec 9, 2024

Conversation

@isoos
Copy link
Collaborator

@isoos isoos commented Dec 9, 2024

  • In the old method changeIndex was built first and then its value was used in a second parsing round.
  • Since the two rounds can be done in a single processing, there is no need to build the array and then discard it.
  • It speeds up index snapshot loading by 2-3% and may also affect the request serving too.

@isoos isoos requested review from jonasfj and sigurdm December 9, 2024 09:49
@isoos isoos merged commit ec19ac0 into dart-lang:master Dec 9, 2024
32 checks passed
@isoos isoos deleted the text-opt branch December 9, 2024 11:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@sigurdm sigurdm sigurdm approved these changes

@jonasfj jonasfj Awaiting requested review from jonasfj

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@isoos @sigurdm