Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

diff: minor cleanup of tokenizer functions #3919

Merged
merged 2 commits into from
Jun 19, 2024
Merged

diff: minor cleanup of tokenizer functions #3919

merged 2 commits into from
Jun 19, 2024

Conversation

yuja
Copy link
Collaborator

@yuja yuja commented Jun 19, 2024

Checklist

If applicable:

  • I have updated CHANGELOG.md
  • I have updated the documentation (README.md, docs/, demos/)
  • I have updated the config schema (cli/src/config-schema.json)
  • I have added tests to cover my changes

yuja added 2 commits June 19, 2024 20:24
@yuja
diff: simply pass tokenizer Fn by value
5df79c1
@yuja
diff: rewrite find_line/nonword_ranges() by using iterator adapters
d20384a 
I don't think .scan() is descriptive name, but it looks slightly better than
placing accumulator out of the closure.
@yuja yuja merged commit f8b87f6 into martinvonz:main Jun 19, 2024
17 checks passed
@yuja yuja deleted the push-nttpxprmoqur branch June 19, 2024 23:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@martinvonz martinvonz martinvonz approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@yuja @martinvonz