Fix critical and high-priority bugs in chunking and regex search - #60
Merged
Conversation
Critical Fixes: - Fix divide-by-zero panic in stride_large_chunk when estimate_tokens returns 0 - Add early return for empty chunks and fallback to default chars_per_token ratio High-Priority Fixes: - Fix off-by-one error in strided chunk line_end calculation - Fix byte offset issues in regex search with mixed line endings (\r\n, \n, \r) - Add accurate line ending length calculation for proper byte positioning Tests Added: - Regression tests for empty chunk striding and zero token estimates - Tests for strided chunk line span calculations - Tests for regex search with Windows (\r\n) and mixed line endings - Unit tests for line ending length calculation helper function All tests pass and code quality checks (clippy) are clean. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
- Consolidate duplicate if blocks for '\n' and '\r' line endings - Use more elegant ends_with(['\n', '\r']) pattern matching - Maintain all functionality while improving code quality 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR fixes 3 critical and high-priority runtime bugs that could cause panics and incorrect search results:
Critical Fixes
stride_large_chunkwhenestimate_tokens()returns 0High-Priority Fixes
line_endcalculation that caused incorrect span boundaries\r\n,\n,\r) that caused misaligned search results on Windows and mixed-format filesImplementation Details
split_lines_with_endings()function properly handles all line ending formatssearch_file_streaming()withBufReaderfor memory efficiency\rcharactersTest Results
Test plan
🤖 Generated with Claude Code