Bug Fixes
Fixed symbol corruption in files with non-ASCII characters (#88)
tree-sitter reports node positions as byte offsets into the UTF-8 encoded source, but several language strategies were using those offsets to index Python str (character-indexed) content. In any file containing multi-byte characters — comments in Chinese or Portuguese, accented letters, arrows, em dashes — every symbol extracted after the first non-ASCII character came back shifted and corrupted (greet → (): s), which also broke get_symbol_body lookups and degraded the index to text-search-only for non-English codebases.
What changed:
- All symbol/signature/import extraction now slices the same UTF-8 bytes that were fed to the parser, via a shared byte-safe helper in the strategy base class.
- Affected strategies fixed: TypeScript (15 sites), Java (7), JavaScript (3), Kotlin (2 fallback paths + slow re-encode branches), Zig (3).
- Zig line numbers were also wrong in non-ASCII files; they now come from tree-sitter's native row positions.
- C# and Rust were already byte-safe; Go, Objective-C and Python strategies don't use tree-sitter offsets and were never affected.
- Added 17 regression tests, including ASCII/non-ASCII twin fixtures for each affected language.
Thanks to @geewiz18 for the detailed report and @kajiyap for the follow-up audit that mapped the full extent of the issue.
Contributors
kajiyap and geewiz18