text-splitters: fix text splitter start index

[aowen87]

Description:
I've updated the base text splitter so that it correctly captures the start_index. Without this fix, there are many cases of start_index receiving -1. This was a result of the chunk overlap being in number of tokens, not characters.

The search for a matching text would often begin at an index that was past the true starting point of the chunk. In the best case scenario, we would end up with -1 as a start_index. In the worst case scenario, we would end up with an incorrect start_index due to finding another instance of that chunk later in the text.

Issue:
Fixes #29884

Dependencies:
None

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Skipped Deployment

Name	Status	Preview	Comments	Updated (UTC)
langchain	⬜️ Ignored (Inspect)	Visit Preview		Jul 3, 2025 3:26pm

[codspeed-hq]

CodSpeed WallTime Performance Report

Merging #31222 will not alter performance

_{Comparing aowen87:bugfix/text_splitter_start_index (f2e5795) with master (572020c)}

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

Summary

✅ 13 untouched benchmarks

[codspeed-hq]

CodSpeed Instrumentation Performance Report

Merging #31222 will not alter performance

_{Comparing aowen87:bugfix/text_splitter_start_index (f2e5795) with master (572020c)}

Summary

✅ 14 untouched benchmarks

[aowen87]

Unfortunately, this approach of finding the index offset from the tokenized/encoded chunk doesn't work because the tokenizers and encoders often don't preserve the original text (removing whitespace, for example). I don't see an easy way to fix this without a heavy rewrite.