text-splitters: fix text_splitter keep_seprator git bug

[wenngong]

Description: fix text_splitter keep_seprator git bug:

1. both _split_text_with_regex and self._merge_splits functions dill with the self._keep_separator param. So remove the _keep_separator relate process in self._merge_splits;
2. To minimize the impact of code changes, Just keep the self._merge_splits second separator param to "". I can further modify the _merge_splits function to remove separator related codes by a further patch.

Issue: #23394

from langchain.text_splitter import RecursiveCharacterTextSplitter


if __name__ == "__main__":
    # Wrong behaviour, using \s instead of regular space
    splitter_keep = RecursiveCharacterTextSplitter(
        separators=[r"\s"],
        keep_separator=False,
        is_separator_regex=True,
        chunk_size=15,
        chunk_overlap=0,
        strip_whitespace=False)
    assert splitter_keep.split_text("Hello world")[0] == r"Helloworld"

    # Expected behaviour, keeping regular space
    splitter_no_keep = RecursiveCharacterTextSplitter(
        separators=[r"\s"],
        keep_separator=True,
        is_separator_regex=True,
        chunk_size=15,
        chunk_overlap=0,
        strip_whitespace=False)
    assert splitter_no_keep.split_text("Hello world")[0] == r"Hello world"

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
langchain	⬜️ Ignored (Inspect)	Visit Preview		Jun 25, 2024 11:30am

[wenngong]

wrong fix up now, I will fix by another patch...