You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
hwchase17
changed the title
Add more complex ways of splitting text, rather than just character based.
Add huggingface tokenizer support for splitting text
Nov 8, 2022
Would be good to have some methods that split on tokens as in the OpenAI example
https://github.com/openai/openai-cookbook/blob/459afa7d9bf026c4434f54458dc7d9e7d9f9f5fe/examples/Obtain_dataset.ipynb
The text was updated successfully, but these errors were encountered: