[Question] Is there any to apply custom token separators for a field

[sudheervarmam]

Description

I want to apply a custom token separator to a specific filed. By default type sense is splitting given word with space, new-line characters and we have ability to add list of symbols or special characters to be used for splitting the text into individual words in addition to space and new-line characters.

But in my case i want to spilt the text into multiple token by positions similar to elastic search.

Given Text: pine apple fruit

i want above text to be tokenized into below tokens

• pine apple fruit
• pineapplefruit
• pineapple fruit
• pine applefruit
• pineapple
• applefruit
• pine
• apple
• fruit

Metadata

Typsense Version: v.0.23.1

OS:

[jasonbosco]

For this particular type of splitting based on space, you can use the split_join_tokens search parameter, documented under this table: https://typesense.org/docs/0.23.1/api/search.html#typo-tolerance-parameters

[kishorenc]

Closing in favor of #1369