-
Notifications
You must be signed in to change notification settings - Fork 269
Closed
Closed
Copy link
Description
What is the use case?
I want to add my custom Splitter likely cocoindex.functions.SplitRecursively()
I want to custom with @op.executor_class
Describe the solution you'd like
My code when use with llama-index
# Merge small chunks
chunks = merge_small_chunks(chunks, min_size=256)
# Sentence Splitter
splitter2 = CustomSentenceSplitter(
chunk_size=chunk_size,
chunk_overlap=0,
separator="\n",
paragraph_separator="\n\n\n",
secondary_chunking_regex="[^,.;。?!]+[,.;。?!]?",
)
chunks = splitter2(chunks)
# Context Retrieval
chunks = process_nodes(chunks, max_workers=10)
return chunks
Additional context
❤️ Contributors, please refer to 📙Contributing Guide.
Unless the PR can be sent immediately (e.g. just a few lines of code), we recommend you to leave a comment on the issue like I'm working on it or Can I work on this issue? to avoid duplicating work. Our Discord server is always open and friendly.
Metadata
Metadata
Assignees
Labels
No labels
Type
Projects
Status
Done