What would you like to see?
Resource based chunking
I understand that AnythingLLM handles chunking independently before passing the chunks to the embedding model. While we can set the max input size based on the embedder's token window, chunking should be tailored to the type of input data.
For example, the default chunking works well for text data, but for a code repository, chunking should better align with the code structure to preserve semantic meaning—such as using ASTs (Abstract Syntax Trees) or function-level segmentation.
What would you like to see?
Resource based chunking
I understand that AnythingLLM handles chunking independently before passing the chunks to the embedding model. While we can set the max input size based on the embedder's token window, chunking should be tailored to the type of input data.
For example, the default chunking works well for text data, but for a code repository, chunking should better align with the code structure to preserve semantic meaning—such as using ASTs (Abstract Syntax Trees) or function-level segmentation.