Making EpicSplitter persistent?

[zkx06111]

I really appreciate the notebook for efficiently building the index by using the previously persisted index.

However, this can still be slow as the splitting process still needs to run from scratch for every instance (correct me if I'm wrong).

Do you think the splitting process can also be incremental and utilizing previously split data? I think tree-sitter is incremental in parsing the AST.

[aorwall]

Yes, would make a lot of sense. Right now all files are also parsed/split every time even if they're not changed since the last commit also. So both these changes would be nice to improve performance.

For the embedding part of the ingestion I've also been thinking about ways to detect if there are significant changes and only embed chunks again if there are any. Haven't figured out how though...