优化知识库文档相关操作 #1413
Merged
优化知识库文档相关操作 #1413
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…的情况下，实现： 1、支持chunk_size和chunk_overlap参数 2、支持自定义text_splitter 3、支持自定义docs 修复：csv文件不使用text_splitter
- 知识库管理中的add_docs/delete_docs/update_docs均支持批量操作，并利用多线程提高效率 - API的重建知识库接口支持多线程 - add_docs可提供参数控制上传文件后是否继续进行向量化 - add_docs/update_docs支持传入自定义docs(以json形式)。后续考虑区分完整或补充式自定义docs - download_doc接口添加`preview`参数，支持下载或预览 - kb_service增加`save_vector_store`方法，便于保存向量库（仅FAISS，其它无操作） - 将document_loader & text_splitter逻辑从KnowledgeFile中抽离出来，为后续对内存文件进行向量化做准备 - KowledgeFile支持docs & splitted_docs的缓存，方便在中间过程做一些自定义 其它： - 将部分错误输出由print改为logger.error
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
将KnowledgeFile的file2text拆分成file2docs、docs2texts和file2text三个部分，在保持接口不变…的情况下，实现：
93b133f
1、支持chunk_size和chunk_overlap参数
2、支持自定义text_splitter
3、支持自定义docs
修复：csv文件不使用text_splitter。
======================================
新功能:
preview参数，支持下载或预览save_vector_store方法，便于保存向量库（仅FAISS，其它无操作）其它：