Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docker部署出错,无法正常问答和知识库应用 #375

Open
2 tasks done
nevermorez opened this issue May 31, 2024 · 3 comments
Open
2 tasks done

docker部署出错,无法正常问答和知识库应用 #375

nevermorez opened this issue May 31, 2024 · 3 comments

Comments

@nevermorez
Copy link

是否已有关于该错误的issue或讨论? | Is there an existing issue / discussion for this?

  • 我已经搜索过已有的issues和讨论 | I have searched the existing issues / discussions

该问题是否在FAQ中有解答? | Is there an existing answer for this in FAQ?

  • 我已经搜索过FAQ | I have searched FAQ

当前行为 | Current Behavior

按照5.17日的安装文档,docker部署,run.sh安装未报错,启动容器终端输出未见异常,访问UI可以正常访问,遇见两个问题:
1.无法正常上传文件,上传第一个文件一直报错:milvus插入失败,请稍后再试,第二个第三个没问题,但出现该问题后,该知识库无法正常删除,报错请求失败,该文件同样无法删除,docx,ppt,txt均尝试了。
2.无法正常问答,问答一直报错请刷新后重试。

测试了ollama,本地Qwen,应该不是模型问题,看sanic.log,问题应该出在es上面。

期望行为 | Expected Behavior

解决BUG

运行环境 | Environment

- OS:Centos8
- NVIDIA Driver:535
- CUDA:11.8
- docker:2.24.0
- docker-compose:2.27.0
- NVIDIA GPU:4090
- NVIDIA GPU Memory:24GB

QAnything日志 | QAnything logs

新建知识库时应该还是正常的:

INFO:debug_logger:check_user_exist []
INFO:debug_logger:list_kbs zzp
INFO:debug_logger:all kb infos: [{'kb_id': 'KB7e4842c2615c40649e78754c8e39203b', 'kb_name': '默认知识库'}]
INFO:elastic_transport.transport:GET http://es-container-local:9200/ [status:200 duration:0.002s]
INFO:debug_logger:##ES## - success to connect to {'name': 'd9dec8e8aa4a', 'cluster_name': 'docker-cluster', 'cluster_uuid': 'mocOuywHSNa08Yp98Kcy7A', 'version': {'number': '8.11.4', 'build_flavor': 'default', 'build_type': 'docker', 'build_hash': 'da06c53fd49b7e676ccf8a32d6655c5155c16d81', 'build_date': '2024-01-08T10:05:08.438562403Z', 'build_snapshot': False, 'lucene_version': '9.8.0', 'minimum_wire_compatibility_version': '7.17.0', 'minimum_index_compatibility_version': '7.0.0'}, 'tagline': 'You Know, for Search'}

随后在插入文件时发现有一个es的404(已用########突出):

INFO:debug_logger:XXX-wxs.pptx, cd4eca5871ac4bb7a43fb7ca0058815d, success
INFO:debug_logger:success init localfile XXX-wxs.pptx
INFO:debug_logger:insert_files_to_milvus: KB6fda41a1c95f4605ab812263d921740a
INFO:debug_logger:collection zzp exists
INFO:debug_logger:partitions: ['KB6fda41a1c95f4605ab812263d921740a']
INFO:elastic_transport.transport:GET http://es-container-local:9200/ [status:200 duration:0.002s]
INFO:debug_logger:##ES## - success to connect to {'name': 'd9dec8e8aa4a', 'cluster_name': 'docker-cluster', 'cluster_uuid': 'mocOuywHSNa08Yp98Kcy7A', 'version': {'number': '8.11.4', 'build_flavor': 'default', 'build_type': 'docker', 'build_hash': 'da06c53fd49b7e676ccf8a32d6655c5155c16d81', 'build_date': '2024-01-08T10:05:08.438562403Z', 'build_snapshot': False, 'lucene_version': '9.8.0', 'minimum_wire_compatibility_version': '7.17.0', 'minimum_index_compatibility_version': '7.0.0'}, 'tagline': 'You Know, for Search'}
INFO:debug_logger:before 2nd split doc lens: 53
INFO:debug_logger:after 2nd split doc lens: 44
INFO:debug_logger:langchain analysis content head: XXX
INFO:debug_logger:split time: 0.3504297733306885 44
INFO:debug_logger:embedding number: 3

0%| | 0/3 [00:00<?, ?it/s]
33%|███▎ | 1/3 [00:00<00:00, 8.08it/s]
100%|██████████| 3/3 [00:00<00:00, 24.23it/s]
INFO:debug_logger:embedding time: 0.12532377243041992 44
INFO:debug_logger:now inser_file XXX-wxs.pptx
INFO:debug_logger:Inserting into Milvus...
INFO:debug_logger:XXX-wxs.pptx (insert count: 44, delete count: 0, upsert count: 0, timestamp: 450132530566529026, success count: 44, err count: 0)
INFO:debug_logger:now inser_file for es: XXX-wxs.pptx
INFO:debug_logger:Inserting into es ...
#############INFO:elastic_transport.transport:HEAD http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a [status:404 duration:0.003s]
INFO:debug_logger:##ES## - Creating index zzp++kb6fda41a1c95f4605ab812263d921740a with:
mappings: {'properties': {'file_id': {'type': 'keyword', 'index': True}, 'content': {'type': 'text', 'similarity': 'custom_bm25', 'index': True, 'analyzer': 'ik_smart', 'search_analyzer': 'ik_smart'}}}
settings: {'index': {'similarity': {'custom_bm25': {'type': 'BM25', 'k1': '1.3', 'b': '0.6'}}}}
INFO:elastic_transport.transport:PUT http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a [status:N/A duration:10.010s]
##############WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)> has failed for 1 times in a row, putting on 1 second timeout
ERROR:debug_logger:ES insert file_id: cd4eca5871ac4bb7a43fb7ca0058815d
file_name: XXX-wxs.pptx
failed: Connection timed out
INFO:debug_logger:insert time: 10.019953966140747
INFO:debug_logger:insert_to_milvus: success num: 0, failed num: 1
INFO:debug_logger:upload_files zzp
INFO:debug_logger:mode: strong
WARNING:root:missing use_local_file in request
INFO:debug_logger:check_kb_exist [('KB6fda41a1c95f4605ab812263d921740a',)]

第二次再次插入一个文档,前端UI显示成功,但日志中有一堆warning:

INFO:debug_logger:ori name: 离线部署centos文档.docx
INFO:debug_logger:decode name: 离线部署centos文档.docx
INFO:debug_logger:cleaned name: 离线部署centos文档.docx
INFO:debug_logger:check_user_exist [('zzp',)]
INFO:debug_logger:check_kb_exist [('KB6fda41a1c95f4605ab812263d921740a',)]
INFO:debug_logger:add_file: bbd20c38f53b473babbafe01f5aab6b7
INFO:debug_logger:离线部署centos文档.docx, bbd20c38f53b473babbafe01f5aab6b7, success
INFO:debug_logger:success init localfile 离线部署centos文档.docx
INFO:debug_logger:insert_files_to_milvus: KB6fda41a1c95f4605ab812263d921740a
INFO:debug_logger:match milvus_client: <qanything_kernel.connector.database.milvus.milvus_client.MilvusClient object at 0x7f37c10e3f70>
INFO:debug_logger:before 2nd split doc lens: 170
INFO:debug_logger:after 2nd split doc lens: 170
INFO:debug_logger:langchain analysis content head: 搭建私有yum仓库,安装gcc gcc-c++,ffmpeg(三步法),nginx(nginx在centos8.
INFO:debug_logger:split time: 0.04621243476867676 170
INFO:debug_logger:embedding number: 11

0%| | 0/11 [00:00<?, ?it/s]
100%|██████████| 11/11 [00:00<00:00, 286.36it/s]
INFO:debug_logger:embedding time: 0.04653787612915039 170
INFO:debug_logger:now inser_file 离线部署centos文档.docx
INFO:debug_logger:Inserting into Milvus...
INFO:debug_logger:离线部署centos文档.docx (insert count: 170, delete count: 0, upsert count: 0, timestamp: 450132533227028483, success count: 170, err count: 0)
INFO:debug_logger:now inser_file for es: 离线部署centos文档.docx
INFO:debug_logger:Inserting into es ...
INFO:elastic_transport.transport:HEAD http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a [status:200 duration:0.003s]
WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)> has been marked alive after a successful request
INFO:debug_logger:##ES## - Index zzp++kb6fda41a1c95f4605ab812263d921740a already exists. Skipping creation.
INFO:debug_logger:list_docs zzp
INFO:debug_logger:kb_id: KB7e4842c2615c40649e78754c8e39203b
INFO:debug_logger:delete_knowledge_base zzp
INFO:debug_logger:check_kb_exist [('KB7e4842c2615c40649e78754c8e39203b',)]
INFO:debug_logger:collection zzp exists
INFO:debug_logger:partitions: ['KB7e4842c2615c40649e78754c8e39203b']
INFO:elastic_transport.transport:GET http://es-container-local:9200/ [status:200 duration:0.002s]
INFO:debug_logger:##ES## - success to connect to {'name': 'd9dec8e8aa4a', 'cluster_name': 'docker-cluster', 'cluster_uuid': 'mocOuywHSNa08Yp98Kcy7A', 'version': {'number': '8.11.4', 'build_flavor': 'default', 'build_type': 'docker', 'build_hash': 'da06c53fd49b7e676ccf8a32d6655c5155c16d81', 'build_date': '2024-01-08T10:05:08.438562403Z', 'build_snapshot': False, 'lucene_version': '9.8.0', 'minimum_wire_compatibility_version': '7.17.0', 'minimum_index_compatibility_version': '7.0.0'}, 'tagline': 'You Know, for Search'}
INFO:elastic_transport.transport:DELETE http://es-container-local:9200/zzp%2B%2Bkb7e4842c2615c40649e78754c8e39203b?ignore_unavailable=true [status:200 duration:0.001s]
INFO:debug_logger:##ES## - success to delete index: ['zzp++kb7e4842c2615c40649e78754c8e39203b']
INFO:debug_logger:##ES## - success delete kb_ids: KB7e4842c2615c40649e78754c8e39203b
INFO:debug_logger:delete_knowledge_base: ['KB7e4842c2615c40649e78754c8e39203b']
INFO:debug_logger:list_kbs zzp
INFO:debug_logger:all kb infos: [{'kb_id': 'KB6fda41a1c95f4605ab812263d921740a', 'kb_name': 'aaa'}]
INFO:elastic_transport.transport:PUT http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_bulk?refresh=false [status:N/A duration:10.010s]
WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)> has failed for 1 times in a row, putting on 1 second timeout
INFO:debug_logger:离线部署centos文档.docx Error adding texts: Connection timed out
INFO:debug_logger:insert time: 10.025246858596802
INFO:debug_logger:insert_to_milvus: success num: 1, failed num: 0
INFO:debug_logger:list_docs zzp
INFO:debug_logger:kb_id: KB6fda41a1c95f4605ab812263d921740a
INFO:debug_logger:list_docs zzp
INFO:debug_logger:kb_id: KB6fda41a1c95f4605ab812263d921740a
INFO:debug_logger:delete_docs zzp
INFO:debug_logger:check_kb_exist [('KB6fda41a1c95f4605ab812263d921740a',)]
INFO:debug_logger:check_file_exist [('cd4eca5871ac4bb7a43fb7ca0058815d', 'yellow')]
INFO:debug_logger:match milvus_client: <qanything_kernel.connector.database.milvus.milvus_client.MilvusClient object at 0x7f37c10e3f70>
INFO:debug_logger:milvus delete files_id: ['cd4eca5871ac4bb7a43fb7ca0058815d']
INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(http://es-container-local:9200)> (force=False)
INFO:elastic_transport.transport:HEAD http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a [status:200 duration:0.002s]
WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)> has been marked alive after a successful request
INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(http://es-container-local:9200)> (force=False)
INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.006s]
WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)> has failed for 1 times in a row, putting on 1 second timeout
WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 0 of 3)
INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.001s]
WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)> has failed for 2 times in a row, putting on 2 second timeout
WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 1 of 3)
INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.001s]
WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)> has failed for 3 times in a row, putting on 4 second timeout
WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 2 of 3)
INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.001s]
WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)> has failed for 4 times in a row, putting on 8 second timeout

随后删除报错log如下:

[2024-05-31 09:35:57 +0800] [861] [ERROR] Exception occurred while handling uri: 'http://localhost:8777/api/local_doc_qa/delete_files'
Traceback (most recent call last):
File "handle_request", line 97, in handle_request
File "/workspace/qanything_local/qanything_kernel/qanything_server/handler.py", line 247, in delete_docs
milvus_kb.delete_files(file_ids)
File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/milvus_client.py", line 288, in delete_files
es_records = self.client.search(files_id, field='file_id')
File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/es_client.py", line 182, in search
response = self.client.search(
File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/utils.py", line 446, in wrapped
return api(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/init.py", line 3836, in search
return self.perform_request( # type: ignore[return-value]
File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/_base.py", line 320, in perform_request
raise HTTP_EXCEPTIONS.get(meta.status, ApiError)(
elasticsearch.ApiError: ApiError(503, 'search_phase_execution_exception', None)
ERROR:sanic.error:Exception occurred while handling uri: 'http://localhost:8777/api/local_doc_qa/delete_files'
Traceback (most recent call last):
File "handle_request", line 97, in handle_request
File "/workspace/qanything_local/qanything_kernel/qanything_server/handler.py", line 247, in delete_docs
milvus_kb.delete_files(file_ids)
File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/milvus_client.py", line 288, in delete_files
es_records = self.client.search(files_id, field='file_id')
File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/es_client.py", line 182, in search
response = self.client.search(
File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/utils.py", line 446, in wrapped
return api(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/init.py", line 3836, in search
return self.perform_request( # type: ignore[return-value]
File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/_base.py", line 320, in perform_request
raise HTTP_EXCEPTIONS.get(meta.status, ApiError)(
elasticsearch.ApiError: ApiError(503, 'search_phase_execution_exception', None)

尝试问答,报错log如下:

INFO:debug_logger:local_doc_chat zzp
INFO:debug_logger:rerank True
INFO:debug_logger:history: []
INFO:debug_logger:question: 我想知道
INFO:debug_logger:kb_ids: ['KB6fda41a1c95f4605ab812263d921740a']
INFO:debug_logger:user_id: zzp
INFO:debug_logger:check_kb_exist [('KB6fda41a1c95f4605ab812263d921740a',)]
INFO:debug_logger:match milvus_client: <qanything_kernel.connector.database.milvus.milvus_client.MilvusClient object at 0x7f37c10e3f70>
INFO:debug_logger:streaming: True
INFO:debug_logger:start generate answer
INFO:debug_logger:start generate...
INFO:debug_logger:embedding number: 1
函数 get_knowledge_based_answer 执行耗时: 7.152557373046875e-07 秒

0%| | 0/1 [00:00<?, ?it/s]
100%|██████████| 1/1 [00:00<00:00, 52.80it/s]
INFO:debug_logger:milvus group number: 1
INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(http://es-container-local:9200)> (force=False)
INFO:elastic_transport.transport:HEAD http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a [status:200 duration:0.001s]
WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)> has been marked alive after a successful request
INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(http://es-container-local:9200)> (force=False)
INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.002s]
WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)> has failed for 1 times in a row, putting on 1 second timeout
WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 0 of 3)
INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(http://es-container-local:9200)> (force=False)
INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.001s]
WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)> has failed for 2 times in a row, putting on 2 second timeout
WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 1 of 3)
INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(http://es-container-local:9200)> (force=False)
INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.001s]
WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)> has failed for 3 times in a row, putting on 4 second timeout
WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 2 of 3)
INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.001s]
WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)> has failed for 4 times in a row, putting on 8 second timeout
[2024-05-31 09:36:07 +0800] [861] [ERROR] Exception occurred while handling uri: 'http://localhost:8777/api/local_doc_qa/local_doc_chat'
Traceback (most recent call last):
File "handle_request", line 132, in handle_request
"_asgi_lifespan",
File "/usr/local/lib/python3.10/dist-packages/sanic/response/types.py", line 547, in stream
await self.streaming_fn(self)
File "/workspace/qanything_local/qanything_kernel/qanything_server/handler.py", line 355, in generate_answer
for resp, next_history in local_doc_qa.get_knowledge_based_answer(
File "/workspace/qanything_local/qanything_kernel/core/local_doc_qa.py", line 225, in get_knowledge_based_answer
source_documents = self.get_source_documents(retrieval_queries, milvus_kb)
File "/workspace/qanything_local/qanything_kernel/core/local_doc_qa.py", line 133, in get_source_documents
batch_result = milvus_kb.search_emb_async(embs=embs, top_k=top_k, queries=queries)
File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/milvus_client.py", line 181, in search_emb_async
return future.result()
File "/usr/lib/python3.10/concurrent/futures/_base.py", line 458, in result
return self.__get_result()
File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
raise self._exception
File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/milvus_client.py", line 170, in __search_emb_sync
es_records = self.client.search(queries)
File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/es_client.py", line 182, in search
response = self.client.search(
File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/utils.py", line 446, in wrapped
return api(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/init.py", line 3836, in search
return self.perform_request( # type: ignore[return-value]
File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/_base.py", line 320, in perform_request
raise HTTP_EXCEPTIONS.get(meta.status, ApiError)(
elasticsearch.ApiError: ApiError(503, 'search_phase_execution_exception', None)
ERROR:sanic.error:Exception occurred while handling uri: 'http://localhost:8777/api/local_doc_qa/local_doc_chat'
Traceback (most recent call last):
File "handle_request", line 132, in handle_request
"_asgi_lifespan",
File "/usr/local/lib/python3.10/dist-packages/sanic/response/types.py", line 547, in stream
await self.streaming_fn(self)
File "/workspace/qanything_local/qanything_kernel/qanything_server/handler.py", line 355, in generate_answer
for resp, next_history in local_doc_qa.get_knowledge_based_answer(
File "/workspace/qanything_local/qanything_kernel/core/local_doc_qa.py", line 225, in get_knowledge_based_answer
source_documents = self.get_source_documents(retrieval_queries, milvus_kb)
File "/workspace/qanything_local/qanything_kernel/core/local_doc_qa.py", line 133, in get_source_documents
batch_result = milvus_kb.search_emb_async(embs=embs, top_k=top_k, queries=queries)
File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/milvus_client.py", line 181, in search_emb_async
return future.result()
File "/usr/lib/python3.10/concurrent/futures/_base.py", line 458, in result
return self.__get_result()
File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
raise self._exception
File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/milvus_client.py", line 170, in __search_emb_sync
es_records = self.client.search(queries)
File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/es_client.py", line 182, in search
response = self.client.search(
File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/utils.py", line 446, in wrapped
return api(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/init.py", line 3836, in search
return self.perform_request( # type: ignore[return-value]
File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/_base.py", line 320, in perform_request
raise HTTP_EXCEPTIONS.get(meta.status, ApiError)(
elasticsearch.ApiError: ApiError(503, 'search_phase_execution_exception', None)

复现方法 | Steps To Reproduce

按照文档下载源码,bash run.sh -c cloud -i 0 下载镜像,起容器,出现上述问题

备注 | Anything else?

No response

@yanghp86
Copy link

我也遇到了一样的问题了,INFO:debug_logger:milvus delete files_id: ['804d5229f04d4fdbb76fa565a53aa994']
INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(**http://es-container-local:9200**)> (force=False)
INFO:elastic_transport.transport:HEAD http://es-container-local:9200/zzp%2B%2Bkb726f380c94ac4583b4176cc113654544 [status:200 duration:0.002s]
WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)> has been marked alive after a successful request
INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(http://es-container-local:9200)> (force=False)
INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb726f380c94ac4583b4176cc113654544/_search [status:503 duration:0.002s]
WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)> has failed for 1 times in a row, putting on 1 second timeout
WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 0 of 3)
INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(http://es-container-local:9200)> (force=False)
INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb726f380c94ac4583b4176cc113654544/_search [status:503 duration:0.001s]
WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)> has failed for 2 times in a row, putting on 2 second timeout
WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 1 of 3)
INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb726f380c94ac4583b4176cc113654544/_search [status:503 duration:0.001s]
WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)> has failed for 3 times in a row, putting on 4 second timeout
WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 2 of 3)
INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb726f380c94ac4583b4176cc113654544/_search [status:503 duration:0.002s]
WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)> has failed for 4 times in a row, putting on 8 second timeout
[2024-07-10 14:51:31 +0800] [678] [ERROR] Exception occurred while handling uri: 'http://10.10.0.232:8777/api/local_doc_qa/delete_files'

http://es-container-local:9200这个链接一看就是错,还有milvus的容器,用docker ps看,是没有暴露端口的,我也想不明白为什么

@yangpeng666
Copy link

我也遇到了同样的问题,请问如何解决

@yangpeng666
Copy link

我也遇到了一样的问题了,INFO:debug_logger:milvus delete files_id: ['804d5229f04d4fdbb76fa565a53aa994'] INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(**http://es-container-local:9200**)>%3E) (force=False) INFO:elastic_transport.transport:HEAD http://es-container-local:9200/zzp%2B%2Bkb726f380c94ac4583b4176cc113654544 [status:200 duration:0.002s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has been marked alive after a successful request INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(http://es-container-local:9200)>%3E) (force=False) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb726f380c94ac4583b4176cc113654544/_search [status:503 duration:0.002s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 1 times in a row, putting on 1 second timeout WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 0 of 3) INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(http://es-container-local:9200)>%3E) (force=False) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb726f380c94ac4583b4176cc113654544/_search [status:503 duration:0.001s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 2 times in a row, putting on 2 second timeout WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 1 of 3) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb726f380c94ac4583b4176cc113654544/_search [status:503 duration:0.001s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 3 times in a row, putting on 4 second timeout WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 2 of 3) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb726f380c94ac4583b4176cc113654544/_search [status:503 duration:0.002s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 4 times in a row, putting on 8 second timeout [2024-07-10 14:51:31 +0800] [678] [ERROR] Exception occurred while handling uri: 'http://10.10.0.232:8777/api/local_doc_qa/delete_files'

http://es-container-local:9200这个链接一看就是错,还有milvus的容器,用docker ps看,是没有暴露端口的,我也想不明白为什么

是否已有关于该错误的issue或讨论? | Is there an existing issue / discussion for this?

* [x]  我已经搜索过已有的issues和讨论 | I have searched the existing issues / discussions

该问题是否在FAQ中有解答? | Is there an existing answer for this in FAQ?

* [x]  我已经搜索过FAQ | I have searched FAQ

当前行为 | Current Behavior

按照5.17日的安装文档,docker部署,run.sh安装未报错,启动容器终端输出未见异常,访问UI可以正常访问,遇见两个问题: 1.无法正常上传文件,上传第一个文件一直报错:milvus插入失败,请稍后再试,第二个第三个没问题,但出现该问题后,该知识库无法正常删除,报错请求失败,该文件同样无法删除,docx,ppt,txt均尝试了。 2.无法正常问答,问答一直报错请刷新后重试。

测试了ollama,本地Qwen,应该不是模型问题,看sanic.log,问题应该出在es上面。

期望行为 | Expected Behavior

解决BUG

运行环境 | Environment

- OS:Centos8
- NVIDIA Driver:535
- CUDA:11.8
- docker:2.24.0
- docker-compose:2.27.0
- NVIDIA GPU:4090
- NVIDIA GPU Memory:24GB

QAnything日志 | QAnything logs

新建知识库时应该还是正常的:

INFO:debug_logger:check_user_exist [] INFO:debug_logger:list_kbs zzp INFO:debug_logger:all kb infos: [{'kb_id': 'KB7e4842c2615c40649e78754c8e39203b', 'kb_name': '默认知识库'}] INFO:elastic_transport.transport:GET http://es-container-local:9200/ [status:200 duration:0.002s] INFO:debug_logger:##ES## - success to connect to {'name': 'd9dec8e8aa4a', 'cluster_name': 'docker-cluster', 'cluster_uuid': 'mocOuywHSNa08Yp98Kcy7A', 'version': {'number': '8.11.4', 'build_flavor': 'default', 'build_type': 'docker', 'build_hash': 'da06c53fd49b7e676ccf8a32d6655c5155c16d81', 'build_date': '2024-01-08T10:05:08.438562403Z', 'build_snapshot': False, 'lucene_version': '9.8.0', 'minimum_wire_compatibility_version': '7.17.0', 'minimum_index_compatibility_version': '7.0.0'}, 'tagline': 'You Know, for Search'}

随后在插入文件时发现有一个es的404(已用########突出):

INFO:debug_logger:XXX-wxs.pptx, cd4eca5871ac4bb7a43fb7ca0058815d, success INFO:debug_logger:success init localfile XXX-wxs.pptx INFO:debug_logger:insert_files_to_milvus: KB6fda41a1c95f4605ab812263d921740a INFO:debug_logger:collection zzp exists INFO:debug_logger:partitions: ['KB6fda41a1c95f4605ab812263d921740a'] INFO:elastic_transport.transport:GET http://es-container-local:9200/ [status:200 duration:0.002s] INFO:debug_logger:##ES## - success to connect to {'name': 'd9dec8e8aa4a', 'cluster_name': 'docker-cluster', 'cluster_uuid': 'mocOuywHSNa08Yp98Kcy7A', 'version': {'number': '8.11.4', 'build_flavor': 'default', 'build_type': 'docker', 'build_hash': 'da06c53fd49b7e676ccf8a32d6655c5155c16d81', 'build_date': '2024-01-08T10:05:08.438562403Z', 'build_snapshot': False, 'lucene_version': '9.8.0', 'minimum_wire_compatibility_version': '7.17.0', 'minimum_index_compatibility_version': '7.0.0'}, 'tagline': 'You Know, for Search'} INFO:debug_logger:before 2nd split doc lens: 53 INFO:debug_logger:after 2nd split doc lens: 44 INFO:debug_logger:langchain analysis content head: XXX INFO:debug_logger:split time: 0.3504297733306885 44 INFO:debug_logger:embedding number: 3

0%| | 0/3 [00:00<?, ?it/s] 33%|███▎ | 1/3 [00:00<00:00, 8.08it/s] 100%|██████████| 3/3 [00:00<00:00, 24.23it/s] INFO:debug_logger:embedding time: 0.12532377243041992 44 INFO:debug_logger:now inser_file XXX-wxs.pptx INFO:debug_logger:Inserting into Milvus... INFO:debug_logger:XXX-wxs.pptx (insert count: 44, delete count: 0, upsert count: 0, timestamp: 450132530566529026, success count: 44, err count: 0) INFO:debug_logger:now inser_file for es: XXX-wxs.pptx INFO:debug_logger:Inserting into es ... #############INFO:elastic_transport.transport:HEAD http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a [status:404 duration:0.003s] INFO:debug_logger:##ES## - Creating index zzp++kb6fda41a1c95f4605ab812263d921740a with: mappings: {'properties': {'file_id': {'type': 'keyword', 'index': True}, 'content': {'type': 'text', 'similarity': 'custom_bm25', 'index': True, 'analyzer': 'ik_smart', 'search_analyzer': 'ik_smart'}}} settings: {'index': {'similarity': {'custom_bm25': {'type': 'BM25', 'k1': '1.3', 'b': '0.6'}}}} INFO:elastic_transport.transport:PUT http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a [status:N/A duration:10.010s] ##############WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 1 times in a row, putting on 1 second timeout ERROR:debug_logger:ES insert file_id: cd4eca5871ac4bb7a43fb7ca0058815d file_name: XXX-wxs.pptx failed: Connection timed out INFO:debug_logger:insert time: 10.019953966140747 INFO:debug_logger:insert_to_milvus: success num: 0, failed num: 1 INFO:debug_logger:upload_files zzp INFO:debug_logger:mode: strong WARNING:root:missing use_local_file in request INFO:debug_logger:check_kb_exist [('KB6fda41a1c95f4605ab812263d921740a',)]

第二次再次插入一个文档,前端UI显示成功,但日志中有一堆warning:

INFO:debug_logger:ori name: 离线部署centos文档.docx INFO:debug_logger:decode name: 离线部署centos文档.docx INFO:debug_logger:cleaned name: 离线部署centos文档.docx INFO:debug_logger:check_user_exist [('zzp',)] INFO:debug_logger:check_kb_exist [('KB6fda41a1c95f4605ab812263d921740a',)] INFO:debug_logger:add_file: bbd20c38f53b473babbafe01f5aab6b7 INFO:debug_logger:离线部署centos文档.docx, bbd20c38f53b473babbafe01f5aab6b7, success INFO:debug_logger:success init localfile 离线部署centos文档.docx INFO:debug_logger:insert_files_to_milvus: KB6fda41a1c95f4605ab812263d921740a INFO:debug_logger:match milvus_client: <qanything_kernel.connector.database.milvus.milvus_client.MilvusClient object at 0x7f37c10e3f70> INFO:debug_logger:before 2nd split doc lens: 170 INFO:debug_logger:after 2nd split doc lens: 170 INFO:debug_logger:langchain analysis content head: 搭建私有yum仓库,安装gcc gcc-c++,ffmpeg(三步法),nginx(nginx在centos8. INFO:debug_logger:split time: 0.04621243476867676 170 INFO:debug_logger:embedding number: 11

0%| | 0/11 [00:00<?, ?it/s] 100%|██████████| 11/11 [00:00<00:00, 286.36it/s] INFO:debug_logger:embedding time: 0.04653787612915039 170 INFO:debug_logger:now inser_file 离线部署centos文档.docx INFO:debug_logger:Inserting into Milvus... INFO:debug_logger:离线部署centos文档.docx (insert count: 170, delete count: 0, upsert count: 0, timestamp: 450132533227028483, success count: 170, err count: 0) INFO:debug_logger:now inser_file for es: 离线部署centos文档.docx INFO:debug_logger:Inserting into es ... INFO:elastic_transport.transport:HEAD http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a [status:200 duration:0.003s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has been marked alive after a successful request INFO:debug_logger:##ES## - Index zzp++kb6fda41a1c95f4605ab812263d921740a already exists. Skipping creation. INFO:debug_logger:list_docs zzp INFO:debug_logger:kb_id: KB7e4842c2615c40649e78754c8e39203b INFO:debug_logger:delete_knowledge_base zzp INFO:debug_logger:check_kb_exist [('KB7e4842c2615c40649e78754c8e39203b',)] INFO:debug_logger:collection zzp exists INFO:debug_logger:partitions: ['KB7e4842c2615c40649e78754c8e39203b'] INFO:elastic_transport.transport:GET http://es-container-local:9200/ [status:200 duration:0.002s] INFO:debug_logger:##ES## - success to connect to {'name': 'd9dec8e8aa4a', 'cluster_name': 'docker-cluster', 'cluster_uuid': 'mocOuywHSNa08Yp98Kcy7A', 'version': {'number': '8.11.4', 'build_flavor': 'default', 'build_type': 'docker', 'build_hash': 'da06c53fd49b7e676ccf8a32d6655c5155c16d81', 'build_date': '2024-01-08T10:05:08.438562403Z', 'build_snapshot': False, 'lucene_version': '9.8.0', 'minimum_wire_compatibility_version': '7.17.0', 'minimum_index_compatibility_version': '7.0.0'}, 'tagline': 'You Know, for Search'} INFO:elastic_transport.transport:DELETE http://es-container-local:9200/zzp%2B%2Bkb7e4842c2615c40649e78754c8e39203b?ignore_unavailable=true [status:200 duration:0.001s] INFO:debug_logger:##ES## - success to delete index: ['zzp++kb7e4842c2615c40649e78754c8e39203b'] INFO:debug_logger:##ES## - success delete kb_ids: KB7e4842c2615c40649e78754c8e39203b INFO:debug_logger:delete_knowledge_base: ['KB7e4842c2615c40649e78754c8e39203b'] INFO:debug_logger:list_kbs zzp INFO:debug_logger:all kb infos: [{'kb_id': 'KB6fda41a1c95f4605ab812263d921740a', 'kb_name': 'aaa'}] INFO:elastic_transport.transport:PUT http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_bulk?refresh=false [status:N/A duration:10.010s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 1 times in a row, putting on 1 second timeout INFO:debug_logger:离线部署centos文档.docx Error adding texts: Connection timed out INFO:debug_logger:insert time: 10.025246858596802 INFO:debug_logger:insert_to_milvus: success num: 1, failed num: 0 INFO:debug_logger:list_docs zzp INFO:debug_logger:kb_id: KB6fda41a1c95f4605ab812263d921740a INFO:debug_logger:list_docs zzp INFO:debug_logger:kb_id: KB6fda41a1c95f4605ab812263d921740a INFO:debug_logger:delete_docs zzp INFO:debug_logger:check_kb_exist [('KB6fda41a1c95f4605ab812263d921740a',)] INFO:debug_logger:check_file_exist [('cd4eca5871ac4bb7a43fb7ca0058815d', 'yellow')] INFO:debug_logger:match milvus_client: <qanything_kernel.connector.database.milvus.milvus_client.MilvusClient object at 0x7f37c10e3f70> INFO:debug_logger:milvus delete files_id: ['cd4eca5871ac4bb7a43fb7ca0058815d'] INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(http://es-container-local:9200)>%3E) (force=False) INFO:elastic_transport.transport:HEAD http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a [status:200 duration:0.002s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has been marked alive after a successful request INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(http://es-container-local:9200)>%3E) (force=False) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.006s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 1 times in a row, putting on 1 second timeout WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 0 of 3) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.001s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 2 times in a row, putting on 2 second timeout WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 1 of 3) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.001s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 3 times in a row, putting on 4 second timeout WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 2 of 3) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.001s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 4 times in a row, putting on 8 second timeout

随后删除报错log如下:

[2024-05-31 09:35:57 +0800] [861] [ERROR] Exception occurred while handling uri: 'http://localhost:8777/api/local_doc_qa/delete_files' Traceback (most recent call last): File "handle_request", line 97, in handle_request File "/workspace/qanything_local/qanything_kernel/qanything_server/handler.py", line 247, in delete_docs milvus_kb.delete_files(file_ids) File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/milvus_client.py", line 288, in delete_files es_records = self.client.search(files_id, field='file_id') File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/es_client.py", line 182, in search response = self.client.search( File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/utils.py", line 446, in wrapped return api(*args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/init.py", line 3836, in search return self.perform_request( # type: ignore[return-value] File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/_base.py", line 320, in perform_request raise HTTP_EXCEPTIONS.get(meta.status, ApiError)( elasticsearch.ApiError: ApiError(503, 'search_phase_execution_exception', None) ERROR:sanic.error:Exception occurred while handling uri: 'http://localhost:8777/api/local_doc_qa/delete_files' Traceback (most recent call last): File "handle_request", line 97, in handle_request File "/workspace/qanything_local/qanything_kernel/qanything_server/handler.py", line 247, in delete_docs milvus_kb.delete_files(file_ids) File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/milvus_client.py", line 288, in delete_files es_records = self.client.search(files_id, field='file_id') File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/es_client.py", line 182, in search response = self.client.search( File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/utils.py", line 446, in wrapped return api(*args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/init.py", line 3836, in search return self.perform_request( # type: ignore[return-value] File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/_base.py", line 320, in perform_request raise HTTP_EXCEPTIONS.get(meta.status, ApiError)( elasticsearch.ApiError: ApiError(503, 'search_phase_execution_exception', None)

尝试问答,报错log如下:

INFO:debug_logger:local_doc_chat zzp INFO:debug_logger:rerank True INFO:debug_logger:history: [] INFO:debug_logger:question: 我想知道 INFO:debug_logger:kb_ids: ['KB6fda41a1c95f4605ab812263d921740a'] INFO:debug_logger:user_id: zzp INFO:debug_logger:check_kb_exist [('KB6fda41a1c95f4605ab812263d921740a',)] INFO:debug_logger:match milvus_client: <qanything_kernel.connector.database.milvus.milvus_client.MilvusClient object at 0x7f37c10e3f70> INFO:debug_logger:streaming: True INFO:debug_logger:start generate answer INFO:debug_logger:start generate... INFO:debug_logger:embedding number: 1 函数 get_knowledge_based_answer 执行耗时: 7.152557373046875e-07 秒

0%| | 0/1 [00:00<?, ?it/s] 100%|██████████| 1/1 [00:00<00:00, 52.80it/s] INFO:debug_logger:milvus group number: 1 INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(http://es-container-local:9200)>%3E) (force=False) INFO:elastic_transport.transport:HEAD http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a [status:200 duration:0.001s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has been marked alive after a successful request INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(http://es-container-local:9200)>%3E) (force=False) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.002s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 1 times in a row, putting on 1 second timeout WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 0 of 3) INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(http://es-container-local:9200)>%3E) (force=False) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.001s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 2 times in a row, putting on 2 second timeout WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 1 of 3) INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(http://es-container-local:9200)>%3E) (force=False) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.001s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 3 times in a row, putting on 4 second timeout WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 2 of 3) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.001s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 4 times in a row, putting on 8 second timeout [2024-05-31 09:36:07 +0800] [861] [ERROR] Exception occurred while handling uri: 'http://localhost:8777/api/local_doc_qa/local_doc_chat' Traceback (most recent call last): File "handle_request", line 132, in handle_request "_asgi_lifespan", File "/usr/local/lib/python3.10/dist-packages/sanic/response/types.py", line 547, in stream await self.streaming_fn(self) File "/workspace/qanything_local/qanything_kernel/qanything_server/handler.py", line 355, in generate_answer for resp, next_history in local_doc_qa.get_knowledge_based_answer( File "/workspace/qanything_local/qanything_kernel/core/local_doc_qa.py", line 225, in get_knowledge_based_answer source_documents = self.get_source_documents(retrieval_queries, milvus_kb) File "/workspace/qanything_local/qanything_kernel/core/local_doc_qa.py", line 133, in get_source_documents batch_result = milvus_kb.search_emb_async(embs=embs, top_k=top_k, queries=queries) File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/milvus_client.py", line 181, in search_emb_async return future.result() File "/usr/lib/python3.10/concurrent/futures/_base.py", line 458, in result return self.__get_result() File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result raise self._exception File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, **self.kwargs) File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/milvus_client.py", line 170, in __search_emb_sync es_records = self.client.search(queries) File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/es_client.py", line 182, in search response = self.client.search( File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/utils.py", line 446, in wrapped return api(*args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/init.py", line 3836, in search return self.perform_request( # type: ignore[return-value] File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/_base.py", line 320, in perform_request raise HTTP_EXCEPTIONS.get(meta.status, ApiError)( elasticsearch.ApiError: ApiError(503, 'search_phase_execution_exception', None) ERROR:sanic.error:Exception occurred while handling uri: 'http://localhost:8777/api/local_doc_qa/local_doc_chat' Traceback (most recent call last): File "handle_request", line 132, in handle_request "_asgi_lifespan", File "/usr/local/lib/python3.10/dist-packages/sanic/response/types.py", line 547, in stream await self.streaming_fn(self) File "/workspace/qanything_local/qanything_kernel/qanything_server/handler.py", line 355, in generate_answer for resp, next_history in local_doc_qa.get_knowledge_based_answer( File "/workspace/qanything_local/qanything_kernel/core/local_doc_qa.py", line 225, in get_knowledge_based_answer source_documents = self.get_source_documents(retrieval_queries, milvus_kb) File "/workspace/qanything_local/qanything_kernel/core/local_doc_qa.py", line 133, in get_source_documents batch_result = milvus_kb.search_emb_async(embs=embs, top_k=top_k, queries=queries) File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/milvus_client.py", line 181, in search_emb_async return future.result() File "/usr/lib/python3.10/concurrent/futures/_base.py", line 458, in result return self.__get_result() File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result raise self._exception File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, **self.kwargs) File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/milvus_client.py", line 170, in __search_emb_sync es_records = self.client.search(queries) File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/es_client.py", line 182, in search response = self.client.search( File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/utils.py", line 446, in wrapped return api(*args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/init.py", line 3836, in search return self.perform_request( # type: ignore[return-value] File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/_base.py", line 320, in perform_request raise HTTP_EXCEPTIONS.get(meta.status, ApiError)( elasticsearch.ApiError: ApiError(503, 'search_phase_execution_exception', None)

复现方法 | Steps To Reproduce

按照文档下载源码,bash run.sh -c cloud -i 0 下载镜像,起容器,出现上述问题

备注 | Anything else?

No response

是否已有关于该错误的issue或讨论? | Is there an existing issue / discussion for this?

* [x]  我已经搜索过已有的issues和讨论 | I have searched the existing issues / discussions

该问题是否在FAQ中有解答? | Is there an existing answer for this in FAQ?

* [x]  我已经搜索过FAQ | I have searched FAQ

当前行为 | Current Behavior

按照5.17日的安装文档,docker部署,run.sh安装未报错,启动容器终端输出未见异常,访问UI可以正常访问,遇见两个问题: 1.无法正常上传文件,上传第一个文件一直报错:milvus插入失败,请稍后再试,第二个第三个没问题,但出现该问题后,该知识库无法正常删除,报错请求失败,该文件同样无法删除,docx,ppt,txt均尝试了。 2.无法正常问答,问答一直报错请刷新后重试。

测试了ollama,本地Qwen,应该不是模型问题,看sanic.log,问题应该出在es上面。

期望行为 | Expected Behavior

解决BUG

运行环境 | Environment

- OS:Centos8
- NVIDIA Driver:535
- CUDA:11.8
- docker:2.24.0
- docker-compose:2.27.0
- NVIDIA GPU:4090
- NVIDIA GPU Memory:24GB

QAnything日志 | QAnything logs

新建知识库时应该还是正常的:

INFO:debug_logger:check_user_exist [] INFO:debug_logger:list_kbs zzp INFO:debug_logger:all kb infos: [{'kb_id': 'KB7e4842c2615c40649e78754c8e39203b', 'kb_name': '默认知识库'}] INFO:elastic_transport.transport:GET http://es-container-local:9200/ [status:200 duration:0.002s] INFO:debug_logger:##ES## - success to connect to {'name': 'd9dec8e8aa4a', 'cluster_name': 'docker-cluster', 'cluster_uuid': 'mocOuywHSNa08Yp98Kcy7A', 'version': {'number': '8.11.4', 'build_flavor': 'default', 'build_type': 'docker', 'build_hash': 'da06c53fd49b7e676ccf8a32d6655c5155c16d81', 'build_date': '2024-01-08T10:05:08.438562403Z', 'build_snapshot': False, 'lucene_version': '9.8.0', 'minimum_wire_compatibility_version': '7.17.0', 'minimum_index_compatibility_version': '7.0.0'}, 'tagline': 'You Know, for Search'}

随后在插入文件时发现有一个es的404(已用########突出):

INFO:debug_logger:XXX-wxs.pptx, cd4eca5871ac4bb7a43fb7ca0058815d, success INFO:debug_logger:success init localfile XXX-wxs.pptx INFO:debug_logger:insert_files_to_milvus: KB6fda41a1c95f4605ab812263d921740a INFO:debug_logger:collection zzp exists INFO:debug_logger:partitions: ['KB6fda41a1c95f4605ab812263d921740a'] INFO:elastic_transport.transport:GET http://es-container-local:9200/ [status:200 duration:0.002s] INFO:debug_logger:##ES## - success to connect to {'name': 'd9dec8e8aa4a', 'cluster_name': 'docker-cluster', 'cluster_uuid': 'mocOuywHSNa08Yp98Kcy7A', 'version': {'number': '8.11.4', 'build_flavor': 'default', 'build_type': 'docker', 'build_hash': 'da06c53fd49b7e676ccf8a32d6655c5155c16d81', 'build_date': '2024-01-08T10:05:08.438562403Z', 'build_snapshot': False, 'lucene_version': '9.8.0', 'minimum_wire_compatibility_version': '7.17.0', 'minimum_index_compatibility_version': '7.0.0'}, 'tagline': 'You Know, for Search'} INFO:debug_logger:before 2nd split doc lens: 53 INFO:debug_logger:after 2nd split doc lens: 44 INFO:debug_logger:langchain analysis content head: XXX INFO:debug_logger:split time: 0.3504297733306885 44 INFO:debug_logger:embedding number: 3

0%| | 0/3 [00:00<?, ?it/s] 33%|███▎ | 1/3 [00:00<00:00, 8.08it/s] 100%|██████████| 3/3 [00:00<00:00, 24.23it/s] INFO:debug_logger:embedding time: 0.12532377243041992 44 INFO:debug_logger:now inser_file XXX-wxs.pptx INFO:debug_logger:Inserting into Milvus... INFO:debug_logger:XXX-wxs.pptx (insert count: 44, delete count: 0, upsert count: 0, timestamp: 450132530566529026, success count: 44, err count: 0) INFO:debug_logger:now inser_file for es: XXX-wxs.pptx INFO:debug_logger:Inserting into es ... #############INFO:elastic_transport.transport:HEAD http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a [status:404 duration:0.003s] INFO:debug_logger:##ES## - Creating index zzp++kb6fda41a1c95f4605ab812263d921740a with: mappings: {'properties': {'file_id': {'type': 'keyword', 'index': True}, 'content': {'type': 'text', 'similarity': 'custom_bm25', 'index': True, 'analyzer': 'ik_smart', 'search_analyzer': 'ik_smart'}}} settings: {'index': {'similarity': {'custom_bm25': {'type': 'BM25', 'k1': '1.3', 'b': '0.6'}}}} INFO:elastic_transport.transport:PUT http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a [status:N/A duration:10.010s] ##############WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 1 times in a row, putting on 1 second timeout ERROR:debug_logger:ES insert file_id: cd4eca5871ac4bb7a43fb7ca0058815d file_name: XXX-wxs.pptx failed: Connection timed out INFO:debug_logger:insert time: 10.019953966140747 INFO:debug_logger:insert_to_milvus: success num: 0, failed num: 1 INFO:debug_logger:upload_files zzp INFO:debug_logger:mode: strong WARNING:root:missing use_local_file in request INFO:debug_logger:check_kb_exist [('KB6fda41a1c95f4605ab812263d921740a',)]

第二次再次插入一个文档,前端UI显示成功,但日志中有一堆warning:

INFO:debug_logger:ori name: 离线部署centos文档.docx INFO:debug_logger:decode name: 离线部署centos文档.docx INFO:debug_logger:cleaned name: 离线部署centos文档.docx INFO:debug_logger:check_user_exist [('zzp',)] INFO:debug_logger:check_kb_exist [('KB6fda41a1c95f4605ab812263d921740a',)] INFO:debug_logger:add_file: bbd20c38f53b473babbafe01f5aab6b7 INFO:debug_logger:离线部署centos文档.docx, bbd20c38f53b473babbafe01f5aab6b7, success INFO:debug_logger:success init localfile 离线部署centos文档.docx INFO:debug_logger:insert_files_to_milvus: KB6fda41a1c95f4605ab812263d921740a INFO:debug_logger:match milvus_client: <qanything_kernel.connector.database.milvus.milvus_client.MilvusClient object at 0x7f37c10e3f70> INFO:debug_logger:before 2nd split doc lens: 170 INFO:debug_logger:after 2nd split doc lens: 170 INFO:debug_logger:langchain analysis content head: 搭建私有yum仓库,安装gcc gcc-c++,ffmpeg(三步法),nginx(nginx在centos8. INFO:debug_logger:split time: 0.04621243476867676 170 INFO:debug_logger:embedding number: 11

0%| | 0/11 [00:00<?, ?it/s] 100%|██████████| 11/11 [00:00<00:00, 286.36it/s] INFO:debug_logger:embedding time: 0.04653787612915039 170 INFO:debug_logger:now inser_file 离线部署centos文档.docx INFO:debug_logger:Inserting into Milvus... INFO:debug_logger:离线部署centos文档.docx (insert count: 170, delete count: 0, upsert count: 0, timestamp: 450132533227028483, success count: 170, err count: 0) INFO:debug_logger:now inser_file for es: 离线部署centos文档.docx INFO:debug_logger:Inserting into es ... INFO:elastic_transport.transport:HEAD http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a [status:200 duration:0.003s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has been marked alive after a successful request INFO:debug_logger:##ES## - Index zzp++kb6fda41a1c95f4605ab812263d921740a already exists. Skipping creation. INFO:debug_logger:list_docs zzp INFO:debug_logger:kb_id: KB7e4842c2615c40649e78754c8e39203b INFO:debug_logger:delete_knowledge_base zzp INFO:debug_logger:check_kb_exist [('KB7e4842c2615c40649e78754c8e39203b',)] INFO:debug_logger:collection zzp exists INFO:debug_logger:partitions: ['KB7e4842c2615c40649e78754c8e39203b'] INFO:elastic_transport.transport:GET http://es-container-local:9200/ [status:200 duration:0.002s] INFO:debug_logger:##ES## - success to connect to {'name': 'd9dec8e8aa4a', 'cluster_name': 'docker-cluster', 'cluster_uuid': 'mocOuywHSNa08Yp98Kcy7A', 'version': {'number': '8.11.4', 'build_flavor': 'default', 'build_type': 'docker', 'build_hash': 'da06c53fd49b7e676ccf8a32d6655c5155c16d81', 'build_date': '2024-01-08T10:05:08.438562403Z', 'build_snapshot': False, 'lucene_version': '9.8.0', 'minimum_wire_compatibility_version': '7.17.0', 'minimum_index_compatibility_version': '7.0.0'}, 'tagline': 'You Know, for Search'} INFO:elastic_transport.transport:DELETE http://es-container-local:9200/zzp%2B%2Bkb7e4842c2615c40649e78754c8e39203b?ignore_unavailable=true [status:200 duration:0.001s] INFO:debug_logger:##ES## - success to delete index: ['zzp++kb7e4842c2615c40649e78754c8e39203b'] INFO:debug_logger:##ES## - success delete kb_ids: KB7e4842c2615c40649e78754c8e39203b INFO:debug_logger:delete_knowledge_base: ['KB7e4842c2615c40649e78754c8e39203b'] INFO:debug_logger:list_kbs zzp INFO:debug_logger:all kb infos: [{'kb_id': 'KB6fda41a1c95f4605ab812263d921740a', 'kb_name': 'aaa'}] INFO:elastic_transport.transport:PUT http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_bulk?refresh=false [status:N/A duration:10.010s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 1 times in a row, putting on 1 second timeout INFO:debug_logger:离线部署centos文档.docx Error adding texts: Connection timed out INFO:debug_logger:insert time: 10.025246858596802 INFO:debug_logger:insert_to_milvus: success num: 1, failed num: 0 INFO:debug_logger:list_docs zzp INFO:debug_logger:kb_id: KB6fda41a1c95f4605ab812263d921740a INFO:debug_logger:list_docs zzp INFO:debug_logger:kb_id: KB6fda41a1c95f4605ab812263d921740a INFO:debug_logger:delete_docs zzp INFO:debug_logger:check_kb_exist [('KB6fda41a1c95f4605ab812263d921740a',)] INFO:debug_logger:check_file_exist [('cd4eca5871ac4bb7a43fb7ca0058815d', 'yellow')] INFO:debug_logger:match milvus_client: <qanything_kernel.connector.database.milvus.milvus_client.MilvusClient object at 0x7f37c10e3f70> INFO:debug_logger:milvus delete files_id: ['cd4eca5871ac4bb7a43fb7ca0058815d'] INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(http://es-container-local:9200)>%3E) (force=False) INFO:elastic_transport.transport:HEAD http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a [status:200 duration:0.002s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has been marked alive after a successful request INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(http://es-container-local:9200)>%3E) (force=False) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.006s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 1 times in a row, putting on 1 second timeout WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 0 of 3) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.001s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 2 times in a row, putting on 2 second timeout WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 1 of 3) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.001s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 3 times in a row, putting on 4 second timeout WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 2 of 3) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.001s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 4 times in a row, putting on 8 second timeout

随后删除报错log如下:

[2024-05-31 09:35:57 +0800] [861] [ERROR] Exception occurred while handling uri: 'http://localhost:8777/api/local_doc_qa/delete_files' Traceback (most recent call last): File "handle_request", line 97, in handle_request File "/workspace/qanything_local/qanything_kernel/qanything_server/handler.py", line 247, in delete_docs milvus_kb.delete_files(file_ids) File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/milvus_client.py", line 288, in delete_files es_records = self.client.search(files_id, field='file_id') File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/es_client.py", line 182, in search response = self.client.search( File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/utils.py", line 446, in wrapped return api(*args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/init.py", line 3836, in search return self.perform_request( # type: ignore[return-value] File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/_base.py", line 320, in perform_request raise HTTP_EXCEPTIONS.get(meta.status, ApiError)( elasticsearch.ApiError: ApiError(503, 'search_phase_execution_exception', None) ERROR:sanic.error:Exception occurred while handling uri: 'http://localhost:8777/api/local_doc_qa/delete_files' Traceback (most recent call last): File "handle_request", line 97, in handle_request File "/workspace/qanything_local/qanything_kernel/qanything_server/handler.py", line 247, in delete_docs milvus_kb.delete_files(file_ids) File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/milvus_client.py", line 288, in delete_files es_records = self.client.search(files_id, field='file_id') File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/es_client.py", line 182, in search response = self.client.search( File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/utils.py", line 446, in wrapped return api(*args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/init.py", line 3836, in search return self.perform_request( # type: ignore[return-value] File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/_base.py", line 320, in perform_request raise HTTP_EXCEPTIONS.get(meta.status, ApiError)( elasticsearch.ApiError: ApiError(503, 'search_phase_execution_exception', None)

尝试问答,报错log如下:

INFO:debug_logger:local_doc_chat zzp INFO:debug_logger:rerank True INFO:debug_logger:history: [] INFO:debug_logger:question: 我想知道 INFO:debug_logger:kb_ids: ['KB6fda41a1c95f4605ab812263d921740a'] INFO:debug_logger:user_id: zzp INFO:debug_logger:check_kb_exist [('KB6fda41a1c95f4605ab812263d921740a',)] INFO:debug_logger:match milvus_client: <qanything_kernel.connector.database.milvus.milvus_client.MilvusClient object at 0x7f37c10e3f70> INFO:debug_logger:streaming: True INFO:debug_logger:start generate answer INFO:debug_logger:start generate... INFO:debug_logger:embedding number: 1 函数 get_knowledge_based_answer 执行耗时: 7.152557373046875e-07 秒

0%| | 0/1 [00:00<?, ?it/s] 100%|██████████| 1/1 [00:00<00:00, 52.80it/s] INFO:debug_logger:milvus group number: 1 INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(http://es-container-local:9200)>%3E) (force=False) INFO:elastic_transport.transport:HEAD http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a [status:200 duration:0.001s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has been marked alive after a successful request INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(http://es-container-local:9200)>%3E) (force=False) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.002s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 1 times in a row, putting on 1 second timeout WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 0 of 3) INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(http://es-container-local:9200)>%3E) (force=False) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.001s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 2 times in a row, putting on 2 second timeout WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 1 of 3) INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(http://es-container-local:9200)>%3E) (force=False) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.001s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 3 times in a row, putting on 4 second timeout WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 2 of 3) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.001s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 4 times in a row, putting on 8 second timeout [2024-05-31 09:36:07 +0800] [861] [ERROR] Exception occurred while handling uri: 'http://localhost:8777/api/local_doc_qa/local_doc_chat' Traceback (most recent call last): File "handle_request", line 132, in handle_request "_asgi_lifespan", File "/usr/local/lib/python3.10/dist-packages/sanic/response/types.py", line 547, in stream await self.streaming_fn(self) File "/workspace/qanything_local/qanything_kernel/qanything_server/handler.py", line 355, in generate_answer for resp, next_history in local_doc_qa.get_knowledge_based_answer( File "/workspace/qanything_local/qanything_kernel/core/local_doc_qa.py", line 225, in get_knowledge_based_answer source_documents = self.get_source_documents(retrieval_queries, milvus_kb) File "/workspace/qanything_local/qanything_kernel/core/local_doc_qa.py", line 133, in get_source_documents batch_result = milvus_kb.search_emb_async(embs=embs, top_k=top_k, queries=queries) File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/milvus_client.py", line 181, in search_emb_async return future.result() File "/usr/lib/python3.10/concurrent/futures/_base.py", line 458, in result return self.__get_result() File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result raise self._exception File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, **self.kwargs) File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/milvus_client.py", line 170, in __search_emb_sync es_records = self.client.search(queries) File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/es_client.py", line 182, in search response = self.client.search( File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/utils.py", line 446, in wrapped return api(*args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/init.py", line 3836, in search return self.perform_request( # type: ignore[return-value] File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/_base.py", line 320, in perform_request raise HTTP_EXCEPTIONS.get(meta.status, ApiError)( elasticsearch.ApiError: ApiError(503, 'search_phase_execution_exception', None) ERROR:sanic.error:Exception occurred while handling uri: 'http://localhost:8777/api/local_doc_qa/local_doc_chat' Traceback (most recent call last): File "handle_request", line 132, in handle_request "_asgi_lifespan", File "/usr/local/lib/python3.10/dist-packages/sanic/response/types.py", line 547, in stream await self.streaming_fn(self) File "/workspace/qanything_local/qanything_kernel/qanything_server/handler.py", line 355, in generate_answer for resp, next_history in local_doc_qa.get_knowledge_based_answer( File "/workspace/qanything_local/qanything_kernel/core/local_doc_qa.py", line 225, in get_knowledge_based_answer source_documents = self.get_source_documents(retrieval_queries, milvus_kb) File "/workspace/qanything_local/qanything_kernel/core/local_doc_qa.py", line 133, in get_source_documents batch_result = milvus_kb.search_emb_async(embs=embs, top_k=top_k, queries=queries) File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/milvus_client.py", line 181, in search_emb_async return future.result() File "/usr/lib/python3.10/concurrent/futures/_base.py", line 458, in result return self.__get_result() File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result raise self._exception File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, **self.kwargs) File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/milvus_client.py", line 170, in __search_emb_sync es_records = self.client.search(queries) File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/es_client.py", line 182, in search response = self.client.search( File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/utils.py", line 446, in wrapped return api(*args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/init.py", line 3836, in search return self.perform_request( # type: ignore[return-value] File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/_base.py", line 320, in perform_request raise HTTP_EXCEPTIONS.get(meta.status, ApiError)( elasticsearch.ApiError: ApiError(503, 'search_phase_execution_exception', None)

复现方法 | Steps To Reproduce

按照文档下载源码,bash run.sh -c cloud -i 0 下载镜像,起容器,出现上述问题

备注 | Anything else?

_No respons

是否已有关于该错误的issue或讨论? | Is there an existing issue / discussion for this?

* [x]  我已经搜索过已有的issues和讨论 | I have searched the existing issues / discussions

该问题是否在FAQ中有解答? | Is there an existing answer for this in FAQ?

* [x]  我已经搜索过FAQ | I have searched FAQ

当前行为 | Current Behavior

按照5.17日的安装文档,docker部署,run.sh安装未报错,启动容器终端输出未见异常,访问UI可以正常访问,遇见两个问题: 1.无法正常上传文件,上传第一个文件一直报错:milvus插入失败,请稍后再试,第二个第三个没问题,但出现该问题后,该知识库无法正常删除,报错请求失败,该文件同样无法删除,docx,ppt,txt均尝试了。 2.无法正常问答,问答一直报错请刷新后重试。

测试了ollama,本地Qwen,应该不是模型问题,看sanic.log,问题应该出在es上面。

期望行为 | Expected Behavior

解决BUG

运行环境 | Environment

- OS:Centos8
- NVIDIA Driver:535
- CUDA:11.8
- docker:2.24.0
- docker-compose:2.27.0
- NVIDIA GPU:4090
- NVIDIA GPU Memory:24GB

QAnything日志 | QAnything logs

新建知识库时应该还是正常的:

INFO:debug_logger:check_user_exist [] INFO:debug_logger:list_kbs zzp INFO:debug_logger:all kb infos: [{'kb_id': 'KB7e4842c2615c40649e78754c8e39203b', 'kb_name': '默认知识库'}] INFO:elastic_transport.transport:GET http://es-container-local:9200/ [status:200 duration:0.002s] INFO:debug_logger:##ES## - success to connect to {'name': 'd9dec8e8aa4a', 'cluster_name': 'docker-cluster', 'cluster_uuid': 'mocOuywHSNa08Yp98Kcy7A', 'version': {'number': '8.11.4', 'build_flavor': 'default', 'build_type': 'docker', 'build_hash': 'da06c53fd49b7e676ccf8a32d6655c5155c16d81', 'build_date': '2024-01-08T10:05:08.438562403Z', 'build_snapshot': False, 'lucene_version': '9.8.0', 'minimum_wire_compatibility_version': '7.17.0', 'minimum_index_compatibility_version': '7.0.0'}, 'tagline': 'You Know, for Search'}

随后在插入文件时发现有一个es的404(已用########突出):

INFO:debug_logger:XXX-wxs.pptx, cd4eca5871ac4bb7a43fb7ca0058815d, success INFO:debug_logger:success init localfile XXX-wxs.pptx INFO:debug_logger:insert_files_to_milvus: KB6fda41a1c95f4605ab812263d921740a INFO:debug_logger:collection zzp exists INFO:debug_logger:partitions: ['KB6fda41a1c95f4605ab812263d921740a'] INFO:elastic_transport.transport:GET http://es-container-local:9200/ [status:200 duration:0.002s] INFO:debug_logger:##ES## - success to connect to {'name': 'd9dec8e8aa4a', 'cluster_name': 'docker-cluster', 'cluster_uuid': 'mocOuywHSNa08Yp98Kcy7A', 'version': {'number': '8.11.4', 'build_flavor': 'default', 'build_type': 'docker', 'build_hash': 'da06c53fd49b7e676ccf8a32d6655c5155c16d81', 'build_date': '2024-01-08T10:05:08.438562403Z', 'build_snapshot': False, 'lucene_version': '9.8.0', 'minimum_wire_compatibility_version': '7.17.0', 'minimum_index_compatibility_version': '7.0.0'}, 'tagline': 'You Know, for Search'} INFO:debug_logger:before 2nd split doc lens: 53 INFO:debug_logger:after 2nd split doc lens: 44 INFO:debug_logger:langchain analysis content head: XXX INFO:debug_logger:split time: 0.3504297733306885 44 INFO:debug_logger:embedding number: 3

0%| | 0/3 [00:00<?, ?it/s] 33%|███▎ | 1/3 [00:00<00:00, 8.08it/s] 100%|██████████| 3/3 [00:00<00:00, 24.23it/s] INFO:debug_logger:embedding time: 0.12532377243041992 44 INFO:debug_logger:now inser_file XXX-wxs.pptx INFO:debug_logger:Inserting into Milvus... INFO:debug_logger:XXX-wxs.pptx (insert count: 44, delete count: 0, upsert count: 0, timestamp: 450132530566529026, success count: 44, err count: 0) INFO:debug_logger:now inser_file for es: XXX-wxs.pptx INFO:debug_logger:Inserting into es ... #############INFO:elastic_transport.transport:HEAD http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a [status:404 duration:0.003s] INFO:debug_logger:##ES## - Creating index zzp++kb6fda41a1c95f4605ab812263d921740a with: mappings: {'properties': {'file_id': {'type': 'keyword', 'index': True}, 'content': {'type': 'text', 'similarity': 'custom_bm25', 'index': True, 'analyzer': 'ik_smart', 'search_analyzer': 'ik_smart'}}} settings: {'index': {'similarity': {'custom_bm25': {'type': 'BM25', 'k1': '1.3', 'b': '0.6'}}}} INFO:elastic_transport.transport:PUT http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a [status:N/A duration:10.010s] ##############WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 1 times in a row, putting on 1 second timeout ERROR:debug_logger:ES insert file_id: cd4eca5871ac4bb7a43fb7ca0058815d file_name: XXX-wxs.pptx failed: Connection timed out INFO:debug_logger:insert time: 10.019953966140747 INFO:debug_logger:insert_to_milvus: success num: 0, failed num: 1 INFO:debug_logger:upload_files zzp INFO:debug_logger:mode: strong WARNING:root:missing use_local_file in request INFO:debug_logger:check_kb_exist [('KB6fda41a1c95f4605ab812263d921740a',)]

第二次再次插入一个文档,前端UI显示成功,但日志中有一堆warning:

INFO:debug_logger:ori name: 离线部署centos文档.docx INFO:debug_logger:decode name: 离线部署centos文档.docx INFO:debug_logger:cleaned name: 离线部署centos文档.docx INFO:debug_logger:check_user_exist [('zzp',)] INFO:debug_logger:check_kb_exist [('KB6fda41a1c95f4605ab812263d921740a',)] INFO:debug_logger:add_file: bbd20c38f53b473babbafe01f5aab6b7 INFO:debug_logger:离线部署centos文档.docx, bbd20c38f53b473babbafe01f5aab6b7, success INFO:debug_logger:success init localfile 离线部署centos文档.docx INFO:debug_logger:insert_files_to_milvus: KB6fda41a1c95f4605ab812263d921740a INFO:debug_logger:match milvus_client: <qanything_kernel.connector.database.milvus.milvus_client.MilvusClient object at 0x7f37c10e3f70> INFO:debug_logger:before 2nd split doc lens: 170 INFO:debug_logger:after 2nd split doc lens: 170 INFO:debug_logger:langchain analysis content head: 搭建私有yum仓库,安装gcc gcc-c++,ffmpeg(三步法),nginx(nginx在centos8. INFO:debug_logger:split time: 0.04621243476867676 170 INFO:debug_logger:embedding number: 11

0%| | 0/11 [00:00<?, ?it/s] 100%|██████████| 11/11 [00:00<00:00, 286.36it/s] INFO:debug_logger:embedding time: 0.04653787612915039 170 INFO:debug_logger:now inser_file 离线部署centos文档.docx INFO:debug_logger:Inserting into Milvus... INFO:debug_logger:离线部署centos文档.docx (insert count: 170, delete count: 0, upsert count: 0, timestamp: 450132533227028483, success count: 170, err count: 0) INFO:debug_logger:now inser_file for es: 离线部署centos文档.docx INFO:debug_logger:Inserting into es ... INFO:elastic_transport.transport:HEAD http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a [status:200 duration:0.003s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has been marked alive after a successful request INFO:debug_logger:##ES## - Index zzp++kb6fda41a1c95f4605ab812263d921740a already exists. Skipping creation. INFO:debug_logger:list_docs zzp INFO:debug_logger:kb_id: KB7e4842c2615c40649e78754c8e39203b INFO:debug_logger:delete_knowledge_base zzp INFO:debug_logger:check_kb_exist [('KB7e4842c2615c40649e78754c8e39203b',)] INFO:debug_logger:collection zzp exists INFO:debug_logger:partitions: ['KB7e4842c2615c40649e78754c8e39203b'] INFO:elastic_transport.transport:GET http://es-container-local:9200/ [status:200 duration:0.002s] INFO:debug_logger:##ES## - success to connect to {'name': 'd9dec8e8aa4a', 'cluster_name': 'docker-cluster', 'cluster_uuid': 'mocOuywHSNa08Yp98Kcy7A', 'version': {'number': '8.11.4', 'build_flavor': 'default', 'build_type': 'docker', 'build_hash': 'da06c53fd49b7e676ccf8a32d6655c5155c16d81', 'build_date': '2024-01-08T10:05:08.438562403Z', 'build_snapshot': False, 'lucene_version': '9.8.0', 'minimum_wire_compatibility_version': '7.17.0', 'minimum_index_compatibility_version': '7.0.0'}, 'tagline': 'You Know, for Search'} INFO:elastic_transport.transport:DELETE http://es-container-local:9200/zzp%2B%2Bkb7e4842c2615c40649e78754c8e39203b?ignore_unavailable=true [status:200 duration:0.001s] INFO:debug_logger:##ES## - success to delete index: ['zzp++kb7e4842c2615c40649e78754c8e39203b'] INFO:debug_logger:##ES## - success delete kb_ids: KB7e4842c2615c40649e78754c8e39203b INFO:debug_logger:delete_knowledge_base: ['KB7e4842c2615c40649e78754c8e39203b'] INFO:debug_logger:list_kbs zzp INFO:debug_logger:all kb infos: [{'kb_id': 'KB6fda41a1c95f4605ab812263d921740a', 'kb_name': 'aaa'}] INFO:elastic_transport.transport:PUT http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_bulk?refresh=false [status:N/A duration:10.010s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 1 times in a row, putting on 1 second timeout INFO:debug_logger:离线部署centos文档.docx Error adding texts: Connection timed out INFO:debug_logger:insert time: 10.025246858596802 INFO:debug_logger:insert_to_milvus: success num: 1, failed num: 0 INFO:debug_logger:list_docs zzp INFO:debug_logger:kb_id: KB6fda41a1c95f4605ab812263d921740a INFO:debug_logger:list_docs zzp INFO:debug_logger:kb_id: KB6fda41a1c95f4605ab812263d921740a INFO:debug_logger:delete_docs zzp INFO:debug_logger:check_kb_exist [('KB6fda41a1c95f4605ab812263d921740a',)] INFO:debug_logger:check_file_exist [('cd4eca5871ac4bb7a43fb7ca0058815d', 'yellow')] INFO:debug_logger:match milvus_client: <qanything_kernel.connector.database.milvus.milvus_client.MilvusClient object at 0x7f37c10e3f70> INFO:debug_logger:milvus delete files_id: ['cd4eca5871ac4bb7a43fb7ca0058815d'] INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(http://es-container-local:9200)>%3E) (force=False) INFO:elastic_transport.transport:HEAD http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a [status:200 duration:0.002s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has been marked alive after a successful request INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(http://es-container-local:9200)>%3E) (force=False) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.006s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 1 times in a row, putting on 1 second timeout WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 0 of 3) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.001s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 2 times in a row, putting on 2 second timeout WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 1 of 3) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.001s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 3 times in a row, putting on 4 second timeout WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 2 of 3) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.001s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 4 times in a row, putting on 8 second timeout

随后删除报错log如下:

[2024-05-31 09:35:57 +0800] [861] [ERROR] Exception occurred while handling uri: 'http://localhost:8777/api/local_doc_qa/delete_files' Traceback (most recent call last): File "handle_request", line 97, in handle_request File "/workspace/qanything_local/qanything_kernel/qanything_server/handler.py", line 247, in delete_docs milvus_kb.delete_files(file_ids) File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/milvus_client.py", line 288, in delete_files es_records = self.client.search(files_id, field='file_id') File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/es_client.py", line 182, in search response = self.client.search( File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/utils.py", line 446, in wrapped return api(*args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/init.py", line 3836, in search return self.perform_request( # type: ignore[return-value] File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/_base.py", line 320, in perform_request raise HTTP_EXCEPTIONS.get(meta.status, ApiError)( elasticsearch.ApiError: ApiError(503, 'search_phase_execution_exception', None) ERROR:sanic.error:Exception occurred while handling uri: 'http://localhost:8777/api/local_doc_qa/delete_files' Traceback (most recent call last): File "handle_request", line 97, in handle_request File "/workspace/qanything_local/qanything_kernel/qanything_server/handler.py", line 247, in delete_docs milvus_kb.delete_files(file_ids) File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/milvus_client.py", line 288, in delete_files es_records = self.client.search(files_id, field='file_id') File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/es_client.py", line 182, in search response = self.client.search( File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/utils.py", line 446, in wrapped return api(*args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/init.py", line 3836, in search return self.perform_request( # type: ignore[return-value] File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/_base.py", line 320, in perform_request raise HTTP_EXCEPTIONS.get(meta.status, ApiError)( elasticsearch.ApiError: ApiError(503, 'search_phase_execution_exception', None)

尝试问答,报错log如下:

INFO:debug_logger:local_doc_chat zzp INFO:debug_logger:rerank True INFO:debug_logger:history: [] INFO:debug_logger:question: 我想知道 INFO:debug_logger:kb_ids: ['KB6fda41a1c95f4605ab812263d921740a'] INFO:debug_logger:user_id: zzp INFO:debug_logger:check_kb_exist [('KB6fda41a1c95f4605ab812263d921740a',)] INFO:debug_logger:match milvus_client: <qanything_kernel.connector.database.milvus.milvus_client.MilvusClient object at 0x7f37c10e3f70> INFO:debug_logger:streaming: True INFO:debug_logger:start generate answer INFO:debug_logger:start generate... INFO:debug_logger:embedding number: 1 函数 get_knowledge_based_answer 执行耗时: 7.152557373046875e-07 秒

0%| | 0/1 [00:00<?, ?it/s] 100%|██████████| 1/1 [00:00<00:00, 52.80it/s] INFO:debug_logger:milvus group number: 1 INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(http://es-container-local:9200)>%3E) (force=False) INFO:elastic_transport.transport:HEAD http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a [status:200 duration:0.001s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has been marked alive after a successful request INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(http://es-container-local:9200)>%3E) (force=False) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.002s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 1 times in a row, putting on 1 second timeout WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 0 of 3) INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(http://es-container-local:9200)>%3E) (force=False) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.001s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 2 times in a row, putting on 2 second timeout WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 1 of 3) INFO:elastic_transport.node_pool:Resurrected node <Urllib3HttpNode(http://es-container-local:9200)>%3E) (force=False) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.001s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 3 times in a row, putting on 4 second timeout WARNING:elastic_transport.transport:Retrying request after non-successful status 503 (attempt 2 of 3) INFO:elastic_transport.transport:POST http://es-container-local:9200/zzp%2B%2Bkb6fda41a1c95f4605ab812263d921740a/_search [status:503 duration:0.001s] WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(http://es-container-local:9200)>%3E) has failed for 4 times in a row, putting on 8 second timeout [2024-05-31 09:36:07 +0800] [861] [ERROR] Exception occurred while handling uri: 'http://localhost:8777/api/local_doc_qa/local_doc_chat' Traceback (most recent call last): File "handle_request", line 132, in handle_request "_asgi_lifespan", File "/usr/local/lib/python3.10/dist-packages/sanic/response/types.py", line 547, in stream await self.streaming_fn(self) File "/workspace/qanything_local/qanything_kernel/qanything_server/handler.py", line 355, in generate_answer for resp, next_history in local_doc_qa.get_knowledge_based_answer( File "/workspace/qanything_local/qanything_kernel/core/local_doc_qa.py", line 225, in get_knowledge_based_answer source_documents = self.get_source_documents(retrieval_queries, milvus_kb) File "/workspace/qanything_local/qanything_kernel/core/local_doc_qa.py", line 133, in get_source_documents batch_result = milvus_kb.search_emb_async(embs=embs, top_k=top_k, queries=queries) File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/milvus_client.py", line 181, in search_emb_async return future.result() File "/usr/lib/python3.10/concurrent/futures/_base.py", line 458, in result return self.__get_result() File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result raise self._exception File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, **self.kwargs) File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/milvus_client.py", line 170, in __search_emb_sync es_records = self.client.search(queries) File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/es_client.py", line 182, in search response = self.client.search( File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/utils.py", line 446, in wrapped return api(*args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/init.py", line 3836, in search return self.perform_request( # type: ignore[return-value] File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/_base.py", line 320, in perform_request raise HTTP_EXCEPTIONS.get(meta.status, ApiError)( elasticsearch.ApiError: ApiError(503, 'search_phase_execution_exception', None) ERROR:sanic.error:Exception occurred while handling uri: 'http://localhost:8777/api/local_doc_qa/local_doc_chat' Traceback (most recent call last): File "handle_request", line 132, in handle_request "_asgi_lifespan", File "/usr/local/lib/python3.10/dist-packages/sanic/response/types.py", line 547, in stream await self.streaming_fn(self) File "/workspace/qanything_local/qanything_kernel/qanything_server/handler.py", line 355, in generate_answer for resp, next_history in local_doc_qa.get_knowledge_based_answer( File "/workspace/qanything_local/qanything_kernel/core/local_doc_qa.py", line 225, in get_knowledge_based_answer source_documents = self.get_source_documents(retrieval_queries, milvus_kb) File "/workspace/qanything_local/qanything_kernel/core/local_doc_qa.py", line 133, in get_source_documents batch_result = milvus_kb.search_emb_async(embs=embs, top_k=top_k, queries=queries) File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/milvus_client.py", line 181, in search_emb_async return future.result() File "/usr/lib/python3.10/concurrent/futures/_base.py", line 458, in result return self.__get_result() File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result raise self._exception File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, **self.kwargs) File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/milvus_client.py", line 170, in __search_emb_sync es_records = self.client.search(queries) File "/workspace/qanything_local/qanything_kernel/connector/database/milvus/es_client.py", line 182, in search response = self.client.search( File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/utils.py", line 446, in wrapped return api(*args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/init.py", line 3836, in search return self.perform_request( # type: ignore[return-value] File "/usr/local/lib/python3.10/dist-packages/elasticsearch/_sync/client/_base.py", line 320, in perform_request raise HTTP_EXCEPTIONS.get(meta.status, ApiError)( elasticsearch.ApiError: ApiError(503, 'search_phase_execution_exception', None)

复现方法 | Steps To Reproduce

按照文档下载源码,bash run.sh -c cloud -i 0 下载镜像,起容器,出现上述问题

备注 | Anything else?

No response

我也遇到了该问题。定位为磁盘容量超过95%时elasticsearch将转为只读模式,清理磁盘至95%以下后该问题解决。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants