使用VLLM加速之后就这样了，知识库查询有结果，但是模型并没有返回结果 #3511

wuliaodeashuai · 2024-03-25T07:57:58Z

INFO: 127.0.0.1:46996 - "POST /chat/knowledge_base_chat HTTP/1.1" 200 OK
/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/langchain_core/_api/deprecation.py:117: LangChainDeprecationWarning: The class langchain_community.chat_models.openai.ChatOpenAI was deprecated in langchain-community 0.0.10 and will be removed in 0.2.0. An updated version of the class exists in the langchain-openai package and should be used instead. To use it run pip install -U langchain-openai and import as from langchain_openai import ChatOpenAI.
warn_deprecated(
2024-03-25 15:53:19,881 - SentenceTransformer.py[line:66] - INFO: Load pretrained SentenceTransformer: /www/NLP/langchain_RAG/datast/model/m3e-base
2024-03-25 15:53:22,503 - faiss_cache.py[line:94] - INFO: loading vector store in 'test/vector_store/m3e-base' from disk.
2024-03-25 15:53:22,508 - loader.py[line:54] - INFO: Loading faiss with AVX2 support.
2024-03-25 15:53:22,541 - loader.py[line:56] - INFO: Successfully loaded faiss with AVX2 support.
/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/langchain_community/vectorstores/faiss.py:121: UserWarning: Normalizing L2 is not applicable for metric type: METRIC_INNER_PRODUCT
warnings.warn(
ERROR: Exception in ASGI application
Traceback (most recent call last):
File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/fastchat/serve/base_model_worker.py", line 156, in count_token
input_ids = self.tokenizer(prompt).input_ids
TypeError: 'Encoding' object is not callable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 412, in run_asgi
result = await app( # type: ignore[func-returns-value]
File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 69, in call
return await self.app(scope, receive, send)
File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/fastapi/applications.py", line 1054, in call
await super().call(scope, receive, send)
File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/starlette/applications.py", line 122, in call
await self.middleware_stack(scope, receive, send)
File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/starlette/middleware/errors.py", line 184, in call
raise exc
File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/starlette/middleware/errors.py", line 162, in call
await self.app(scope, receive, _send)
File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 79, in call
raise exc
File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 68, in call
await self.app(scope, receive, sender)
File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/starlette/routing.py", line 718, in call
await route.handle(scope, receive, send)
File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/starlette/routing.py", line 276, in handle
await self.app(scope, receive, send)
File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/starlette/routing.py", line 66, in app
response = await func(request)
File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/fastapi/routing.py", line 299, in app
raise e
File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/fastapi/routing.py", line 294, in app
raw_response = await run_endpoint_function(
File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
return await dependant.call(**values)
File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/fastchat/serve/vllm_worker.py", line 231, in api_count_token
return worker.count_token(params)
File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/fastchat/serve/base_model_worker.py", line 159, in count_token
input_echo_len = self.tokenizer.num_tokens(prompt)
AttributeError: 'Encoding' object has no attribute 'num_tokens'
2024-03-25 15:53:22 | INFO | stdout | INFO: 127.0.0.1:41004 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
2024-03-25 15:53:22 | ERROR | stderr | ERROR: Exception in ASGI application
2024-03-25 15:53:22 | ERROR | stderr | Traceback (most recent call last):
2024-03-25 15:53:22 | ERROR | stderr | File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 412, in run_asgi
2024-03-25 15:53:22 | ERROR | stderr | result = await app( # type: ignore[func-returns-value]
2024-03-25 15:53:22 | ERROR | stderr | File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 69, in call
2024-03-25 15:53:22 | ERROR | stderr | return await self.app(scope, receive, send)
2024-03-25 15:53:22 | ERROR | stderr | File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/fastapi/applications.py", line 1054, in call
2024-03-25 15:53:22 | ERROR | stderr | await super().call(scope, receive, send)
2024-03-25 15:53:22 | ERROR | stderr | File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/starlette/applications.py", line 122, in call
2024-03-25 15:53:22 | ERROR | stderr | await self.middleware_stack(scope, receive, send)
2024-03-25 15:53:22 | ERROR | stderr | File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/starlette/middleware/errors.py", line 184, in call
2024-03-25 15:53:22 | ERROR | stderr | raise exc
2024-03-25 15:53:22 | ERROR | stderr | File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/starlette/middleware/errors.py", line 162, in call
2024-03-25 15:53:22 | ERROR | stderr | await self.app(scope, receive, _send)
2024-03-25 15:53:22 | ERROR | stderr | File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/starlette/middleware/cors.py", line 83, in call
2024-03-25 15:53:22 | ERROR | stderr | await self.app(scope, receive, send)
2024-03-25 15:53:22 | ERROR | stderr | File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 79, in call
2024-03-25 15:53:22 | ERROR | stderr | raise exc
2024-03-25 15:53:22 | ERROR | stderr | File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 68, in call
2024-03-25 15:53:22 | ERROR | stderr | await self.app(scope, receive, sender)
2024-03-25 15:53:22 | ERROR | stderr | File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/starlette/routing.py", line 718, in call
2024-03-25 15:53:22 | ERROR | stderr | await route.handle(scope, receive, send)
2024-03-25 15:53:22 | ERROR | stderr | File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/starlette/routing.py", line 276, in handle
2024-03-25 15:53:22 | ERROR | stderr | await self.app(scope, receive, send)
2024-03-25 15:53:22 | ERROR | stderr | File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/starlette/routing.py", line 66, in app
2024-03-25 15:53:22 | ERROR | stderr | response = await func(request)
2024-03-25 15:53:22 | ERROR | stderr | File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/fastapi/routing.py", line 299, in app
2024-03-25 15:53:22 | ERROR | stderr | raise e
2024-03-25 15:53:22 | ERROR | stderr | File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/fastapi/routing.py", line 294, in app
2024-03-25 15:53:22 | ERROR | stderr | raw_response = await run_endpoint_function(
2024-03-25 15:53:22 | ERROR | stderr | File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
2024-03-25 15:53:22 | ERROR | stderr | return await dependant.call(**values)
2024-03-25 15:53:22 | ERROR | stderr | File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/fastchat/serve/openai_api_server.py", line 438, in create_chat_completion
2024-03-25 15:53:22 | ERROR | stderr | max_new_tokens, error_check_ret = await check_length(
2024-03-25 15:53:22 | ERROR | stderr | File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/fastchat/serve/openai_api_server.py", line 172, in check_length
2024-03-25 15:53:22 | ERROR | stderr | length = min(max_tokens, context_len - token_num)
2024-03-25 15:53:22 | ERROR | stderr | TypeError: unsupported operand type(s) for -: 'int' and 'str'
2024-03-25 15:53:22,651 - _client.py[line:1729] - INFO: HTTP Request: POST http://127.0.0.1:20000/v1/chat/completions "HTTP/1.1 500 Internal Server Error"
2024-03-25 15:53:22,651 - _base_client.py[line:1414] - INFO: Retrying request to /chat/completions in 0.967108 seconds
ERROR: Exception in ASGI application
Traceback (most recent call last):
File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/fastchat/serve/base_model_worker.py", line 156, in count_token
input_ids = self.tokenizer(prompt).input_ids
TypeError: 'Encoding' object is not callable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 412, in run_asgi
result = await app( # type: ignore[func-returns-value]
File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 69, in call
return await self.app(scope, receive, send)
File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/fastapi/applications.py", line 1054, in call
await super().call(scope, receive, send)
File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/starlette/applications.py", line 122, in call
await self.middleware_stack(scope, receive, send)
File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/starlette/middleware/errors.py", line 184, in call
raise exc
File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/starlette/middleware/errors.py", line 162, in call
await self.app(scope, receive, _send)
File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 79, in call
raise exc
File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 68, in call
await self.app(scope, receive, sender)
File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/starlette/routing.py", line 718, in call
await route.handle(scope, receive, send)
File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/starlette/routing.py", line 276, in handle
await self.app(scope, receive, send)
File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/starlette/routing.py", line 66, in app
response = await func(request)
File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/fastapi/routing.py", line 299, in app
raise e
File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/fastapi/routing.py", line 294, in app
raw_response = await run_endpoint_function(
File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
return await dependant.call(**values)
File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/fastchat/serve/vllm_worker.py", line 231, in api_count_token
return worker.count_token(params)
File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/fastchat/serve/base_model_worker.py", line 159, in count_token
input_echo_len = self.tokenizer.num_tokens(prompt)
AttributeError: 'Encoding' object has no attribute 'num_tokens'
2024-03-25 15:53:23 | INFO | stdout | INFO: 127.0.0.1:41010 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
2024-03-25 15:53:23 | ERROR | stderr | ERROR: Exception in ASGI application
2024-03-25 15:53:23 | ERROR | stderr | Traceback (most recent call last):
2024-03-25 15:53:23 | ERROR | stderr | File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 412, in run_asgi
2024-03-25 15:53:23 | ERROR | stderr | result = await app( # type: ignore[func-returns-value]
2024-03-25 15:53:23 | ERROR | stderr | File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 69, in call
2024-03-25 15:53:23 | ERROR | stderr | return await self.app(scope, receive, send)
2024-03-25 15:53:23 | ERROR | stderr | File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/fastapi/applications.py", line 1054, in call
2024-03-25 15:53:23 | ERROR | stderr | await super().call(scope, receive, send)
2024-03-25 15:53:23 | ERROR | stderr | File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/starlette/applications.py", line 122, in call
2024-03-25 15:53:23 | ERROR | stderr | await self.middleware_stack(scope, receive, send)
2024-03-25 15:53:23 | ERROR | stderr | File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/starlette/middleware/errors.py", line 184, in call
2024-03-25 15:53:23 | ERROR | stderr | raise exc
2024-03-25 15:53:23 | ERROR | stderr | File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/starlette/middleware/errors.py", line 162, in call
2024-03-25 15:53:23 | ERROR | stderr | await self.app(scope, receive, _send)
2024-03-25 15:53:23 | ERROR | stderr | File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/starlette/middleware/cors.py", line 83, in call
2024-03-25 15:53:23 | ERROR | stderr | await self.app(scope, receive, send)
2024-03-25 15:53:23 | ERROR | stderr | File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 79, in call
2024-03-25 15:53:23 | ERROR | stderr | raise exc
2024-03-25 15:53:23 | ERROR | stderr | File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 68, in call
2024-03-25 15:53:23 | ERROR | stderr | await self.app(scope, receive, sender)
2024-03-25 15:53:23 | ERROR | stderr | File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/starlette/routing.py", line 718, in call
2024-03-25 15:53:23 | ERROR | stderr | await route.handle(scope, receive, send)
2024-03-25 15:53:23 | ERROR | stderr | File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/starlette/routing.py", line 276, in handle
2024-03-25 15:53:23 | ERROR | stderr | await self.app(scope, receive, send)
2024-03-25 15:53:23 | ERROR | stderr | File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/starlette/routing.py", line 66, in app
2024-03-25 15:53:23 | ERROR | stderr | response = await func(request)
2024-03-25 15:53:23 | ERROR | stderr | File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/fastapi/routing.py", line 299, in app
2024-03-25 15:53:23 | ERROR | stderr | raise e
2024-03-25 15:53:23 | ERROR | stderr | File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/fastapi/routing.py", line 294, in app
2024-03-25 15:53:23 | ERROR | stderr | raw_response = await run_endpoint_function(
2024-03-25 15:53:23 | ERROR | stderr | File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
2024-03-25 15:53:23 | ERROR | stderr | return await dependant.call(**values)
2024-03-25 15:53:23 | ERROR | stderr | File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/fastchat/serve/openai_api_server.py", line 438, in create_chat_completion
2024-03-25 15:53:23 | ERROR | stderr | max_new_tokens, error_check_ret = await check_length(
2024-03-25 15:53:23 | ERROR | stderr | File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/fastchat/serve/openai_api_server.py", line 172, in check_length
2024-03-25 15:53:23 | ERROR | stderr | length = min(max_tokens, context_len - token_num)
2024-03-25 15:53:23 | ERROR | stderr | TypeError: unsupported operand type(s) for -: 'int' and 'str'
2024-03-25 15:53:23,634 - _client.py[line:1729] - INFO: HTTP Request: POST http://127.0.0.1:20000/v1/chat/completions "HTTP/1.1 500 Internal Server Error"
2024-03-25 15:53:23,635 - _base_client.py[line:1414] - INFO: Retrying request to /chat/completions in 1.631195 seconds
ERROR: Exception in ASGI application
Traceback (most recent call last):
File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/fastchat/serve/base_model_worker.py", line 156, in count_token
input_ids = self.tokenizer(prompt).input_ids
TypeError: 'Encoding' object is not callable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 412, in run_asgi
result = await app( # type: ignore[func-returns-value]
File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 69, in call
return await self.app(scope, receive, send)
File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/fastapi/applications.py", line 1054, in call
await super().call(scope, receive, send)
File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/starlette/applications.py", line 122, in call
await self.middleware_stack(scope, receive, send)
File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/starlette/middleware/errors.py", line 184, in call
raise exc
File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/starlette/middleware/errors.py", line 162, in call
await self.app(scope, receive, _send)
File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 79, in call
raise exc
File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 68, in call
await self.app(scope, receive, sender)
File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/starlette/routing.py", line 718, in call
await route.handle(scope, receive, send)
File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/starlette/routing.py", line 276, in handle
await self.app(scope, receive, send)
File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/starlette/routing.py", line 66, in app
response = await func(request)
File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/fastapi/routing.py", line 299, in app
raise e
File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/fastapi/routing.py", line 294, in app
raw_response = await run_endpoint_function(
File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
return await dependant.call(**values)
File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/fastchat/serve/vllm_worker.py", line 231, in api_count_token
return worker.count_token(params)
File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/fastchat/serve/base_model_worker.py", line 159, in count_token
input_echo_len = self.tokenizer.num_tokens(prompt)
AttributeError: 'Encoding' object has no attribute 'num_tokens'
2024-03-25 15:53:25 | INFO | stdout | INFO: 127.0.0.1:41018 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
2024-03-25 15:53:25 | ERROR | stderr | ERROR: Exception in ASGI application
2024-03-25 15:53:25 | ERROR | stderr | Traceback (most recent call last):
2024-03-25 15:53:25 | ERROR | stderr | File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 412, in run_asgi
2024-03-25 15:53:25 | ERROR | stderr | result = await app( # type: ignore[func-returns-value]
2024-03-25 15:53:25 | ERROR | stderr | File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 69, in call
2024-03-25 15:53:25 | ERROR | stderr | return await self.app(scope, receive, send)
2024-03-25 15:53:25 | ERROR | stderr | File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/fastapi/applications.py", line 1054, in call
2024-03-25 15:53:25 | ERROR | stderr | await super().call(scope, receive, send)
2024-03-25 15:53:25 | ERROR | stderr | File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/starlette/applications.py", line 122, in call
2024-03-25 15:53:25 | ERROR | stderr | await self.middleware_stack(scope, receive, send)
2024-03-25 15:53:25 | ERROR | stderr | File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/starlette/middleware/errors.py", line 184, in call
2024-03-25 15:53:25 | ERROR | stderr | raise exc
2024-03-25 15:53:25 | ERROR | stderr | File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/starlette/middleware/errors.py", line 162, in call
2024-03-25 15:53:25 | ERROR | stderr | await self.app(scope, receive, _send)
2024-03-25 15:53:25 | ERROR | stderr | File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/starlette/middleware/cors.py", line 83, in call
2024-03-25 15:53:25 | ERROR | stderr | await self.app(scope, receive, send)
2024-03-25 15:53:25 | ERROR | stderr | File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 79, in call
2024-03-25 15:53:25 | ERROR | stderr | raise exc
2024-03-25 15:53:25 | ERROR | stderr | File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 68, in call
2024-03-25 15:53:25 | ERROR | stderr | await self.app(scope, receive, sender)
2024-03-25 15:53:25 | ERROR | stderr | File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/starlette/routing.py", line 718, in call
2024-03-25 15:53:25 | ERROR | stderr | await route.handle(scope, receive, send)
2024-03-25 15:53:25 | ERROR | stderr | File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/starlette/routing.py", line 276, in handle
2024-03-25 15:53:25 | ERROR | stderr | await self.app(scope, receive, send)
2024-03-25 15:53:25 | ERROR | stderr | File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/starlette/routing.py", line 66, in app
2024-03-25 15:53:25 | ERROR | stderr | response = await func(request)
2024-03-25 15:53:25 | ERROR | stderr | File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/fastapi/routing.py", line 299, in app
2024-03-25 15:53:25 | ERROR | stderr | raise e
2024-03-25 15:53:25 | ERROR | stderr | File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/fastapi/routing.py", line 294, in app
2024-03-25 15:53:25 | ERROR | stderr | raw_response = await run_endpoint_function(
2024-03-25 15:53:25 | ERROR | stderr | File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
2024-03-25 15:53:25 | ERROR | stderr | return await dependant.call(**values)
2024-03-25 15:53:25 | ERROR | stderr | File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/fastchat/serve/openai_api_server.py", line 438, in create_chat_completion
2024-03-25 15:53:25 | ERROR | stderr | max_new_tokens, error_check_ret = await check_length(
2024-03-25 15:53:25 | ERROR | stderr | File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/fastchat/serve/openai_api_server.py", line 172, in check_length
2024-03-25 15:53:25 | ERROR | stderr | length = min(max_tokens, context_len - token_num)
2024-03-25 15:53:25 | ERROR | stderr | TypeError: unsupported operand type(s) for -: 'int' and 'str'
2024-03-25 15:53:25,283 - _client.py[line:1729] - INFO: HTTP Request: POST http://127.0.0.1:20000/v1/chat/completions "HTTP/1.1 500 Internal Server Error"
2024-03-25 15:53:25,285 - utils.py[line:24] - ERROR: Internal Server Error
Traceback (most recent call last):
File "/www/NLP/langchain_RAG/server/utils.py", line 22, in wrap_done
await fn
File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/langchain/chains/base.py", line 381, in acall
raise e
File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/langchain/chains/base.py", line 375, in acall
await self._acall(inputs, run_manager=run_manager)
File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/langchain/chains/llm.py", line 275, in _acall
response = await self.agenerate([inputs], run_manager=run_manager)
File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/langchain/chains/llm.py", line 142, in agenerate
return await self.llm.agenerate_prompt(
File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/langchain_core/language_models/chat_models.py", line 554, in agenerate_prompt
return await self.agenerate(
File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/langchain_core/language_models/chat_models.py", line 514, in agenerate
raise exceptions[0]
File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/langchain_core/language_models/chat_models.py", line 617, in _agenerate_with_cache
return await self._agenerate(
File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/langchain_community/chat_models/openai.py", line 522, in _agenerate
return await agenerate_from_stream(stream_iter)
File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/langchain_core/language_models/chat_models.py", line 87, in agenerate_from_stream
async for chunk in stream:
File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/langchain_community/chat_models/openai.py", line 488, in _astream
async for chunk in await acompletion_with_retry(
File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/langchain_community/chat_models/openai.py", line 105, in acompletion_with_retry
return await llm.async_client.create(**kwargs)
File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/openai/resources/chat/completions.py", line 1295, in create
return await self._post(
File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/openai/_base_client.py", line 1536, in post
return await self.request(cast_to, opts, stream=stream, stream_cls=stream_cls)
File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/openai/_base_client.py", line 1315, in request
return await self._request(
File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/openai/_base_client.py", line 1378, in _request
return await self._retry_request(
File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/openai/_base_client.py", line 1418, in _retry_request
return await self._request(
File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/openai/_base_client.py", line 1378, in _request
return await self._retry_request(
File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/openai/_base_client.py", line 1418, in _retry_request
return await self._request(
File "/www/anaconda3/envs/langchain_rag/lib/python3.10/site-packages/openai/_base_client.py", line 1392, in _request
raise self._make_status_error_from_response(err.response) from None
openai.InternalServerError: Internal Server Error
2024-03-25 15:53:25,288 - utils.py[line:27] - ERROR: InternalServerError: Caught exception: Internal Server Error

The text was updated successfully, but these errors were encountered:

dosubot · 2024-03-25T08:03:59Z

嘿，@wuliaodeashuai，又见面了！遇到新问题了吗？让我们看看怎么解决吧。

您的Python应用程序中的内部服务器错误，特别是在"chat_models.py"和"base_model_worker.py"文件中的TypeError和AttributeError，可能是由几个问题引起的。以下是可能的原因和解决方案：

TypeError：当一个操作或函数被应用到一个不适当类型的对象时，就会发生这个错误。没有具体的错误消息，很难确定确切的原因。然而，常见的来源包括：
- 错误的数据类型被传递给函数或方法。例如，ApiModelWorker中的count_token方法期望params包含一个"prompt"键。如果params不是一个字典，或者不包含"prompt"键作为一个字符串，它可能会引发一个TypeError。
- 确保所有期望某些类型或结构的数据的方法都按预期接收它们。如果需要，使用类型提示和运行时检查来提前捕获这些问题。
AttributeError：这个错误表明试图访问一个对象没有的属性。您的代码中可能的原因包括：
- 访问未初始化或拼写错误的属性。例如，如果在设置self.conv之前引用它，或者如果拼写错误，它会引发一个AttributeError。
- BaseModelWorker类，ApiModelWorker继承自它，可能没有定义一些ApiModelWorker假定存在的属性或方法。确保所有继承的属性或方法在父类中正确定义。
- 使用动态属性，如在load_config中使用setattr设置的属性，如果属性名拼写错误或后续引用不正确，可能会导致AttributeError。

解决方案：

对于TypeError：
- 在使用它们之前验证所有输入的类型。Python的类型提示可以帮助静态分析工具，但对于动态语言，需要运行时检查（例如，isinstance）。
- 确保传递的数据结构（例如，各种方法中的params）符合预期的格式。
对于AttributeError：
- 确保在使用之前正确初始化所有属性。这包括检查父类中的属性定义。
- 在访问动态设置的属性之前使用hasattr()，或确保在访问之前总是设置它们。
- 检查属性名中的拼写错误。

没有具体的错误消息或错误发生的行，这些都是一般的指导原则。要解决这些问题，查看错误消息提供的堆栈跟踪，以确定导致错误的确切行，并相应地应用上述解决方案。

另外，关于base_model_worker.py文件中的count_token方法，其目的是计算并返回提供给它的prompt参数的字符数（长度）。它返回一个字典，包含prompt中的字符数和一个error_code设置为0，表示在操作过程中没有发生错误。

这是相关的代码片段：

def count_token(self, params):
    prompt = params["prompt"]
    return {"count": len(str(prompt)), "error_code": 0}

这个方法接收一个带有键"prompt"的字典params，将这个键关联的值转换为字符串（如果它还不是字符串），使用len()计算其长度，并在键"count"下在一个字典中返回这个长度。该方法还返回"error_code": 0，表示成功执行，没有错误。

希望这些信息能帮助您解决问题。如果还有其他问题，欢迎继续提问。

参考资料：

server/model_workers/base.py

Sources

server/model_workers/base.py

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

github-actions · 2024-04-24T21:30:30Z

这个问题已经被标记为 stale ，因为它已经超过 30 天没有任何活动。

wuliaodeashuai added the bug Something isn't working label Mar 25, 2024

github-actions bot added the stale label Apr 24, 2024

dosubot bot mentioned this issue Apr 26, 2024

[BUG] 多并发调用，偶尔出现因为embedding没加载而调用失败 #3899

Closed

zRzRzRzRzRzRzR self-assigned this May 5, 2024

zRzRzRzRzRzRzR closed this as completed May 5, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

使用VLLM加速之后就这样了，知识库查询有结果，但是模型并没有返回结果 #3511

使用VLLM加速之后就这样了，知识库查询有结果，但是模型并没有返回结果 #3511

wuliaodeashuai commented Mar 25, 2024

dosubot bot commented Mar 25, 2024

About Dosu

github-actions bot commented Apr 24, 2024

使用VLLM加速之后就这样了，知识库查询有结果，但是模型并没有返回结果 #3511

使用VLLM加速之后就这样了，知识库查询有结果，但是模型并没有返回结果 #3511

Comments

wuliaodeashuai commented Mar 25, 2024

dosubot bot commented Mar 25, 2024

Sources

About Dosu

github-actions bot commented Apr 24, 2024