Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: ValueError: Out of range float values are not JSON compliant when requesting logprobs from awq model #328

Closed
sgsdxzy opened this issue Mar 15, 2024 · 1 comment 路 Fixed by #329
Closed

[Bug]: ValueError: Out of range float values are not JSON compliant when requesting logprobs from awq model #328

sgsdxzy opened this issue Mar 15, 2024 · 1 comment 路 Fixed by #329
Labels
bug Something isn't working

Comments

@sgsdxzy
Copy link
Collaborator

sgsdxzy commented Mar 15, 2024

Your current environment

wget https://raw.githubusercontent.com/PygmalionAI/aphrodite-engine/env.py
400: Invalid request
It's a fresh conda environment for py311+torch220+aphrodite 0.5.0

馃悰 Describe the bug

launch aphrodite with
python -m aphrodite.endpoints.openai.api_server --host 127.0.0.1 --port 5000 --dtype float16 --max-log-len 0 --block-size 16 -tp 4 --gpu-memory-utilization 1.0 --model Qwen/Qwen1.5-72B-Chat-AWQ -q awq --max-model-len 14496 --enforce-eager --kv-cache-dtype auto --served-model-name Qwen1.5-72B-Chat-AWQ

Test script:

curl http://localhost:5000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer sk" \
-d '{
  "model": "Qwen1.5-72B-Chat-AWQ",
  "messages": [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Who won the world series in 2020?"},
    {"role": "assistant", "content": "The Los Angeles Dodgers won the World Series in 2020."},
    {"role": "user", "content": "Where was it played?"}
  ],
  "stream": false,
  "logprobs": true,
  "top_logprobs": 10
}'

Result (aphrodite backtrace):

  ......
  Expected `int` but got `str` - serialized value may not be as expected
  Expected `int` but got `str` - serialized value may not be as expected
  return self.__pydantic_serializer__.to_python(
INFO:     127.0.0.1:47572 - "POST /v1/chat/completions HTTP/1.1" 500
ERROR:    Exception in ASGI application
ERROR:      + Exception Group Traceback (most recent call last):
ERROR:      |   File "/home/sgsdxzy/miniforge3/envs/aphrodite/lib/python3.11/site-packages/starlette/_utils.py", line 87, in collapse_excgroups
ERROR:      |     yield
ERROR:      |   File "/home/sgsdxzy/miniforge3/envs/aphrodite/lib/python3.11/site-packages/starlette/middleware/base.py", line 190, in __call__                              ERROR:      |     async with anyio.create_task_group() as task_group:
ERROR:      |   File "/home/sgsdxzy/miniforge3/envs/aphrodite/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 678, in __aexit__
ERROR:      |     raise BaseExceptionGroup(
ERROR:      | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
ERROR:      +-+---------------- 1 ----------------
ERROR:        | Traceback (most recent call last):
ERROR:        |   File "/home/sgsdxzy/miniforge3/envs/aphrodite/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 408, in run_asgi
ERROR:        |     result = await app(  # type: ignore[func-returns-value]
ERROR:        |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR:        |   File "/home/sgsdxzy/miniforge3/envs/aphrodite/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 69, in __call__
ERROR:        |     return await self.app(scope, receive, send)
ERROR:        |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR:        |   File "/home/sgsdxzy/miniforge3/envs/aphrodite/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
ERROR:        |     await super().__call__(scope, receive, send)
ERROR:        |   File "/home/sgsdxzy/miniforge3/envs/aphrodite/lib/python3.11/site-packages/starlette/applications.py", line 123, in __call__
ERROR:        |     await self.middleware_stack(scope, receive, send)                                                                                                        ERROR:        |   File "/home/sgsdxzy/miniforge3/envs/aphrodite/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__                          ERROR:        |     raise exc
ERROR:        |   File "/home/sgsdxzy/miniforge3/envs/aphrodite/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
ERROR:        |     await self.app(scope, receive, _send)
ERROR:        |   File "/home/sgsdxzy/miniforge3/envs/aphrodite/lib/python3.11/site-packages/starlette/middleware/base.py", line 189, in __call__
ERROR:        |     with collapse_excgroups():
ERROR:        |   File "/home/sgsdxzy/miniforge3/envs/aphrodite/lib/python3.11/contextlib.py", line 158, in __exit__
ERROR:        |     self.gen.throw(typ, value, traceback)
ERROR:        |   File "/home/sgsdxzy/miniforge3/envs/aphrodite/lib/python3.11/site-packages/starlette/_utils.py", line 93, in collapse_excgroups
ERROR:        |     raise exc
ERROR:        |   File "/home/sgsdxzy/miniforge3/envs/aphrodite/lib/python3.11/site-packages/starlette/middleware/base.py", line 191, in __call__
ERROR:        |     response = await self.dispatch_func(request, call_next)
ERROR:        |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR:        |   File "/home/sgsdxzy/Programs/aphrodite-engine/aphrodite/endpoints/openai/api_server.py", line 539, in authentication
ERROR:        |     return await call_next(request)
ERROR:        |            ^^^^^^^^^^^^^^^^^^^^^^^^
ERROR:        |   File "/home/sgsdxzy/miniforge3/envs/aphrodite/lib/python3.11/site-packages/starlette/middleware/base.py", line 165, in call_next
ERROR:        |     raise app_exc
ERROR:        |   File "/home/sgsdxzy/miniforge3/envs/aphrodite/lib/python3.11/site-packages/starlette/middleware/base.py", line 151, in coro
ERROR:        |     await self.app(scope, receive_or_disconnect, send_no_error)
ERROR:        |   File "/home/sgsdxzy/miniforge3/envs/aphrodite/lib/python3.11/site-packages/starlette/middleware/cors.py", line 83, in __call__
ERROR:        |     await self.app(scope, receive, send)
ERROR:        |   File "/home/sgsdxzy/miniforge3/envs/aphrodite/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
ERROR:        |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
ERROR:        |   File "/home/sgsdxzy/miniforge3/envs/aphrodite/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
ERROR:        |     raise exc
ERROR:        |   File "/home/sgsdxzy/miniforge3/envs/aphrodite/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
ERROR:        |     await app(scope, receive, sender)
ERROR:        |   File "/home/sgsdxzy/miniforge3/envs/aphrodite/lib/python3.11/site-packages/starlette/routing.py", line 758, in __call__
ERROR:        |     await self.middleware_stack(scope, receive, send)
ERROR:        |   File "/home/sgsdxzy/miniforge3/envs/aphrodite/lib/python3.11/site-packages/starlette/routing.py", line 778, in app
ERROR:        |     await route.handle(scope, receive, send)
ERROR:        |   File "/home/sgsdxzy/miniforge3/envs/aphrodite/lib/python3.11/site-packages/starlette/routing.py", line 299, in handle
ERROR:        |     await self.app(scope, receive, send)
ERROR:        |   File "/home/sgsdxzy/miniforge3/envs/aphrodite/lib/python3.11/site-packages/starlette/routing.py", line 79, in app
ERROR:        |     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
ERROR:        |   File "/home/sgsdxzy/miniforge3/envs/aphrodite/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
ERROR:        |     raise exc
ERROR:        |   File "/home/sgsdxzy/miniforge3/envs/aphrodite/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
ERROR:        |     await app(scope, receive, sender)
ERROR:        |   File "/home/sgsdxzy/miniforge3/envs/aphrodite/lib/python3.11/site-packages/starlette/routing.py", line 74, in app
ERROR:        |     response = await func(request)
ERROR:        |                ^^^^^^^^^^^^^^^^^^^
ERROR:        |   File "/home/sgsdxzy/miniforge3/envs/aphrodite/lib/python3.11/site-packages/fastapi/routing.py", line 278, in app
ERROR:        |     raw_response = await run_endpoint_function(
ERROR:        |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR:        |   File "/home/sgsdxzy/miniforge3/envs/aphrodite/lib/python3.11/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
ERROR:        |     return await dependant.call(**values)
ERROR:        |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR:        |   File "/home/sgsdxzy/Programs/aphrodite-engine/aphrodite/endpoints/openai/api_server.py", line 246, in create_chat_completion
ERROR:        |     return JSONResponse(content=generator.model_dump())
ERROR:        |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR:        |   File "/home/sgsdxzy/miniforge3/envs/aphrodite/lib/python3.11/site-packages/starlette/responses.py", line 183, in __init__
ERROR:        |     super().__init__(content, status_code, headers, media_type, background)
ERROR:        |   File "/home/sgsdxzy/miniforge3/envs/aphrodite/lib/python3.11/site-packages/starlette/responses.py", line 41, in __init__
ERROR:        |     self.body = self.render(content)
ERROR:        |                 ^^^^^^^^^^^^^^^^^^^^
ERROR:        |   File "/home/sgsdxzy/miniforge3/envs/aphrodite/lib/python3.11/site-packages/starlette/responses.py", line 186, in render
ERROR:        |     return json.dumps(
ERROR:        |            ^^^^^^^^^^^
ERROR:        |   File "/home/sgsdxzy/miniforge3/envs/aphrodite/lib/python3.11/json/__init__.py", line 238, in dumps
ERROR:        |     **kw).encode(obj)
ERROR:        |           ^^^^^^^^^^^
ERROR:        |   File "/home/sgsdxzy/miniforge3/envs/aphrodite/lib/python3.11/json/encoder.py", line 200, in encode
ERROR:        |     chunks = self.iterencode(o, _one_shot=True)
ERROR:        |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR:        |   File "/home/sgsdxzy/miniforge3/envs/aphrodite/lib/python3.11/json/encoder.py", line 258, in iterencode
ERROR:        |     return _iterencode(o, 0)
ERROR:        |            ^^^^^^^^^^^^^^^^^
ERROR:        | ValueError: Out of range float values are not JSON compliant
ERROR:        +------------------------------------
ERROR:
ERROR:    During handling of the above exception, another exception occurred:
ERROR:                                                                                                                                                    15:24:20 [359/1856]
ERROR:    Traceback (most recent call last):
ERROR:      File "/home/sgsdxzy/miniforge3/envs/aphrodite/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 408, in run_asgi
ERROR:        result = await app(  # type: ignore[func-returns-value]
ERROR:                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR:      File "/home/sgsdxzy/miniforge3/envs/aphrodite/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 69, in __call__
ERROR:        return await self.app(scope, receive, send)
ERROR:               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR:      File "/home/sgsdxzy/miniforge3/envs/aphrodite/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
ERROR:        await super().__call__(scope, receive, send)
ERROR:      File "/home/sgsdxzy/miniforge3/envs/aphrodite/lib/python3.11/site-packages/starlette/applications.py", line 123, in __call__
ERROR:        await self.middleware_stack(scope, receive, send)
ERROR:      File "/home/sgsdxzy/miniforge3/envs/aphrodite/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__
ERROR:        raise exc
ERROR:      File "/home/sgsdxzy/miniforge3/envs/aphrodite/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
ERROR:        await self.app(scope, receive, _send)
ERROR:      File "/home/sgsdxzy/miniforge3/envs/aphrodite/lib/python3.11/site-packages/starlette/middleware/base.py", line 189, in __call__
ERROR:        with collapse_excgroups():
ERROR:      File "/home/sgsdxzy/miniforge3/envs/aphrodite/lib/python3.11/contextlib.py", line 158, in __exit__
ERROR:        self.gen.throw(typ, value, traceback)
ERROR:      File "/home/sgsdxzy/miniforge3/envs/aphrodite/lib/python3.11/site-packages/starlette/_utils.py", line 93, in collapse_excgroups
ERROR:        raise exc
ERROR:      File "/home/sgsdxzy/miniforge3/envs/aphrodite/lib/python3.11/site-packages/starlette/middleware/base.py", line 191, in __call__
ERROR:        response = await self.dispatch_func(request, call_next)
ERROR:                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR:      File "/home/sgsdxzy/Programs/aphrodite-engine/aphrodite/endpoints/openai/api_server.py", line 539, in authentication
ERROR:        return await call_next(request)
ERROR:               ^^^^^^^^^^^^^^^^^^^^^^^^
ERROR:      File "/home/sgsdxzy/miniforge3/envs/aphrodite/lib/python3.11/site-packages/starlette/middleware/base.py", line 165, in call_next
ERROR:        raise app_exc
ERROR:      File "/home/sgsdxzy/miniforge3/envs/aphrodite/lib/python3.11/site-packages/starlette/middleware/base.py", line 151, in coro
ERROR:        await self.app(scope, receive_or_disconnect, send_no_error)
ERROR:      File "/home/sgsdxzy/miniforge3/envs/aphrodite/lib/python3.11/site-packages/starlette/middleware/cors.py", line 83, in __call__
ERROR:        await self.app(scope, receive, send)
ERROR:      File "/home/sgsdxzy/miniforge3/envs/aphrodite/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
ERROR:        await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
ERROR:      File "/home/sgsdxzy/miniforge3/envs/aphrodite/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
ERROR:        raise exc
ERROR:      File "/home/sgsdxzy/miniforge3/envs/aphrodite/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
ERROR:        await app(scope, receive, sender)
ERROR:      File "/home/sgsdxzy/miniforge3/envs/aphrodite/lib/python3.11/site-packages/starlette/routing.py", line 758, in __call__
ERROR:        await self.middleware_stack(scope, receive, send)
ERROR:      File "/home/sgsdxzy/miniforge3/envs/aphrodite/lib/python3.11/site-packages/starlette/routing.py", line 778, in app
ERROR:        await route.handle(scope, receive, send)
ERROR:      File "/home/sgsdxzy/miniforge3/envs/aphrodite/lib/python3.11/site-packages/starlette/routing.py", line 299, in handle
ERROR:        await self.app(scope, receive, send)
ERROR:      File "/home/sgsdxzy/miniforge3/envs/aphrodite/lib/python3.11/site-packages/starlette/routing.py", line 79, in app
ERROR:        await wrap_app_handling_exceptions(app, request)(scope, receive, send)
ERROR:      File "/home/sgsdxzy/miniforge3/envs/aphrodite/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
ERROR:        raise exc
ERROR:      File "/home/sgsdxzy/miniforge3/envs/aphrodite/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
ERROR:        await app(scope, receive, sender)
ERROR:      File "/home/sgsdxzy/miniforge3/envs/aphrodite/lib/python3.11/site-packages/starlette/routing.py", line 74, in app
ERROR:        response = await func(request)
ERROR:                   ^^^^^^^^^^^^^^^^^^^
ERROR:      File "/home/sgsdxzy/miniforge3/envs/aphrodite/lib/python3.11/site-packages/fastapi/routing.py", line 278, in app
ERROR:        raw_response = await run_endpoint_function(
ERROR:                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR:      File "/home/sgsdxzy/miniforge3/envs/aphrodite/lib/python3.11/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
ERROR:        return await dependant.call(**values)
ERROR:               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR:      File "/home/sgsdxzy/Programs/aphrodite-engine/aphrodite/endpoints/openai/api_server.py", line 246, in create_chat_completion
ERROR:        return JSONResponse(content=generator.model_dump())
ERROR:               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR:      File "/home/sgsdxzy/miniforge3/envs/aphrodite/lib/python3.11/site-packages/starlette/responses.py", line 183, in __init__
ERROR:        super().__init__(content, status_code, headers, media_type, background)
ERROR:      File "/home/sgsdxzy/miniforge3/envs/aphrodite/lib/python3.11/site-packages/starlette/responses.py", line 41, in __init__
ERROR:        self.body = self.render(content)
ERROR:                    ^^^^^^^^^^^^^^^^^^^^
ERROR:      File "/home/sgsdxzy/miniforge3/envs/aphrodite/lib/python3.11/site-packages/starlette/responses.py", line 186, in render
ERROR:        return json.dumps(
ERROR:               ^^^^^^^^^^^
ERROR:      File "/home/sgsdxzy/miniforge3/envs/aphrodite/lib/python3.11/json/__init__.py", line 238, in dumps
ERROR:        **kw).encode(obj)
ERROR:              ^^^^^^^^^^^
ERROR:      File "/home/sgsdxzy/miniforge3/envs/aphrodite/lib/python3.11/json/encoder.py", line 200, in encode
ERROR:        chunks = self.iterencode(o, _one_shot=True)
ERROR:                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR:      File "/home/sgsdxzy/miniforge3/envs/aphrodite/lib/python3.11/json/encoder.py", line 258, in iterencode
ERROR:        return _iterencode(o, 0)
ERROR:               ^^^^^^^^^^^^^^^^^
ERROR:    ValueError: Out of range float values are not JSON compliant

Chat completion works fine if not requesting logprobs. The problem doesn't occur for v1/completions, only for v1/chat/completions. And the gptq Q4 of the same model doesn't have this problem either.
@sgsdxzy sgsdxzy added the bug Something isn't working label Mar 15, 2024
@AlpinDale AlpinDale mentioned this issue Mar 15, 2024
Merged
@AlpinDale
Copy link
Member

Can you test #329? Should fix this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: logprobs when inf or nan PygmalionAI/aphrodite-engine
2 participants
@sgsdxzy @AlpinDale