[Bug]: 500 Internal Server Error when stream is set to false

[RWL-Dittrich]

Steps to Reproduce

1. Do a POST request to the completions endpoint
2. Specify "false" for the stream option and send any message

Expected Behavior

I expect the AI output to appear in a response.

Actual Behavior

The request fails with response status 500 Internal Server Error

Version

6e8dd99

Environment & Context

• I'm using the latest version.
• I have searched existing issues.

[gufengc]

Hi, could you share the whole steps how to reproduce it? How do you setup cluster?

[RWL-Dittrich]

Hi there!

I have two Mac minis (M4 Pro with 64 gigabytes of ram) set up with a thunderbolt bridge between them.

I ran both gpt oss 20b and 120b. Both of them had the same issue.

When I did a manual POST request using Postman I got the internal server error. When I'm at work I'll attach the whole stacktrace.

[RWL-Dittrich]

Here's the stacktrace @gufengc
It seems the AI does fully process the request, but when decoding the response it seems to crash

Nov 07 09:22:46.963 [ERROR   ] request_handler.py:154    Error in _forward_request: Expecting value: line 1 column 1 (char 0)
Traceback (most recent call last):
  File "/Users/niverplast/parallax/src/backend/server/request_handler.py", line 152, in _forward_request
    return JSONResponse(content=json.loads(response))
                                ^^^^^^^^^^^^^^^^^^^^
  File "/Users/niverplast/.pyenv/versions/3.12.12/lib/python3.12/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/niverplast/.pyenv/versions/3.12.12/lib/python3.12/json/decoder.py", line 338, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/niverplast/.pyenv/versions/3.12.12/lib/python3.12/json/decoder.py", line 356, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
INFO:     10.0.131.103:61335 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
Nov 07 09:22:46.965 [INFO    ] request_handler.py:131    Request ID: afee85a5-bf37-4adf-8ea8-f1820a273260 | TPS: 12.72 |  TTFT: 65 ms | Output tokens: 1017 | Input tokens: 2017```

[gufengc]

You can do following tests to detect problem

1. try stream=True
2. call 3000 port on both device (expected: one device work and another fail)
3. try a small model on single device