bug: Executing v1/chat/completions with stream set to false results in an error response after 2 minutes

[wvteijlingen]

Describe the bug
Sending a request to v1/chat/completions endpoint results in a long loading time of about 2 minutes, after which an error 500 response is returned. This only occurs when when stream is set to false. Possibly related to #1852?

Steps to reproduce
Steps to reproduce the behavior:

1. Start the API server
2. Execute the curl request below
3. Notice that nothing happens for about two minutes. The request is logged as incoming in the Jan console.
4. After two minutes, a 500 error is returned from the API.

Expected behavior
An OK response is returned in reasonable time

Environment details

• Operating System: Operating System: macOS 14.4.1
• Jan Version: 0.4.12
• Processor: Apple M1 Pro
• RAM: 32 GB

curl command and output

curl -X 'POST' \
  'http://localhost:1337/v1/chat/completions' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "messages": [
    {
      "content": "You are a helpful assistant.",
      "role": "system"
    },
    {
      "content": "Hello!",
      "role": "user"
    }
  ],
  "model": "llama3-8b-instruct",
  "stream": false,
  "max_tokens": 2048,
  "stop": [
    "hello"
  ],
  "frequency_penalty": 0,
  "presence_penalty": 0,
  "temperature": 0.7,
  "top_p": 0.95
}'

2 minutes later...

{"statusCode":500,"error":"Internal Server Error","message":"Converting circular structure to JSON\n    --> starting at object with constructor 'Socket'\n    |     property 'parser' -> object with constructor 'HTTPParser'\n    --- property 'socket' closes the circle"}%

[Van-QA]

hi @wvteijlingen, thank you, the issue is reproducible, but changing the stop token can help:
"stop": ["<|end_of_text|>","<|eot_id|>"],
Sample fo reference that is working on my side:

Sorry for the inconvenience, we will correct the sample cURL of our API reference soon. 🙏

• Our API reference doc ❌
  

• Our Swagger ❌