TabbyAPI endpoint "The response is incomplete", even though it is

[ghost]

What happened?

Updated this morning to latest containers, and my TabbyAPI completions endpoints did not well with this new version. The last time I updated before this morning was 1 week ago.

Steps to Reproduce

1. Run TabbyAPI

    - name: 'Local'
      apiKey: '${LOCAL_API_KEY}'
      baseURL: 'https://domain.com/v1'
      models:
        default: [ 'local-default' ]
        fetch: false
      titleConvo: true
      titleModel: 'local-default'
      summarize: false
      modelDisplayLabel: 'Local'
      plugins: true
      assistants: true

3. Try one chat completion
4. Something went wrong. Here's the specific error message we encountered: The response is incomplete; it's either still processing, was cancelled, or censored. Refresh or try a different prompt.

What browsers are you seeing the problem on?

Firefox

Relevant log output

TabbyAPI shows no error:

INFO:     Metrics (ID: 36b540f10df34ab8a69060c0989db31c): 39 tokens generated in 2.88
seconds (Queue: 0.0 s, Process: 0 cached tokens and 10 new tokens at 38.03 T/s,
Generate: 14.89 T/s, Context: 10 tokens)
INFO:     127.0.0.1:42688 - "POST /v1/chat/completions HTTP/1.1" 200
INFO:     Received chat completion streaming request c194c945e90c4747b92718d9d3caef89
INFO:     Prompt (ID: c194c945e90c4747b92718d9d3caef89):
INFO:     <s>[INST] Hola[/INST]
INFO:     Response (ID: c194c945e90c4747b92718d9d3caef89):
INFO:      Hola, ¿en qué puedo ayudarte hoy?
INFO:     Finished chat completion streaming request c194c945e90c4747b92718d9d3caef89
INFO:     Generation options: {'request_id': 'c194c945e90c4747b92718d9d3caef89',
'max_tokens': 1000, 'min_tokens': 0, 'stream': True, 'token_repetition_penalty': 1.0,
'token_repetition_range': -1, 'token_repetition_decay': 0, 'token_frequency_penalty':
0.0, 'token_presence_penalty': 0.0, 'temperature': 1.0, 'smoothing_factor': 0.0,
'min_temp': 1.0, 'max_temp': 1.0, 'temp_exponent': 1.0, 'top_k': 0, 'top_p': 0.9,
'top_a': 0.0, 'min_p': 0.0, 'tfs': 1.0, 'typical': 1.0, 'skew': 0.0,
'temperature_last': False, 'mirostat': False, 'mirostat_tau': 1.5, 'mirostat_eta':
0.3, 'mirostat_mu': None, 'token_bias': None, 'cfg_scale': None,
'post_sampling_hooks': [], 'dry_allowed_length': 2, 'dry_base': 1.75,
'dry_multiplier': 0.0, 'dry_sequence_breakers': None, 'dry_range': 0,
'dry_max_ngram': 20, 'ngram_trie': None, 'ngram_index': 0, 'ngram_history':
deque([]), 'token_healing': False, 'auto_scale_penalty_range': False,
'generate_window': 960, 'bos_token_id': 1, 'eos_token_id': [2], 'add_bos_token':
True, 'ban_eos_token': False, 'skip_special_tokens': True, 'speculative_ngram':
False, 'logprobs': 0, 'stop_conditions': [2], 'banned_tokens': [], 'allowed_tokens':
[], 'banned_strings': [], 'logit_bias': None, 'filters': []}

INFO:     Metrics (ID: c194c945e90c4747b92718d9d3caef89): 15 tokens generated in 1.03
seconds (Queue: 0.0 s, Process: 9 cached tokens and 1 new tokens at 43.53 T/s,
Generate: 14.87 T/s, Context: 10 tokens)

Docker logs

2024-09-13 12:17:12 error: [handleAbortError] AI response error; aborting request: Cannot read properties of null (reading 'completion_tokens_details')
2024-09-13 12:17:12 error: [handleAbortError] error while trying to abort message Cannot set headers after they are sent to the client

Seems unrelated, when I get the error multiple time, these errors do not repeat in the logs

Screenshots

Code of Conduct

• I agree to follow this project's Code of Conduct

[danny-avila]

There isn't enough logs here to see what's going on. Check your ./logs directory at project root, in particular the most recent file(s) with "error" in the name after this happens.

[ghost]

I'll try it thanks!

Here are the verbose logs, if that's useful:

{
  "code": "ERR_HTTP_HEADERS_SENT",
  "level": "error",
  "message": "[handleAbortError] error while trying to abort message Cannot set headers after they are sent to the client",
  "stack": "Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the client\n    at ServerResponse.setHeader (node:_http_outgoing:659:11)\n    at ServerResponse.header (/app/node_modules/express/lib/response.js:795:10)\n    at ServerResponse.send (/app/node_modules/express/lib/response.js:175:12)\n    at ServerResponse.json (/app/node_modules/express/lib/response.js:279:15)\n    at ServerResponse.send (/app/node_modules/express/lib/response.js:163:21)\n    at abortMessage (/app/api/server/middleware/abortMiddleware.js:30:28)\n    at handleAbortError (/app/api/server/middleware/abortMiddleware.js:213:20)\n    at AskController (/app/api/server/controllers/AskController.js:170:5)\n    at async /app/api/server/routes/ask/custom.js:24:5"
}
{
  "code": "ERR_HTTP_HEADERS_SENT",
  "level": "error",
  "message": "[handleAbortError] error while trying to abort message Cannot set headers after they are sent to the client",
  "stack": "Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the client\n    at ServerResponse.setHeader (node:_http_outgoing:659:11)\n    at ServerResponse.header (/app/node_modules/express/lib/response.js:795:10)\n    at ServerResponse.send (/app/node_modules/express/lib/response.js:175:12)\n    at ServerResponse.json (/app/node_modules/express/lib/response.js:279:15)\n    at ServerResponse.send (/app/node_modules/express/lib/response.js:163:21)\n    at abortMessage (/app/api/server/middleware/abortMiddleware.js:30:28)\n    at handleAbortError (/app/api/server/middleware/abortMiddleware.js:213:20)\n    at AskController (/app/api/server/controllers/AskController.js:170:5)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async /app/api/server/routes/ask/custom.js:24:5"
}
{
  "level": "error",
  "message": "[handleAbortError] AI response error; aborting request: Cannot read properties of null (reading 'completion_tokens_details')",
  "stack": "TypeError: Cannot read properties of null (reading 'completion_tokens_details')\n    at OpenAIClient.getStreamUsage (/app/api/app/clients/OpenAIClient.js:918:25)\n    at OpenAIClient.sendMessage (/app/api/app/clients/BaseClient.js:611:56)\n    at async AskController (/app/api/server/controllers/AskController.js:125:20)\n    at async /app/api/server/routes/ask/custom.js:24:5"
}
{
  "level": "error",
  "message": "[handleAbortError] AI response error; aborting request: Cannot read properties of null (reading 'completion_tokens_details')",
  "stack": "TypeError: Cannot read properties of null (reading 'completion_tokens_details')\n    at OpenAIClient.getStreamUsage (/app/api/app/clients/OpenAIClient.js:918:25)\n    at OpenAIClient.sendMessage (/app/api/app/clients/BaseClient.js:611:56)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async AskController (/app/api/server/controllers/AskController.js:125:20)\n    at async /app/api/server/routes/ask/custom.js:24:5"
}

[ghost]

Hi, I still have the same issues after updating :/

{
  "level": "error",
  "message": "[handleAbortError] AI response error; aborting request: Cannot read properties of null (reading 'reasoning_tokens')",
  "stack": "TypeError: Cannot read properties of null (reading 'reasoning_tokens')\n    at OpenAIClient.recordTokenUsage (/app/api/app/clients/OpenAIClient.js:1099:51)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async OpenAIClient.sendMessage (/app/api/app/clients/BaseClient.js:622:7)\n    at async AskController (/app/api/server/controllers/AskController.js:125:20)\n    at async /app/api/server/routes/ask/custom.js:24:5"
}

{
  "code": "ERR_HTTP_HEADERS_SENT",
  "level": "error",
  "message": "[handleAbortError] error while trying to abort message Cannot set headers after they are sent to the client",
  "stack": "Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the client\n    at ServerResponse.setHeader (node:_http_outgoing:659:11)\n    at ServerResponse.header (/app/node_modules/express/lib/response.js:795:10)\n    at ServerResponse.send (/app/node_modules/express/lib/response.js:175:12)\n    at ServerResponse.json (/app/node_modules/express/lib/response.js:279:15)\n    at ServerResponse.send (/app/node_modules/express/lib/response.js:163:21)\n    at abortMessage (/app/api/server/middleware/abortMiddleware.js:30:28)\n    at handleAbortError (/app/api/server/middleware/abortMiddleware.js:213:20)\n    at AskController (/app/api/server/controllers/AskController.js:170:5)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async /app/api/server/routes/ask/custom.js:24:5"
}

[danny-avila]

Thank you for your logs, as I can fully address it now

[danny-avila]

Fixed: f1ae267

If you rely on prebuilt docker images, they are still being built at the time of writing

[ghost]

You're incredible