Fixed-length streamed upstream response body is dropped with StreamResponseBody

[Abubakar-00]

Summary

When Apinto proxies an OpenAI-compatible chat completion endpoint that returns a normal JSON response with a fixed Content-Length, the client can receive HTTP 200 with an empty body through Apinto even though the upstream returned a valid JSON body.

I reproduced this with APIPark using Apinto gateway v0.22.19 and a KServe/vLLM OpenAI-compatible endpoint.

Reproduction

Upstream KServe endpoint, called directly from the Apinto container:

wget -S -O - \
  --header="Content-Type: application/json" \
  --post-data='{"model":"smollm2","messages":[{"role":"user","content":"Say hello!"}],"stream":false}' \
  http://host.docker.internal:8080/openai/v1/chat/completions

Direct upstream response:

HTTP/1.1 200 OK
server: uvicorn
content-length: 668
content-type: application/json

The body is a valid OpenAI-compatible chat.completion JSON object.

The same model through APIPark/Apinto originally returned:

HTTP/1.1 200 OK
Server: uvicorn
Content-Type: application/json
Content-Length: 0
X-Ai-Provider: kserve-smollm2
X-Ai-Model: smollm2

No JSON body was returned to the client.

Expected Behavior

For stream:false, Apinto should forward the upstream JSON response body:

{
  "id": "...",
  "object": "chat.completion",
  "model": "smollm2",
  "choices": [...]
}

Actual Behavior

Apinto returns status and headers, but the body becomes empty for this fixed-length response path.

Root Cause

Apinto enables fasthttp response streaming:

StreamResponseBody: true

In node/http-context/context.go, Apinto handles unknown-length body streams:

if response.IsBodyStream() && response.Header.ContentLength() < 0 {
    ...
}

But the normal path then calls:

response.CopyTo(ctx.response.Response)

fasthttp Response.CopyTo explicitly does not copy body streams:

// CopyTo copies resp contents to dst except of body stream.
func (resp *Response) CopyTo(dst *Response) { ... }

So for a response that has a fixed Content-Length and is represented internally as a streamed response, Apinto can copy the headers but not the body.

Additional Issues Found While Testing A Local Fix

While building a local patched Apinto gateway, I hit two adjacent issues:

1. ai-convert/message.go can panic for custom/local model names when tiktoken.GetEncoding(...) returns nil and the error is ignored.
2. ai-convert/openai.go sets content-encoding: utf-8. UTF-8 is a character set, not an HTTP content encoding. Later body reads can treat this as an encoding and clear the response body. The safer behavior is to delete/avoid that header and leave charset handling to Content-Type.

Working Local Patch Shape

This local patch made APIPark return the KServe non-stream JSON body successfully while keeping SSE streaming working.

diff --git a/node/http-context/context.go b/node/http-context/context.go
@@
   response.CopyTo(ctx.response.Response)
+  if response.Header.ContentLength() > 0 && len(ctx.response.Response.Body()) == 0 {
+      body := response.Body()
+      if len(body) > 0 {
+          ctx.response.SetBody(body)
+      }
+  }
+  ctx.response.Response.StreamBody = false
+  ctx.response.Response.ImmediateHeaderFlush = false
   agent.responseBody.Write(ctx.response.Response.Body())

diff --git a/ai-convert/message.go b/ai-convert/message.go
@@
   if err != nil {
       tkm, _ = tiktoken.GetEncoding(tiktoken.MODEL_CL100K_BASE)
   }
+  if tkm == nil {
+      return 0
+  }

diff --git a/ai-convert/openai.go b/ai-convert/openai.go
@@
-  httpContext.Response().SetHeader("content-encoding", "utf-8")
+  httpContext.Response().DelHeader("content-encoding")
   httpContext.Response().SetBody(body)

Verification With Patched Gateway

Non-stream request through APIPark/Apinto:

curl -m 60 -i -sS -X POST http://localhost:8099/localmodel/chat/completions \
  -H "Authorization: <consumer-api-key>" \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [{"role": "user", "content": "Say hello in one word."}],
    "max_tokens": 8,
    "temperature": 0.2,
    "stream": false
  }'

Patched response:

HTTP/1.1 200 OK
Server: uvicorn
Content-Type: application/json
Content-Length: 606
X-Ai-Provider: kserve-smollm2
X-Ai-Model: smollm2

Body was returned successfully as OpenAI-compatible JSON.

Streaming still works:

curl -m 30 -N -sS -X POST http://localhost:8099/localmodel/chat/completions \
  -H "Authorization: <consumer-api-key>" \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [{"role": "user", "content": "Say hello in one sentence."}],
    "max_tokens": 12,
    "temperature": 0.2,
    "stream": true
  }'

Response:

data: {"id":"...","object":"chat.completion.chunk",...}

data: [DONE]

Related Issues Checked

I checked existing issues:

• bug: 启用应用后，GET和DELETE请求的body会置为null #228: request body issue for GET/DELETE; not the same response-body bug.
• bug: 网关异常 #219: broader streaming stability issue; related area, but not this fixed-length response path.
• bug: 网关HTTP响应一直处于pending状态 #230: response/logging issue reported as fixed in v0.22.19; not this bug.
• bug: 附带额外参数应用上线后，文件上传请求报错 opied 0 bytes from body stream instead of 7094 bytes #233: request body stream upload issue; related stream handling area, but not this response-body bug.