AI Proxy - issue when transforming the response body for analytics #14380
Description
Kong version ($ kong version)
3.9
Current Behavior
When I want to call my API
`
2025/03/25 10:41:01 [warn] 1415#0: *133798 [kong] parse-json-response.lua:34 [ai-proxy] failed to decode response body for usage introspection: Expected value but found invalid token at character 1 while sending to client, client: 10.89.4.1, server: kong, request: "POST /mistral-chat HTTP/1.1", host: "localhost:8000", request_id: "ac48673a28745f175b70754ea6a69f59"
�2025/03/25 10:41:01 [error] 1415#0: *133798 [kong] normalize-json-response.lua:31 [ai-proxy] issue when transforming the response body for analytics: transformation failed from type mistral://llm/v1/chat: failed to decode llm/v1/chat response while sending to client, client: 10.89.4.1, server: kong, request: "POST /mistral-chat HTTP/1.1", host: "localhost:8000", request_id: "ac48673a28745f175b70754ea6a69f59"
�10.89.4.1 - - [25/Mar/2025:10:41:01 +0000] "POST /mistral-chat HTTP/1.1" 500 119 "-" "bruno-runtime/1.39.1" kong_request_id: "ac48673a28745f175b70754ea6a69f59"
`
Steps To Reproduce
1- Setup AI proxy like this
{ "name": "ai-proxy", "instance_name": "ai-proxy", "config": { "route_type": "llm/v1/chat", "max_request_body_size": 8192, "logging": { "log_statistics": true, "log_payloads": true }, "model": { "provider": "mistral", "name": "mistral-large-latest", "options": { "upstream_url": "https://api.mistral.ai/v1/chat/completions", "max_tokens": 256, "mistral_format": "openai" } }, "auth": { "param_name": null, "param_value": null, "header_value": "Bearer *****************", "allow_override": false, "gcp_service_account_json": null, "aws_access_key_id": null, "header_name": "Authorization", "aws_secret_access_key": null, "gcp_use_service_account": false, "param_location": null }, "model_name_header": true, "response_streaming": "allow" }, "enabled": true }
`> POST http://localhost:8000/mistral-chat
content-type: application/json
accept: application/json
data
{
"model": "mistral-large-latest",
"messages": [
{
"role": "user",
"content": "***************"
}
],
"response_format": {
"type": "json_object"
}
}
< 500 - Internal Server Error
< content-type: application/json
< transfer-encoding: chunked
< connection: keep-alive
< x-ratelimitbysize-limit-month: 1000000000
< date: Tue, 25 Mar 2025 11:27:07 GMT
< access-control-allow-origin: *
< x-kong-proxy-latency: 5
< cf-cache-status: DYNAMIC
< x-kong-upstream-latency: 3333
< x-kong-request-id: 361c536eff75187f1fb0a034a26b2c1c
< cf-ray: 925e0e953d8d02ad-CDG
< alt-svc: h3=":443"; ma=86400
< x-ratelimitbysize-remaining-month: 999855585
< ratelimitbysize-remaining: 499723
< ratelimitbysize-limit: 500000
< server: cloudflare
< ratelimitbysize-reset: 56
< ratelimitbysize-query-cost: 277
< x-envoy-upstream-service-time: 3243
< x-ratelimitbysize-remaining-minute: 499723
< x-ratelimitbysize-limit-minute: 500000
< x-kong-llm-model: mistral/mistral-large-latest
< via: 1.1 kong/3.9.0`