feat(ai-proxy): send LLM requests through ngx_http_ffi_client - #13778
feat(ai-proxy): send LLM requests through ngx_http_ffi_client#13778shreemaan-abhishek wants to merge 6 commits into
Conversation
ai-proxy, ai-proxy-multi and ai-request-rewrite share ai-transport/http.lua for every outbound LLM request. It now prefers ngx_http_ffi_client, a C client whose object API matches lua-resty-http and which costs about a third of the outbound CPU time. The module only exists when the APISIX runtime was built with it, so the transport resolves the client once per worker and keeps lua-resty-http as the fallback. plugin_attr.ai-proxy.http_client pins the choice: auto (default), ffi, or lua-resty-http. Connection and Transfer-Encoding are now dropped from the forwarded headers. They describe the downstream connection, and passing them on desyncs the upstream one: the client frames the request body with Content-Length itself, and a forwarded "Connection: close" would also keep the connection out of the keepalive pool.
dosubot
Bot
added
size:L
Dropping Connection and Transfer-Encoding from the headers forwarded to the LLM upstream broke an existing ai-proxy-multi retry case: after a stream that dies before its first byte, the retry stopped finding an instance to re-pick and the request returned 502. Forwarding a downstream Connection header to a third-party upstream is still wrong, and the streaming read-error path still leaks the upstream connection without closing it, but neither belongs in a change about which HTTP client the transport builds.
…p_ffi_client plugin_attr.ai-proxy.http_client took auto, ffi or lua-resty-http, and auto made the choice implicit in what the runtime happened to carry. It now takes one of the two client names, ngx_http_ffi_client or lua-resty-http, and defaults to the first. A runtime built without the module still falls back rather than failing the request, and says so at warn level. Error level would put a line in the log for every AI request on such a runtime. t/plugin/ai-transport-http.t TEST 11 drives the lua-resty-http path against a real upstream with nothing stubbed, so the path that config selects is covered end to end rather than only through a stub.
…ing back Resolve the client from plugin_attr.ai-proxy.http_client through a name to module map, validate the name against an enum schema, and cache it only once it has loaded. A missing or unusable client is now an error the request carries, never a silent switch to the other one. Bump APISIX_RUNTIME to 1.3.12, the first runtime built with ngx_http_ffi_client.
Cosockets get their hostnames resolved by apisix/patch.lua, which routes them through core.resolver and so honours dns_resolver, /etc/hosts and the search domains. ngx_http_ffi_client dials from C and never touches a cosocket, so it saw only nginx's `resolver` and failed on every name that layer cannot answer. The transport now resolves the name the same way before handing the address to the C client, keeping the original for the Host header and the SNI. Pin the body-encoding and error-mapping cases to lua-resty-http: they stub resty.http, so under the C-client default they were driving the real client at a dead port instead of the stub.
dosubot
Bot
added
size:XL
| -- core.resolver, which honours dns_resolver, /etc/hosts and the search | ||
| -- domains. The C client dials on its own and only sees nginx's `resolver`, | ||
| -- so the name is resolved here and kept for the Host header and the SNI. | ||
| local function resolve_upstream_host(params) |
There was a problem hiding this comment.
ngx_http_ffi_client dials from C, so it never touches the cosocket that patch.lua wraps — it only sees nginx's resolver, which does not read /etc/hosts or our dns_resolver config. That is why every localhost upstream 500'd. Resolving here puts it back on the same path as every other socket in the gateway, and we keep the name for Host and SNI.
membphis
left a comment
membphis
left a comment
There was a problem hiding this comment.
[P1] Please address these blockers before merge:
-
The current t/plugin/[a-k]*.t CI shard is failing on this head, and it contains the changed t/plugin/ai-transport-http.t coverage. Please identify the cause and make the relevant checks pass on this exact head.
-
Please make the ngx_http_ffi_client revision bundled in apisix-runtime 1.3.12 traceable, and add APISIX-level coverage that runs the real C client through buffered responses, SSE streaming, TLS/DNS, and keepalive. The new tests mostly stub the module, so they do not prove that the new default client includes and preserves the required framing, connection-lifecycle, and error-semantics fixes.
-
The linux_apisix_current_luarocks_in_customed_nginx job is also failing. Because custom runtimes may not include the C module, please verify the supported compatibility path on this head: either include the module or explicitly configure and test lua-resty-http as the fallback for that build.
2 participants
Description
ai-proxy,ai-proxy-multiandai-request-rewriteall send their outboundLLM requests through
apisix/plugins/ai-transport/http.lua. This moves thattransport to
ngx_http_ffi_client, an HTTP client implemented as an nginx Cmodule.
ngx_http_ffi_clientexposes the same object API aslua-resty-http(
new,set_timeout,connect,request,res.body_reader,res:read_body,set_keepalive,close) and does the HTTP/1.1 framing in C,which costs roughly a third of the outbound CPU time on this path. Measurements:
https://github.com/api7/ngx_http_ffi_client/blob/63771541c5a229da8a840ab6008ba3771a22fb71/benchmark/results-full.md
plugin_attr.ai-proxy.http_clientnames the client:The name is validated against an enum schema and mapped to a module, the module
is loaded on the first request, and the result is cached only once a client has
actually loaded. A client that is missing or cannot be created fails the request
and the error names the cause; the transport never substitutes the other client,
so what runs is always what the config asked for.
APISIX_RUNTIMEmoves to 1.3.12, the first runtime built with the module. Ahand-built runtime without it has to set
http_client: lua-resty-http, whichruns exactly the code that ran before.
Which issue(s) this PR fixes:
N/A
Checklist
Test plan
t/plugin/ai-transport-http.tgains six cases covering client selection: the Cclient is the default, a runtime whose C module is not built in fails the
request instead of switching clients, a runtime where the module does not load
at all fails the same way, an unrecognised
http_clientvalue fails schemavalidation,
plugin_attrselectslua-resty-httpover an available C client,and the
lua-resty-httppath is driven against a real upstream with the Cclient stubbed to a sentinel so a regression in the selection cannot pass
silently.
Beyond the plugin tests, the client itself was exercised against a real build:
an OpenResty carrying the module, driven through the exact call sequence this
transport uses, covering a buffered request, an SSE stream read through
body_reader,set_keepalive,close, and the error stringshandle_errormaps to 502/504.
The published
apisix-runtime/1.3.12packages were checked directly: thenginxbinary carries the module's symbols and the package installslualib/resty/ngx_http_ffi_client.lua.