memory usage keeps increasing after upgrading from 3.12 to 3.18

[wliang368-leon]

Hi APISIX team,

I’m investigating a memory usage issue after upgrading Apache APISIX from 3.12 to 3.18.

Our main use case is LLM/AI Gateway, and we are using the ai-proxy-multi plugin.

Environment

• APISIX: 3.18
• Previous version: 3.12
• Deployment: Kubernetes
• Main plugin: ai-proxy-multi
• Traffic: production LLM traffic
• Models/upstream endpoints: essentially unchanged before and after the upgrade

Problem

After upgrading from 3.12 to 3.18, APISIX worker memory usage gradually increases.
We did not observe this behavior with the same workload on APISIX 3.12.

Restarting the APISIX pods temporarily brings memory usage back down, but it gradually increases again.

Questions

1. Are there any known memory issues/regressions related to ai-proxy-multi between 3.12 and 3.18?
2. Did the upstream/connection-pool handling of ai-proxy-multi change between these versions?
3. Can multiple ai-proxy-multi instances/routes targeting the same upstream share the same keepalive connection pool?
4. Is there any known issue where ai-proxy-multi or its upstream connection pools retain Lua objects/connections and cause memory to grow over several days?
5. Are there any recommended metrics or debugging methods to distinguish:
   • Lua/worker memory growth
   • connection pool growth
   • APISIX/OpenResty memory leak
   • Linux page cache caused by log files
6. If anyone has investigated a similar issue in APISIX 3.18, what would be the recommended way to isolate the root cause?

I can provide configuration snippets, memory metrics, worker information, and relevant logs if needed.

Thanks！

[chakravarthyBatna]

I couldn't find a reported leak in ai-proxy-multi between 3.12 and 3.18. But two changes in 3.18 can explain memory that keeps growing slowly after the upgrade.

1. Bigger Prometheus shared dict (#13688). The default size of prometheus-metrics went from 10m/15m to 128m. Linux only commits that memory as the dict fills, so pod memory climbs slowly as new metric series appear. 3.18 also adds new LLM histograms (#13487), which create more series.

Check apisix_shared_dict_free_space_bytes{name="prometheus-metrics"}. If pod memory grows while this value goes down, this is the cause, and it should level off. To limit it, set a smaller size in nginx_config.http.lua_shared_dict (and meta.lua_shared_dict) for prometheus-metrics.

2. New HTTP client for AI requests (#13778). Since 3.18, ai-proxy and ai-proxy-multi send LLM requests through the C module ngx_http_ffi_client instead of lua-resty-http, so upstream connections are handled differently. That also answers your Q2. To test it, switch one pod back to the old client and compare:

plugin_attr:
  ai-proxy:
    http_client: lua-resty-http

Lua heap vs other memory: call the Control API on a pod: curl -X POST http://127.0.0.1:9090/v1/gc (forces a full Lua GC). If memory drops, it was just uncollected Lua garbage. If it doesn't, look at the shared dicts or the new C client.

[wliang368-leon]

Thanks for the suggestion. I performed an A/B test on APISIX 3.18 to compare the two HTTP clients under our real production workload.

Test environment

• APISIX version: 3.18
• Same production environment and configuration
• Same traffic pattern and workload
• Same ai-proxy / ai-proxy-multi usage
• 3 pods using the default C-based HTTP client (ngx_http_ffi_client)
• 3 pods using lua-resty-http
• The only intentional difference was the HTTP client configuration
• The traffic consists entirely of LLM requests
• Approximately 4 million LLM requests per day across the gateway
• The traffic used in this test is real production LLM traffic, not synthetic benchmark traffic

The Lua client was configured as:

plugin_attr:
ai-proxy:
http_client: lua-resty-http

A/B test results

After running the test for approximately 24 hours, the difference was very clear:

• lua-resty-http: memory remained stable at around 1.2 GB
• ngx_http_ffi_client: memory continued to increase and reached around 2.2 GB
• Based on our previous observations, the C client configuration consistently increases memory by approximately 1 GB per day

The same memory growth pattern has been observed repeatedly in our previous tests.

Additional investigation

We also performed a full Lua GC using:

curl -X POST http://127.0.0.1:9090/v1/gc

There was no significant decrease in memory usage after the GC.

We checked the Nginx worker processes as well. The memory was mainly anonymous memory (RssAnon), rather than file-backed memory. For example, the worker processes showed approximately 2 GB RSS each, with RssAnon accounting for almost all of the RSS.

We also checked TCP connections:

• ESTABLISHED: 73
• CLOSE_WAIT: 4

There was no obvious accumulation of CLOSE_WAIT connections.

Conclusion

Based on the A/B test, the memory growth appears to be strongly correlated with the new ngx_http_ffi_client used by APISIX 3.18.

The fact that:

1. Both groups run the same APISIX 3.18 version
2. Both groups handle the same type of LLM traffic
3. Both groups handle approximately the same workload
4. lua-resty-http remains stable
5. ngx_http_ffi_client consistently grows by approximately 1 GB/day
6. The workload is large-scale, with approximately 4 million LLM requests per day
7. The test uses real production traffic rather than synthetic benchmark traffic

makes the HTTP client implementation the primary suspect in our investigation.

Could you please help investigate whether there is a memory leak or memory that is not being released correctly in ngx_http_ffi_client, especially under long-running, high-volume LLM traffic and streaming requests with ai-proxy-multi?

If there are any recommended debugging options, metrics, or tools for investigating the native/C memory used by ngx_http_ffi_client, we would be happy to collect additional information.

Thanks again for your help. We really appreciate any guidance or suggestions on how we can further narrow down the root cause.

[jmrplens]

Yes, it's a leak in ngx_http_ffi_client, and it's per closed upstream connection rather than per request. Every connection the client closes leaves its nginx connection pool behind and, over TLS, its OpenSSL SSL object too. I reproduced it on the official apache/apisix:3.18.0-debian and 3.19.0-debian images. Both ship ngx_http_ffi_client v0.1.3 (APISIX-Runtime 1.3.16 and 1.3.18), which is also the module's latest commit.

Every close path goes through ngx_http_ffi_client_keepalive_close_connection() (src/ngx_http_ffi_client_keepalive.c#L536-L548). That function deletes the timers and calls ngx_close_connection(c), which frees neither c->pool (created for each connection at request.c#L1161) nor c->ssl (from ngx_ssl_create_connection() at request.c#L1875). Neither file calls ngx_ssl_shutdown(), SSL_free() or ngx_destroy_pool(). nginx's own upstream keepalive module and the cosocket close in lua-nginx-module call ngx_ssl_shutdown() and ngx_destroy_pool(c->pool) before ngx_close_connection().

Measured with one worker, ai-proxy-multi and a mock OpenAI-compatible upstream:

Upstream connections	ngx_http_ffi_client	lua-resty-http
TLS, the upstream closes every connection	+38.8 KB per connection, streaming or not	flat
TLS, keep-alive, 64 concurrent streams (above the default keepalive_pool of 30)	+26.6 KB per connection that doesn't fit in the pool	flat
plain HTTP, the upstream closes every connection	+150 B per connection	flat
TLS, keep-alive, 16 concurrent, every connection reused	flat	flat

/v1/gc freed nothing in any run, and the Lua heap stayed around 10 MB, as in your test. With gdb on the worker, 300 TLS requests produced 300 SSL_new() calls and no SSL_free() with the ffi client; with lua-resty-http they were 300 and 300. Patching the running worker to shut TLS down and destroy the pool before closing kept RssAnon flat over 12,000 such connections.

So streaming isn't the cause by itself. Long streams push concurrency above the keepalive pool, and every connection over it gets closed. At 26 to 39 KB each, 1 GB a day per pod is 26,000 to 40,000 closed TLS connections a day. To check it on your pods, compare RssAnon growth with the change in ActiveOpens in /proc/net/snmp inside the pod. In the lab, the growth per new outbound connection came to 38.7 KB with the ffi client and about 0 with lua-resty-http.

http_client: lua-resty-http is the right workaround until the module is fixed. Raising keepalive_pool reduces the churn, but idle timeouts and connections the upstream closes still leak. The fix belongs in api7/ngx_http_ffi_client, which ships inside APISIX-Runtime. I couldn't find an issue or PR there or in APISIX that covers it.

[jmrplens]

Follow-up: the fix is up as api7/ngx_http_ffi_client#48, for api7/ngx_http_ffi_client#47. It shuts TLS down and destroys the connection pool before closing, and adds a test that fails on v0.1.3. Until a release of APISIX-Runtime includes it, http_client: lua-resty-http remains the workaround.