fix(ai-proxy): yield to scheduler in streaming SSE loop to avoid worker CPU starvation

[nic-6443]

What

Add an explicit ngx.sleep(0) at the end of each iteration of the streaming SSE loop in apisix/plugins/ai-providers/base.lua::parse_streaming_response. This guarantees the coroutine yields to the nginx scheduler at least once per upstream chunk.

Why

In production we observed worker processes pinned at 100% CPU during AI proxy traffic. Root cause: when an upstream LLM emits SSE chunks in a tight burst (e.g. a model hallucinating and producing tokens at 100+ per second, or upstreams that batch multiple SSE events into a single TCP segment), the streaming loop runs for an extended period without yielding.

Specifically:

• body_reader() (cosocket socket:receive()) only yields when the recv buffer is empty. If the kernel has already buffered several chunks, successive calls return immediately without yielding.
• ngx.flush(true) (used downstream) only yields when the send buffer is full. A fast downstream client drains immediately, so flush returns without yielding.

Neither end of the loop guarantees a yield. The result: the SSE coroutine monopolizes the worker — starving health checks, concurrent requests on the same worker, and timer callbacks. Even modest traffic can saturate a single core because Lua coroutines on the same OpenResty worker share one OS thread.

ngx.sleep(0) is the canonical OpenResty primitive for this — it queues a 0-second timer and yields the current coroutine, letting the scheduler pick up any other ready coroutines, then resumes.

Cost

• Normal traffic: chunks already arrive with inter-chunk gaps, so body_reader() already yields naturally between chunks. The extra ngx.sleep(0) is invisible.
• Burst traffic: caps per-coroutine runtime to one chunk's worth of work between yields. The yield itself is microseconds.

Test plan

This is a concurrency / scheduling fix where deterministic reproduction in test-nginx is difficult — burst behavior depends on TCP buffering between the mock upstream and the proxy, both of which run in the same nginx instance during tests, so timing rarely matches the real-world scenario. Existing streaming correctness tests (t/plugin/ai-proxy*.t, t/plugin/ai-proxy-client-disconnect.t) cover that the loop still produces correct output and that the new yield doesn't break the disconnect-detection or limit-enforcement paths.

Per the project's testing exception for "concurrency issues that are hard to simulate", I'm relying on existing tests for correctness regression coverage.

[Copilot]

Pull request overview

This PR mitigates OpenResty worker CPU starvation during bursty AI SSE streaming by forcing a cooperative yield in the AI provider streaming loop, ensuring other coroutines (health checks, concurrent requests, timers) can run even when both upstream reads and downstream flushes return immediately.

Changes:

• Add an explicit ngx.sleep(0) yield at the end of each iteration of the SSE streaming loop.
• Document in-code why the yield is necessary under bursty upstream + fast downstream conditions.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.