[Serve][4/5] Add HAProxy ingress request router dispatch path

[eicherseiji]

Why this is needed

PR3 introduces TargetGroup.ingress_request_router_targets. This PR uses it: HAProxy asks a router replica which data-plane replica to use, then pins the request to that replica.

Router contract

HAProxy calls the router with the original request body and expects a JSON replica_id:

POST /internal/route HTTP/1.0
Content-Type: application/json

{ ...original request body... }

{ "replica_id": "<actor name>" }

If the router errors, times out, or returns an unknown replica_id, the request falls through to normal load balancing.

HAProxy routing table

For each router-bearing backend, the frontend gates the Lua action on POST and a matched path prefix:

http-request set-var(txn.ingress_request_router_app) str(<backend>) if is_<backend> !{ var(txn.ingress_request_router_app) -m found }
http-request wait-for-body time 5s if METH_POST has_ingress_request_router_app
http-request lua.route_via_ingress_request_router if METH_POST has_ingress_request_router_app
use_backend <backend>-via-ingress-request-router if is_<backend> { var(txn.via_ingress_request_router) -m found }

The mirror backend pins via use-server and shares health checks with the primary via track:

backend <backend>-via-ingress-request-router
    option redispatch
    use-server <server> if { var(txn.ingress_request_router_target) -m str "<server>" }
    server <server> <host>:<port> track <backend>/<server>

Config shape

ServerConfig(name=..., host=..., port=..., replica_id=...)  # new field
BackendConfig(..., ingress_request_router_servers=[ServerConfig, ...])  # new field

Stack: #62667 -> #62680 -> #62668 -> this -> #62670

Test plan

• Unit coverage in python/ray/serve/tests/test_haproxy_api.py for Lua rendering, frontend ACLs, mirror backend generation, and config-write idempotency
• git diff --check
• Repo commit hooks / linters on commit
• Targeted HAProxy config / integration tests

[gemini-code-assist]

Code Review

This pull request implements an "ingress bypass" optimization for Ray Serve, enabling HAProxy to route data plane traffic directly to replicas via a Lua routing script. This mechanism bypasses the standard Serve request pipeline for specific streaming endpoints to improve throughput. The changes include new state management for router deployments, support for custom ASGI apps within replicas, and updated HAProxy templates to handle custom-routed backends. Review feedback highlights several critical areas for improvement in the HAProxy Lua implementation, including the need for high availability in router selection, reducing internal routing timeouts, avoiding blocking I/O on the request path, and preventing resource naming collisions in multi-tenant environments.

[kouroshHakha]

Nice progression on top of #62668 — the ingress_request_router_targets plumbing from 3/N now has a clean consumer here, and the choice to keep the JSON parsing inside Lua (rather than pulling in lua-cjson as a runtime dep) is the right call given that this script lives on every Serve host. The end-to-end test is the convincing piece: it exercises the wait-for-body → Lua → use-server pin on real HAProxy.

Two things I'd want fixed before merge, both in the Python→Jinja boundary:

1. Lua action can be referenced without being loaded when ingress_request_router_servers exist but _routers_and_targets_by_backend returns an empty result (e.g. all data-plane replicas momentarily have replica_id=None, or backend.servers is empty during scaling). HAProxy rejects the config in that case. This is the Cursor Bugbot finding — it's real. Easy fix: make has_ingress_request_router reflect "the Lua actually got written," not "any backend has router servers."
2. Lua action emission isn't gated on the lua-load-per-thread guard — the global section guards on has_ingress_request_router and ingress_request_router_lua_path, but the frontend section guards only on has_ingress_request_router. Same root cause as (1).

A few smaller things below — fallback observability, buffer sizing for larger prompts, formatting brittleness in the Lua codegen, and a couple of test gaps. None are merge-blockers.

Stack-level note: with the single min((port, host)) router pick + 5s timeout, a single bad router replica will incur up to 5s of latency on every POST until LB takes over (since option redispatch doesn't help here — the call is from Lua, not via a backend). When 5/N lands and LLMRouter is the actual call site, this is the path that determines the upper bound on first-response latency for bypass-mode POSTs under partial failure. Worth tracking even if not addressed in this PR.

Bigger-picture: every replica add/remove/restart currently regenerates this Lua file and triggers a graceful HAProxy reload. The runtime admin socket is already plumbed (_send_socket_command) but used read-only. There's a real follow-up worth scoping where membership data moves to HAProxy maps + runtime-managed servers, so scaling events become socket commands instead of full reloads. Comments on consolidating Lua serialization and on _generate_config_file_internal elaborate.

Note

This review was co-written with AI assistance (Claude Code).