[serve] HAProxy stability fixes: stderr file redirect, broadcast coalescing, redispatch

[harshit-anyscale]

Summary

Three targeted HAProxy fixes addressing the highest-impact failure modes observed under production load. Each is independently motivated.

1. Redirect HAProxy stderr to a file (avoids the pipe-buffer deadlock)

The problem. The proxy actor spawns HAProxy via asyncio.create_subprocess_exec(..., stderr=PIPE). That creates an OS pipe with a fixed 64 KB kernel buffer:

HAProxy ──[write(2, ...)]──► [64 KB kernel buffer] ──► proc.stderr (read by Python)

HAProxy's config uses log local0 debug plus option httplog — high-severity logging on every HTTP request and every health-check transition. In environments where syslog is unreachable (containers without /dev/log, no listener on 127.0.0.1:syslog_port), HAProxy falls back to writing those log lines to stderr. At sustained load, this is thousands of lines per second.

If the proxy actor never reads from proc.stderr, the 64 KB buffer fills in seconds. Once full, HAProxy's write(2) on stderr blocks — including writes from threads serving the admin socket. From the outside, runtime-API commands appear to hang, healthz probes time out, and the actor cascades into reload/restart loops. This was the root cause behind the "60-second admin-socket deadlock" we kept hitting under load.

The fix. Instead of piping stderr back to Python, redirect HAProxy's stderr fd directly to a file at spawn time:

with open(tmp_path, "ab", buffering=0) as stderr_file:
    proc = await asyncio.create_subprocess_exec(..., stderr=stderr_file)
os.replace(tmp_path, f"{socket_path}.stderr.{proc.pid}.log")

The kernel uses dup2() at fork to wire the child's fd 2 to the file. No pipe is created, so the 64 KB buffer that previously deadlocked admin-socket threads doesn't exist — write(2) to a regular file doesn't have the "consumer hasn't read me, block now" semantic that pipes do. The rename after spawn is safe because Linux file descriptors bind to inodes, not paths, so the child keeps writing to the same inode under the new pid-suffixed name.

Once the pipe was eliminated, the admin-socket-deadlock cascades disappeared.

2. Coalesce controller broadcasts into a single reload

Under autoscaling churn the Serve controller fires target_groups and fallback_targets broadcasts independently, often only tens of ms apart. Without coalescing, each broadcast triggers its own config regeneration and graceful reload via -sf.

HAProxyManager now marks state dirty on each broadcast and arms a single sleeping coalesce task; updates arriving during the sleep window are absorbed into the same pending apply. Window defaults to 100 ms via RAY_SERVE_HAPROXY_BROADCAST_COALESCE_S; set to 0 to disable. If an apply fails, the task re-arms and retries on the next tick (up to 3 consecutive failures, then waits for a new broadcast — avoids busy-spinning on a persistent error).

3. option redispatch + retry-on conn-failure empty-response in defaults

Under churn, the first connect attempt to a slot can hit a replica that is briefly down (just-removed, mid-restart, scaling-up slot not yet accepting). Today HAProxy returns the resulting 502/503 directly to the client even though a peer slot is up and ready. The two directives — added to the defaults block so every backend inherits them — make the retry transparent:

• option redispatch sends the retry to a different slot, not the same one.
• retry-on conn-failure empty-response restricts retries to cases that cannot have leaked partial bytes to the client: TCP connect failure (no bytes sent) and empty response (slot died before sending a body). Streaming responses with a partial body are untouched.

option redispatch was previously set only on the *-via-ingress-request-router backend; the primary routing path had no retry semantics, which is why churn-class 5xx surfaced to clients.

Out of scope

The bigger PR (#63159) includes additional architectural changes (server-template slot pools, runtime-API server updates, plus other refinements). This PR is deliberately the minimal high-impact subset for an easier review and lower-risk merge.

Other knobs explored during the investigation (option abortonclose removal, timeout server / timeout connect defaults, hard-stop-after bump) are intentionally not part of this PR — those are better addressed client-side for the load-test workload rather than as new HAProxy defaults.

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request introduces several stability improvements for HAProxy within Ray Serve. Key changes include increasing the hard stop timeout to 1800 seconds and establishing default server and connection timeouts to prevent undefined behavior during reloads. Additionally, the option abortonclose setting was removed from the configuration. A critical fix involves the implementation of a background task to drain HAProxy's stderr into a per-PID log file, which prevents potential deadlocks caused by the OS pipe buffer filling up. Feedback on these changes suggests increasing the logging severity from DEBUG to WARNING if the stderr drainer fails, ensuring that any failure in this mechanism is visible since it could lead back to the deadlock issue.

[harshit-anyscale]

three test runs after these changes:

1. https://console.anyscale-staging.com/cld_kvedZWag2qA8i5BjxUevf5i7/prj_g7p6lsu6r8g7garwbxifppyz23/jobs/prodjob_mcti9khv32ds77r4ihnfdur6r8?job-tab=overview&job-logs-section-tabs=application_logs

2. https://console.anyscale-staging.com/cld_kvedZWag2qA8i5BjxUevf5i7/prj_g7p6lsu6r8g7garwbxifppyz23/jobs/prodjob_vqcrdh1tl8xgddbtdrn8pedtcn?job-tab=overview&job-logs-section-tabs=application_logs

3. https://console.anyscale-staging.com/cld_kvedZWag2qA8i5BjxUevf5i7/prj_g7p6lsu6r8g7garwbxifppyz23/jobs/prodjob_tag4m8zw5v19n7iq2t4ptt2i6f?job-tab=overview&job-logs-section-tabs=application_logs

all have the failure rate < 0.01%.

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{Reviewed by Cursor Bugbot for commit 247b07c. Configure here.}

[cursor]

Retry exhaustion exposes 503 bypassing error normalization to 500

Medium Severity

Adding retry-on conn-failure empty-response changes the failure path: previously a connection failure produced a 502 (caught by errorfile 502 and normalized to 500), but now retry exhaustion produces a 503 which has no matching errorfile 503 directive. This breaks the stated design goal of normalizing proxy errors to 500 and exposes clients to raw 503 responses with HAProxy's built-in error page instead of the expected "Internal Server Error" body. The test in test_http_routes.py was loosened to accept in (500, 503) rather than fixing the gap.

Additional Locations (1)

• python/ray/serve/_private/haproxy_templates.py#L64-L68

 

^{Reviewed by Cursor Bugbot for commit 247b07c. Configure here.}

[harshit-anyscale]

latest 3 runs with these changes:

1. https://console.anyscale-staging.com/cld_kvedZWag2qA8i5BjxUevf5i7/prj_g7p6lsu6r8g7garwbxifppyz23/jobs/prodjob_vw3cjwn9y4h2eizhni7w71pizh?job-tab=overview&job-logs-section-tabs=application_logs

2. https://console.anyscale-staging.com/cld_kvedZWag2qA8i5BjxUevf5i7/prj_g7p6lsu6r8g7garwbxifppyz23/jobs/prodjob_3btnqzv933zh9nsum2z6bdzdip?job-tab=overview&job-logs-section-tabs=application_logs

https://console.anyscale-staging.com/cld_kvedZWag2qA8i5BjxUevf5i7/prj_g7p6lsu6r8g7garwbxifppyz23/jobs/prodjob_cdfkbj1cik2zpedd1w7wgrl79y?job-tab=overview&job-logs-section-tabs=application_logs

number of failed requests < 0.01%

[kouroshHakha]

Great work @harshit-anyscale. We should potentially merge this after we enable ha proxy for the performance tests and let those cook a bit (#63386). I want to make sure we are not trading off stability for speed.

[harshit-anyscale]

We should potentially merge this after we enable ha proxy for the performance tests and let those cook a bit (#63386)

makes sense; cool, will plan to merge it after 3-4 days of merging of #63386 .