fix: refresh health heartbeat on request admission to avoid 503 during long prefill after idle - #1432
Merged
Conversation
sufubao
force-pushed
the
fix/health-503-after-idle-prefill
branch
4 times, most recently
from
August 5, 2026 10:43
74a82d6 to
5d5970a
Compare
…g long prefill after idle
sufubao
force-pushed
the
fix/health-503-after-idle-prefill
branch
from
August 5, 2026 11:07
5d5970a to
43858b5
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
问题
/health在「服务长时间闲置后,突然来了一个长请求」的场景下,会在首个输出 token 产生之前持续返回 503。根因
健康判定逻辑(
lightllm/utils/health_check.py):HEALTH_TIMEOUT(默认 200s)或(无在途请求 且 shm 空闲)latest_success_infer_time_mark只在吐出 token 时刷新闲置期间
last_success早已过期,靠「无在途请求」分支兜底;一旦有请求准入,该分支瞬间失效,而 grace 窗口又已耗尽 —— 出现断层。于是长 prefill 的整段都判为不健康,会被 LB/探针误踢下线。修复
在请求准入时(
generate()入口,自增run_reqs_count的同一把锁内)刷新一次心跳时间戳,重置 grace 窗口,让 prefill 阶段被覆盖。lightllm/server/httpserver/manager.py:准入刷新latest_success_infer_time_marklightllm/server/httpserver_for_pd_master/manager.py:准入刷新latest_success_infer_time(对称逻辑,同样的漏洞)取舍
准入即刷新会带来一个已知放宽:当请求源源不断地准入时,即便后端不再吐 token,心跳也会被持续刷新,
/health不会变 503(即「busy 但卡死」的检测能力在该持续流量下被削弱)。本 PR 优先保证闲置后长 prefill 不被误判下线这一线上实际问题;卡死检测可后续结合 token 产出单独加强。