Skip to content

fix: refresh health heartbeat on request admission to avoid 503 during long prefill after idle - #1432

Merged
sufubao merged 1 commit into
ModelTC:mainfrom
sufubao:fix/health-503-after-idle-prefill
Aug 5, 2026
Merged

fix: refresh health heartbeat on request admission to avoid 503 during long prefill after idle#1432
sufubao merged 1 commit into
ModelTC:mainfrom
sufubao:fix/health-503-after-idle-prefill

Conversation

@sufubao

@sufubao sufubao commented Aug 5, 2026

Copy link
Copy Markdown
Collaborator

问题

/health 在「服务长时间闲置后,突然来了一个长请求」的场景下,会在首个输出 token 产生之前持续返回 503。

根因

健康判定逻辑(lightllm/utils/health_check.py):

  • 健康 = 距上次成功推理 ≤ HEALTH_TIMEOUT(默认 200s)(无在途请求 且 shm 空闲)
  • 心跳 latest_success_infer_time_mark 只在吐出 token 时刷新

闲置期间 last_success 早已过期,靠「无在途请求」分支兜底;一旦有请求准入,该分支瞬间失效,而 grace 窗口又已耗尽 —— 出现断层。于是长 prefill 的整段都判为不健康,会被 LB/探针误踢下线。

修复

在请求准入时(generate() 入口,自增 run_reqs_count 的同一把锁内)刷新一次心跳时间戳,重置 grace 窗口,让 prefill 阶段被覆盖。

  • lightllm/server/httpserver/manager.py:准入刷新 latest_success_infer_time_mark
  • lightllm/server/httpserver_for_pd_master/manager.py:准入刷新 latest_success_infer_time(对称逻辑,同样的漏洞)

取舍

准入即刷新会带来一个已知放宽:当请求源源不断地准入时,即便后端不再吐 token,心跳也会被持续刷新,/health 不会变 503(即「busy 但卡死」的检测能力在该持续流量下被削弱)。本 PR 优先保证闲置后长 prefill 不被误判下线这一线上实际问题;卡死检测可后续结合 token 产出单独加强。

@sufubao
sufubao force-pushed the fix/health-503-after-idle-prefill branch 4 times, most recently from 74a82d6 to 5d5970a Compare August 5, 2026 10:43
@sufubao
sufubao force-pushed the fix/health-503-after-idle-prefill branch from 5d5970a to 43858b5 Compare August 5, 2026 11:07
@sufubao
sufubao merged commit 499a2b1 into ModelTC:main Aug 5, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant