Skip to content

fix(sglang): implement Status RPC to unblock backend-monitor polling#10867

Merged
mudler merged 1 commit into
mudler:masterfrom
pos-ei-don:fix/sglang-backend-status-method-shim
Jul 16, 2026
Merged

fix(sglang): implement Status RPC to unblock backend-monitor polling#10867
mudler merged 1 commit into
mudler:masterfrom
pos-ei-don:fix/sglang-backend-status-method-shim

Conversation

@pos-ei-don

Copy link
Copy Markdown
Contributor

Problem

The sglang Python backend inherits the default Status RPC from backend_pb2_grpc.BackendServicer, which raises NotImplementedError. LocalAI's backend-monitor polls /backend.Backend/Status periodically on every registered backend; when the call fails, /backend/monitor returns HTTP 500 and downstream inference requests to the sglang backend get blocked — even though the model is loaded and answering fine on its direct gRPC endpoint.

Fix

Add a minimal Status shim that mirrors the existing Health method and returns StatusResponse{state=READY} unconditionally. This unblocks the monitor path immediately. A state-aware follow-up (UNINITIALIZED during load, BUSY under active inference) would be an improvement but is intentionally out of scope for this MVP — the current change matches what Health does today, which is what registered backends have effectively been telling the monitor loop anyway.

Reproducer / verification

Hardware: DGX Spark (GB10, arm64-l4t-cuda-13 image). Backend: cuda13-nvidia-l4t-arm64-sglang at sglang v0.5.15, model ucbye/Qwen3-Coder-Next-NVFP4-GB10 (80B NVFP4). Before the shim, /backend/monitor returns 500 and inference requests routed via LocalAI hang; direct gRPC-level calls to the sglang server work. Patching the shim in place on the running container instantly restores both /backend/monitor and inference to the sglang slot. Same pattern has been carrying prod for the sglang coder there since 2026-07-15 (~24 h uptime).

Notes for reviewers

  • The change is 8 lines in backend/python/sglang/backend.py; nothing else is touched.
  • Other Python backends (e.g. vllm, llama-cpp-python) also don't implement Status today — if the monitor polls them too, the same shim would apply. I've only reproduced the failure end-to-end with sglang so this PR is intentionally sglang-only; happy to file follow-ups if useful.
  • Cross-ref: same pattern of "sglang wrapper missing a method that vllm has" as fix(sglang): parse tool_call function arguments before applying the chat template #10558 (function.arguments string→dict parse).

The sglang Python backend inherits the default Status RPC from
backend_pb2_grpc.BackendServicer, which raises NotImplementedError.
LocalAI's backend-monitor polls /backend.Backend/Status periodically on
every registered backend; when the call fails, /backend/monitor returns
HTTP 500 and downstream inference requests to the sglang backend are
blocked even though the model is loaded and answering directly via the
gRPC endpoint.

Add a minimal Status shim that mirrors the existing Health method and
returns StatusResponse{state=READY} unconditionally. This unblocks the
monitor path; a state-aware follow-up (UNINITIALIZED during load, BUSY
under active inference) is left for a subsequent change.

Reproduced on DGX Spark (GB10, arm64-l4t-cuda-13 image) with the sglang
v0.5.15 backend and Qwen3-Coder-Next-NVFP4-GB10; verified locally that
patching the shim in place immediately restores /backend/monitor and
inference across the sglang slot.

@localai-bot localai-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed against the codebase — the fix is correct and verified:

  • core/services/monitoring/backend_monitor.go:93 does poll Status on every registered backend, and the sglang servicer indeed inherits the unimplemented default, so the failure mode described (500 on /backend/monitor for a healthy loaded model) is real.
  • The shim matches the established pattern in other Python backends (speaker-recognition, insightface, rfdetr all implement the same minimal Status).
  • Unconditional READY is the right call here rather than a state-aware variant: self.llm only exists after LoadModel, and the monitor only samples models that already passed CheckIsLoaded, so Health-mirroring behavior is consistent.

One blocker before merge: the DCO check is failing — please amend your commit with a Signed-off-by line (git commit --amend -s and force-push). Cosmetic nit, feel free to ignore: there's a double blank line before Status and none between it and LoadModel.

cc @mudler — good to merge once DCO is fixed.

@mudler
mudler merged commit e62221b into mudler:master Jul 16, 2026
1 check failed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants