docs(config): document grpc.attempts timing + tuning guidance#10868
Merged
mudler merged 1 commit intoJul 16, 2026
Conversation
The gRPC configuration table only listed the two fields with a one-line description each, without defaults, without explaining what the total load window looks like, and without hinting when a user should adjust them. In practice the default 20 attempts x 2 s = 40 s window is way too tight for large NVFP4 / FP8 models on slow storage or first-run CUDA-graph capture, and the resulting kill (exitCode=120, 'context canceled') looks like a backend crash even though the backend is still making legitimate forward progress. Extend the section with: - Defaults column (20 and 2) added to the table - Prose explaining that these govern the readiness handshake between LocalAI and a freshly spawned backend (Health polling loop) - Total-load-window formula - Concrete failure signature so users can recognize a timeout-kill vs. a real backend crash - Example configuration for a ~10 min cold-load window (grpc.attempts 140, attempts_sleep_time 5), with a note that inference-timeouts and the watchdog are unaffected.
localai-bot
approved these changes
Jul 16, 2026
localai-bot
left a comment
Collaborator
There was a problem hiding this comment.
Verified the documented values against the code — all accurate:
- Defaults
20/2: confirmed atpkg/model/loader_options.go:111-112. - The health-poll loop and kill-on-timeout behavior: confirmed at
pkg/model/initializers.go:146-163(pollHealthCheck, sleepgrpcAttemptsDelay, thenstopLoadProcess+grpc service not ready). - The framing (readiness handshake only, not request timeouts / watchdog) is correct.
This is a genuinely useful addition — the exitCode=120 + context canceled failure signature is a common source of confusion for large-model cold loads.
One blocker before merge: the DCO check is failing — please amend with Signed-off-by (git commit --amend -s + force-push).
cc @mudler — good to merge once DCO is fixed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Extends the (very minimal) gRPC Configuration section in
docs/content/advanced/model-configuration.mdwith defaults, a total-load-window formula, a failure-signature hint, and a concrete example for models that need a longer readiness window.Why
The current table lists
grpc.attemptsandgrpc.attempts_sleep_timewith a one-line description each. It doesn't say what the defaults are (they're20and2inpkg/model/loader_options.go), what timing behavior they produce end-to-end, or when a user should touch them.In practice the default
20 × 2 s ≈ 40 swindow is too tight for large NVFP4 / FP8 models on slow storage or first-run CUDA-graph capture. The resulting kill shows up in the LocalAI log asexitCode=120plusrpc error: code = Canceled desc = context canceled— which reads like a backend crash even though the backend's own stderr shows continued forward progress (e.g.Multi-thread loading shards: 3/10 [01:22<03:43]). Without knowing the timing math, users spend time chasing the wrong root cause (I did, before I traced it back topkg/model/initializers.go:145-160).What the doc adds
20and2)grpc.attempts × (grpc.attempts_sleep_time + per-call gRPC dial timeout)attempts: 140,attempts_sleep_time: 5), with a note that inference-request timeouts and the watchdog are unaffectedVerification
Docs-only change; no code paths touched. Verified locally on DGX Spark (GB10, sglang v0.5.15,
ucbye/Qwen3-Coder-Next-NVFP4-GB10) that the example values do produce a stable ~9 min cold-load; without them the load consistently died at ~90 s.Cross-refs