Skip to content

docs(config): document grpc.attempts timing + tuning guidance#10868

Merged
mudler merged 1 commit into
mudler:masterfrom
pos-ei-don:docs/grpc-attempts-timing-guidance
Jul 16, 2026
Merged

docs(config): document grpc.attempts timing + tuning guidance#10868
mudler merged 1 commit into
mudler:masterfrom
pos-ei-don:docs/grpc-attempts-timing-guidance

Conversation

@pos-ei-don

Copy link
Copy Markdown
Contributor

What

Extends the (very minimal) gRPC Configuration section in docs/content/advanced/model-configuration.md with defaults, a total-load-window formula, a failure-signature hint, and a concrete example for models that need a longer readiness window.

Why

The current table lists grpc.attempts and grpc.attempts_sleep_time with a one-line description each. It doesn't say what the defaults are (they're 20 and 2 in pkg/model/loader_options.go), what timing behavior they produce end-to-end, or when a user should touch them.

In practice the default 20 × 2 s ≈ 40 s window is too tight for large NVFP4 / FP8 models on slow storage or first-run CUDA-graph capture. The resulting kill shows up in the LocalAI log as exitCode=120 plus rpc error: code = Canceled desc = context canceled — which reads like a backend crash even though the backend's own stderr shows continued forward progress (e.g. Multi-thread loading shards: 3/10 [01:22<03:43]). Without knowing the timing math, users spend time chasing the wrong root cause (I did, before I traced it back to pkg/model/initializers.go:145-160).

What the doc adds

  • A Default column in the config table (20 and 2)
  • Prose framing: these govern the readiness handshake between LocalAI and a freshly spawned backend (Health-polling loop), not runtime request timeouts
  • The total-load-window approximation: grpc.attempts × (grpc.attempts_sleep_time + per-call gRPC dial timeout)
  • The concrete failure signature so users can recognize a timeout-kill vs. a real backend crash
  • Example values for a ~10 min cold-load window (attempts: 140, attempts_sleep_time: 5), with a note that inference-request timeouts and the watchdog are unaffected

Verification

Docs-only change; no code paths touched. Verified locally on DGX Spark (GB10, sglang v0.5.15, ucbye/Qwen3-Coder-Next-NVFP4-GB10) that the example values do produce a stable ~9 min cold-load; without them the load consistently died at ~90 s.

Cross-refs

The gRPC configuration table only listed the two fields with a one-line
description each, without defaults, without explaining what the total
load window looks like, and without hinting when a user should adjust
them. In practice the default 20 attempts x 2 s = 40 s window is way
too tight for large NVFP4 / FP8 models on slow storage or first-run
CUDA-graph capture, and the resulting kill (exitCode=120, 'context
canceled') looks like a backend crash even though the backend is still
making legitimate forward progress.

Extend the section with:
- Defaults column (20 and 2) added to the table
- Prose explaining that these govern the readiness handshake between
  LocalAI and a freshly spawned backend (Health polling loop)
- Total-load-window formula
- Concrete failure signature so users can recognize a timeout-kill
  vs. a real backend crash
- Example configuration for a ~10 min cold-load window (grpc.attempts
  140, attempts_sleep_time 5), with a note that inference-timeouts and
  the watchdog are unaffected.

@localai-bot localai-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified the documented values against the code — all accurate:

  • Defaults 20 / 2: confirmed at pkg/model/loader_options.go:111-112.
  • The health-poll loop and kill-on-timeout behavior: confirmed at pkg/model/initializers.go:146-163 (poll HealthCheck, sleep grpcAttemptsDelay, then stopLoadProcess + grpc service not ready).
  • The framing (readiness handshake only, not request timeouts / watchdog) is correct.

This is a genuinely useful addition — the exitCode=120 + context canceled failure signature is a common source of confusion for large-model cold loads.

One blocker before merge: the DCO check is failing — please amend with Signed-off-by (git commit --amend -s + force-push).

cc @mudler — good to merge once DCO is fixed.

@mudler
mudler merged commit 06b4a29 into mudler:master Jul 16, 2026
1 check failed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants