fix(kb): reindex waits out a cold local embed model (don't fail the first try)#32
Merged
Merged
Conversation
…irst try) Reindex/embeddings used a 15s timeout, but a cold local embed model (e.g. MLX/oMLX loading bge-m3) takes ~50s on its first request. So the first reindex was cut off at 15s and failed, the model finished loading in the background, and only the SECOND (warm) reindex worked — exactly the "failed first, worked on the second try" seen in the logs (bifrost recorded those embeds completing at 51s/47s/32s). This is the embedding-path twin of the Phase 0 chat cold-load fix: - embed_doc takes a per-embed `timeout`; the backfill path (reindex_pending, used by the manual /api/kb/reindex route AND the scheduler's auto/eager reindex) uses a generous 120s so a cold model is waited out. bifrost's per-provider timeout (300s) is the real upstream ceiling. - Interactive embed-on-add / rename stay at 15s (fail fast; best-effort, backfilled later) so an upload never blocks minutes on a cold model. - gateway.embed now passes the timeout per-request and surfaces an httpx timeout as a clean 504 instead of a raw error. Tests: timeout threads through embed_doc; reindex uses the long timeout while embed-on-add stays fast; a gateway timeout wraps to a 504. Full suite green (650).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Diagnosis (from your logs)
Reindex/embeddings used a 15s timeout, but a cold local embed model takes far longer to load on its first request. bifrost recorded the (eventually successful) embeds completing at
request_duration_ms: 51748, 47845, 32945— ~50s for oMLX to loadbge-m3-mlx-fp16. So:0 embedded, 1 failed. (oMLX keeps loading in the background — that's why bifrost logs those requests finishing at 51s.)Exactly your "failed first, worked on the second try." It's the embedding-path twin of the Phase 0 chat cold-load fix (#25) — I raised the timeout for the chat/agent path but not for embeddings.
Fix
embed_doctakes a per-embedtimeout. The backfill path (reindex_pending— used by the manual Reindex button and the scheduler's auto/eager reindex) uses a generous 120s so a cold model is waited out instead of cut at 15s. (bifrost's per-provider timeout is already 300s from feat(routing): dedicated Agent-tasks model route + cold-load timeout fix #25 — the upstream ceiling.)gateway.embednow passes the timeout per-request and surfaces an httpx timeout as a clean504instead of a raw error.Net effect: a single Reindex click now succeeds on a cold model, and the automatic background reindex will also backfill a just-uploaded doc without you having to click twice.
Tests
Timeout threads through
embed_doc; reindex uses the long timeout while embed-on-add stays fast; a gateway timeout wraps to504. Full backend suite green (650), ruff clean.Verified by log analysis + plumbing tests; worth confirming on your next genuinely-cold start.