Repository navigation
Replies: 1 comment
|
Follow-up with hard numbers, and a concrete proposal for pieces 1+2 (batch budget, per-text cap). We counted tokens exactly against the same llama.cpp server our embedder uses ( Measured
So the recipe default Proposal: for llama.cpp backends, ask the server instead of estimating.
We are building it against current master and will open a PR with tests (CJK density, long single input, endpoint-down fallback, and a default-path-unchanged assertion). We are not asking you to adopt #4874 or #5071 as filed - both were the estimate-side workaround; this replaces the estimate for the one backend where it can be replaced. If you would prefer a different shape - a boolean recipe flag instead of an endpoint descriptor, or usage-based calibration from |
Uh oh!
There was an error while loading. Please reload this page.
Moved from #5075, opened by @G0-0000 on 2026-09-13. Feature requests now live in Discussions > Ideas.
Summary
Production environment: gbrain (upgraded over time from v0.48.x to v0.50.0.0) + self-hosted qwen3-embedding-4b GGUF (served by llama.cpp
llama-server) + Postgres 16/pgvector, ~2.3k pages / ~20.6k chunks, mostly Chinese corpus. Six weeks of production running has exposed four gaps between gbrain's embedding pipeline and llama.cpp server behavior. Two are filed as PRs (#4874, #5071); two remain local patches, which we're glad to upstream in whatever form the maintainers prefer. This issue is an umbrella report — happy to split it into independent PRs anytime.The four pieces
1. Batch token budget for llama-server recipe — PR #4874 (open)
The recipe only sets
max_batch_items: 32; without a token budget the gateway takes the fast path and sends the whole batch unsplit, and CJK-heavy batches overflow llama.cpp's context and fail. Local patch:max_batch_tokens: 4096/chars_per_token: 1. Upstream'sGBRAIN_EMBED_MAX_BATCH_TOKENS(#4650) is a global knob; a recipe-level field lets a mixed fleet set budgets per server.2. Per-text character cap — PR #5071 (open)
MAX_CHARS = 8000is hard-coded; over-long inputs hang llama-server (timeouts). PR adds aGBRAIN_EMBED_MAX_CHARSenv override, byte-for-byte identical by default. Production proof:=2000and a full re-embed of 17,106 chunks with zero timeouts.3. Query-side instruction prefix — reviving #3584's shape
Qwen3-Embedding's official query/document asymmetry means queries need an instruction prefix. @paul-0320's #3584 had a very good shape (query-only,
Instruct: {task}\nQuery:,GBRAIN_QUERY_INSTRUCTenv, empty-string kill switch, knobsHash, standalonequery-instruct.tsmodule) — sadly closed unmerged, but the need remains. Our local variant (Chinese task sentence) measurably rotates the embedding space (cosine ≈ 0.8875 between prefixed and bare text), and A/B evaluation shows better retrieval alignment. If welcome, we can re-file in #3584's original shape (English default, env-overridable).4. MRL truncation client-side — llama.cpp ignores
dimensionsgbrain trusts the provider-declared dimensions (the conclusion of #2692), but llama-server silently ignores the OpenAI-style
dimensionsparameter and always returns the native 2560 dims. Live red test against our production llama-server:(direct
llama-server --embedding, not a LocalAI-style layer). pgvector's HNSW cap is 2000 dims, so client-side MRL truncation + L2 normalization is the only way through. Production proof: 2560→1792 unlocks HNSW, top-10 retrieval ~308ms → ~10ms; full re-embed of all 20,613 chunks with zero errors; post-truncation L2 norm ≈ 1.0. Suggested forms: an MRL-family safety net in gbrain (truncate only when the returned vector width exceeds the configuredembedding_dimensions, a no-op for compliant providers), or a llama-server recipe flag, or pushing a server-side fix to llama.cpp — maintainer's call.What we're asking
All four patches are battle-tested in production; any additional data, red tests, or reshape requests — just say the word.
All reactions