docs(selfhost): document OLLAMA_KV_CACHE_TYPE and recommend bge-m3#4404
Merged
Conversation
Documents two GPU-tuning env vars that existed as an Ollama runtime capability but were never surfaced to self-host operators: OLLAMA_FLASH_ATTENTION (required alongside KV cache quantization for it to take effect) and OLLAMA_KV_CACHE_TYPE=q8_0 (roughly halves per-context KV-cache VRAM with negligible quality impact -- validated live: 4608 MiB at f16 down to 2448 MiB at q8_0 for one loaded context, same generation speed, correct output). Also updates the AI_EMBED_MODEL/QDRANT_DIM example to recommend bge-m3 (1024-d) over nomic-embed-text (768-d) as the default -- it already matches the code's own DEFAULT_DIM assumption, its longer native context window is a better fit for RAG's ~16k-char chunk budget, and it's now the validated production choice.
Contributor
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
github-project-automation
Bot
moved this to Todo
in Self-Hosted Review Stack (LoopOver Orb)
github-project-automation
Bot
moved this from Todo
to Done
in Self-Hosted Review Stack (LoopOver Orb)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
OLLAMA_FLASH_ATTENTION+OLLAMA_KV_CACHE_TYPEin.env.example— an existing Ollama runtime capability that was never surfaced to self-host operators.q8_0roughly halves per-context KV-cache VRAM with negligible quality impact; validated live (4608 MiB at f16 -> 2448 MiB at q8_0 for one loaded context, same generation speed, correct structured-output quality on a real request).AI_EMBED_MODEL/QDRANT_DIMexample to recommend bge-m3 (1024-d) over nomic-embed-text (768-d) — bge-m3 already matches the code's ownDEFAULT_DIMassumption (seesrc/selfhost/qdrant-vectorize.ts), its longer native context window is a better fit for RAG's ~16k-char chunk budget, and it's now the validated production choice.Documentation-only change; no code paths affected.
Scope
wantedPaths(repo-root.env.example)site/,CNAME,**/lovable/**, orCHANGELOG.mdValidation
npm run selfhost:env-reference:check— clean (noenv.*source reads changed, comment-only edit)git diff --check— cleanSafety