Add llama KV prefix cache reuse

[FuJacob]

Summary

Adds KV-cache reuse to the local llama runtime so consecutive autocomplete requests can reuse the decoded common prompt prefix instead of rebuilding a fresh context every time. The runtime still tokenizes and validates the shared token prefix before trusting cached native state, and coordinator lifecycle paths reset the cache on focus, settings, model, or disabled-state changes.

Validation

• xcodebuild build -scheme tabby -project tabby.xcodeproj -destination 'platform=macOS'
  • ** BUILD SUCCEEDED **
• xcodebuild test -scheme tabby -project tabby.xcodeproj -destination 'platform=macOS' CODE_SIGNING_ALLOWED=NO CODE_SIGNING_REQUIRED=NO
  • ** TEST SUCCEEDED **, 30 tests, 0 failures
• Local 10-character typing microbench against installed Qwen3-0.6B-Q4_K_M.gguf
  • Ran selected LlamaRuntimeCacheMicrobenchTests.test_tenCharacterTypingSession_reportsCachedPrefixLatencyDelta locally to quantify the ticket's latency-drop acceptance criterion.
  • ** TEST SUCCEEDED **, 1 selected test, 0 failures
  • Requests: 10-character typing session
  • Baseline total: 851.8 ms
  • Cached total: 323.2 ms
  • Latency drop: 62.1%
  • Baseline avg/p95: 85.2 / 94.9 ms
  • Cached avg/p95: 32.3 / 79.2 ms
  • Baseline per request: 90.0, 78.2, 74.9, 87.9, 88.4, 94.9, 91.1, 81.3, 81.8, 83.3 ms
  • Cached per request: 79.2, 23.1, 23.3, 29.2, 27.8, 28.5, 30.0, 29.9, 25.8, 26.5 ms

Linked issues

Refs #23
Refs #13

Risk / rollout notes

• The cache is intentionally owned by LlamaRuntimeCore, because native llama contexts and KV memory must stay behind the serialized runtime actor.
• Generated continuation tokens are removed after sampling so the retained context represents only the prompt before the next request.
• The microbench was run locally against a real installed GGUF rather than added as a committed test harness, keeping the PR diff focused on the runtime implementation.
• XCTest reported success for the local microbench, but llama.cpp/Metal printed a shutdown-time GGML_ASSERT([rsets->data count] == 0) backtrace after test completion on this machine.

[Copilot]

Pull request overview

Adds KV-cache reuse to the local llama runtime so consecutive autocomplete requests can reuse the decoded common prompt prefix instead of rebuilding a fresh context each time, and wires lifecycle invalidation/reset paths through the coordinator/router.

Changes:

• Introduces a reusable prompt context + KV trimming/validation path in LlamaRuntimeCore with an optional cachedPrefixBytes hint.
• Threads deterministic sampling (randomSeed) and cache-hint tracking through request/config and the llama engine.
• Adds a unified resetCachedGenerationContext() contract and calls it from coordinator lifecycle/focus/settings transitions.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tabbyTests/LlamaPromptRendererTests.swift	Adds unit tests for the prompt cache hint tracker and a helper to build SuggestionRequests.
tabby/Support/SuggestionRequestFactory.swift	Plumbs randomSeed from configuration into built requests.
tabby/Services/Runtime/SuggestionEngineRouter.swift	Adds router fan-out reset for cached generation context.
tabby/Services/Runtime/LlamaSuggestionEngine.swift	Adds cache-hint tracker integration, seed plumbing, and cache reset behavior.
tabby/Services/Runtime/LlamaRuntimeManager.swift	Extends generate API with cache hint + seed; adds prompt-cache reset entry point.
tabby/Services/Runtime/LlamaRuntimeCore.swift	Implements reusable prompt context, KV trimming, token-level validation, and deterministic sampler seeding.
tabby/Services/Runtime/FoundationModelSuggestionEngine.swift	Implements resetCachedGenerationContext() as a no-op for stateless sessions.
tabby/Models/SuggestionSubsystemContracts.swift	Extends SuggestionGenerating with resetCachedGenerationContext().
tabby/Models/SuggestionModels.swift	Adds randomSeed to configuration and request models.
tabby/App/Coordinators/SuggestionCoordinator+Prediction.swift	Resets cached generation context when disabling predictions.
tabby/App/Coordinators/SuggestionCoordinator+Lifecycle.swift	Resets cached generation context on stop/model switch/settings change.
tabby/App/Coordinators/SuggestionCoordinator+Input.swift	Resets cached generation context when focused element changes.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.