Skip to content

Advanced Context Manager

Choose a tag to compare

View all tags
@ayder ayder released this 23 Mar 02:59
· 459 commits to main since this release
release-1.6.1
a54ce26

Overview

Ground-up rewrite of the context manager for Ollama, replacing the blind, cache-destroying implementation with a KV-cache-aware system built on the native ollama Python SDK. The old context manager reordered messages, mutated tool results in-place, and let the system prompt drift between calls — all of which invalidated Ollama's KV-cache and forced full recomputation on every request. The new system is designed around how Ollama actually works.

What Changed

Native Ollama SDK Integration

  • Replaced OpenAI compatibility layer with ollama.AsyncClient for direct API access
  • Full timing data: prompt_eval_duration, eval_duration, load_duration now available on every response
  • Native thinking support via message.thinking field (not parsed from content)
  • keep_alive=-1 on every request prevents model unloading mid-session
  • Native tool calling with proper Mapping[str, Any] arguments (not JSON strings)

KV-Cache-Aware Context Management

  • Stable-prefix layout: system prompt frozen at session start, messages are append-only, tool results truncated at insertion time — the KV-cache prefix is never invalidated
  • Real token counts: compaction decisions use prompt_eval_count from Ollama, not character-based estimates
  • Heuristic compaction: when context fills up, oldest message units are replaced with a text summary (no LLM call)
  • Auto-detected context length: queries /api/show on startup to get the model's real context length, falls back to num_ctx if detection fails

Cache Intelligence

  • CacheMonitor: tracks ns/token ratio to detect KV-cache hits (hot/warm/miss) via speed comparison against cold baseline
  • Adaptive compaction: when cache is hot, compaction threshold pushed from 70% to 90% to preserve the cache
  • Logs cache state on every LLM response: Context: 2540/65536 tokens (4%), completion=375, speed=8500 ns/tok, cache=hot(76%)

Architecture (OCP)

  • ContextManager Protocol: abstract contract with 5 methods (freeze_system_prompt, prepare_messages, update_from_response, get_stats, should_compact)
  • ContextManagerFactory: registry-based factory mapping driver names to implementations. New providers register without modifying existing code.
  • DefaultContextManager: extracted from old monolith for non-Ollama providers (Anthropic, OpenAI, Gemini). Same tiered pruning behavior, now behind the Protocol.
  • OllamaContextManager: new implementation with all KV-cache optimizations
  • Dependency injection: RuntimeComponents carries context_manager, ChatLoop receives it via constructor

Ollama SDK Compliance

  • Rewrote _convert_messages() for full pydantic Message.model_validate() compliance
  • Tool results use tool_name (SDK field) instead of name (silently dropped)
  • Tool call arguments converted from JSON strings to dict (SDK's Mapping[str, Any] type)
  • Dummy/whitespace tool calls filtered out
  • 13 regression tests validate against the real ollama.Message.model_validate()

Agent System Fixes

  • CLI path: agents now initialized in cli_runner.py (was TUI-only)
  • Late-freeze: system prompt captures agent capability prompts appended after create_runtime(), then re-freezes for KV-cache stability
  • Improved prompts: LLM told exact agent names, can dispatch same agent multiple times, won't use unrelated agents

Observability

  • All try/except blocks in ollama.py and chat_loop.py now log warnings (no silent exceptions)
  • INFO log when Ollama falls back to XML protocol
  • LLM checkpoints logged: system prompt freeze, token usage/utilization, cache state, compaction triggers, context length detection

New Tools

  • save_context_memory / load_context_memory: manual named context snapshots to .ayder/memory/<name>.json
Assets 2
Loading