Skip to content

v0.1.36 - KV-friendly prompt assembly hotfix

Choose a tag to compare

@lyc-aon lyc-aon released this 11 Apr 04:33

This hotfix fixes a real performance regression in long local-model sessions: Successor now preserves a KV-cache-friendly request shape instead of quietly rewriting earlier prompt bytes as the conversation grows.

What changed

  • made prompt assembly append-only in the places that matter for local cache reuse
    • runtime-tail context is now always appended as its own synthetic turn instead of being merged into the latest real user message
    • adjacent same-role API messages now preserve their original boundaries instead of being merged together later
  • stabilized native tool-call serialization
    • assistant tool_calls[].function.arguments now use canonical JSON ordering so identical tool payloads serialize to identical bytes
  • added better cache-shape diagnostics
    • turn traces and /perf now record request prefix hash, full request hash, and tail kind alongside the existing stable-system hash and llama timing data
    • this makes it possible to distinguish harness-side request-shape churn from provider-side cache dropout

Verification

  • ruff check src tests
  • PYTHONPATH=src pytest -q
    • 1277 passed in 25.56s
  • live llama.cpp KV verification
    • before fix: repeated late-turn cache_n=0, median first-token latency ~34s, suspected KV misses across many turns
    • after fix: cache_n grows turn-to-turn, hit ratio stays roughly 0.8–0.94, and first-token latency stays sub-second in the extended local-model session