Skip to content

Releases

v0.1.46 - Memory Lifecycle, Dynamic Working Set, Swap Prefetch & Reasoning Recovery

v0.1.46 - Memory Lifecycle, Dynamic Working Set, Swap Prefetch & Reasoning Recovery Latest
Latest

Choose a tag to compare

View all tags
@0xra0 0xra0 released this 09 Nov 11:04
v0.1.46
7306db6

Strata v0.1.46: Memory Lifecycle, Dynamic Working Set, Swap Prefetch & Inference Reliability

This release introduces significant optimizations and stability enhancements for running 125B MoE models (Qwen3.8-Flash-Next, Coder, Swift 1.5, and Unsloth variants) on consumer PCs with a single NVIDIA or AMD graphics card and system RAM.

🌟 What's New & Key Highlights

1. Idle GPU Shadows Reclaim (STRATA_MEMORY_LIFECYCLE=1)

  • Safe Idle Deallocation: The resident expert complement holds pageable host copies of GPU-cache experts to enable prefill scratch borrowing. While idle, duplicate host pages are safely reclaimed without losing weight integrity or requiring disk re-reads.
  • Live Ownership v2: Validates against the live dynamic host_res routing table rather than the stale startup profile (ExpertCache::slot_of()), eliminating wrong-expert restoration bugs and repetitive generation loops during long conversational sessions.
  • Synchronized Bounce Buffer: Leverages HIP/CUDA pinned bounce buffers for host-device copies, avoiding multi-gigabyte pageable registrations with the driver.

2. CPU Vision Compute Workspace Release (vision.release_workspace: true)

  • Transient Memory Freeing: The CPU vision encoder context now frees its GGML graph scheduler compute workspace buffers immediately after producing image embeddings (strata-vision --release-workspace) and trims libc heap memory before engine prefill begins.
  • Weight Retention: Vision model and projector weights remain loaded at unchanged memory addresses for zero-latency subsequent inference.

3. Eviction-First GDN Checkpoint Reuse

  • Reuses the state-vector allocations of the LRU victim checkpoint directly when capturing new GDN checkpoints, eliminating temporary double-allocation spikes during long context chats.

4. Long-Context Dynamic RAM Working Set (STRATA_RAM_WORKING_SET=1)

  • In-Place Hot-Expert Promotion: Dynamically replaces colder CPU-only resident experts with frequently routed file-tier experts within the existing complement allocation (bounded at 4 experts / 8 MiB per pass).
  • Storage Read Reduction: Achieves up to 86.7% fewer logical disk reads on repetitive multi-turn prompts without allocating any additional SSD or RAM arenas.

5. Decoupled Thinking Allowance (reasoning_auto_budget_tokens)

  • Prevents runaway reasoning from consuming massive portions of expanding context windows (e.g. 50,000+ tokens on 64k+ contexts).
  • Sets a sensible, effort-specific default budget (e.g. 4,096 tokens for medium/high, 2,048 for low, 8,192 for xhigh) while preserving user-specified reasoning_budget_tokens.

6. Resident Swap Read-Ahead / Prefetch (STRATA_RESIDENT_SWAP_PREFETCH=1)

  • Samples VmSwap and uses kernel-level mincore probing combined with asynchronous MADV_WILLNEED advice to proactively swap in predicted resident experts paged out by Linux.
  • Features rate caps (128 MiB / 100 ms) and a strict 2,560 MiB free system RAM floor.

7. Cache-Avoidant Prompt Stream Reader (STRATA_PROMPT_STREAM=1)

  • Uses preadv2(RWF_DONTCACHE) and page-aligned O_DIRECT for reading prompt expert spans directly from disk.
  • Protects the OS page cache from being flooded by multi-gigabyte prompt prefill streams, ensuring host resident experts remain in physical RAM.

8. Reasoning-Only Completion Recovery (reasoning_close_retry v2)

  • Detects when a reasoning model reaches natural EOS after thinking without outputting an answer or tool call.
  • Executes an automatic single continuation within the same HTTP connection/stream, ensuring tools like Zed editor, Claude Code, and Cursor receive a complete answer.

🛠️ Rebuild Scripts & Tooling Included

  • tools/rebuild-memory-fix.sh
  • tools/rebuild-long-context-fix.sh
  • tools/rebuild-ownership-fix.sh
  • tools/rebuild-swap-prefetch-fix.sh
  • tools/rebuild-prompt-stream-fix.sh
  • tools/enable_reasoning_answer_fix.py
  • tools/check_translation_http.py

🧪 Verification & Validation

  • Unit & Integration Tests: 900+ test suites passing across Python server endpoints, GDN cache, memory lifecycle, swap prefetch, and prompt file readers.
  • Sanitizers: AddressSanitizer and UndefinedBehaviorSanitizer verified clean.
  • HTTP Compatibility: Verified with OpenAI Chat Completions, Anthropic Messages, OpenAI Responses API, and Zed Editor.
Assets 2
Loading