v0.3.1

[0.3.1] - 2026-03-10

Fixed

• Crash on hybrid/recurrent models (src/engine/model.rs, src/engine/mod.rs)

  • Qwen3.5, Mamba, and other hybrid architectures use llama_memory_recurrent instead of
    the standard attention KV cache. Calling llama_memory_seq_cp on those models triggered
    GGML_ASSERT(is_full && "seq_cp() is only supported for full KV buffers") inside
    llama.cpp, terminating the process with SIGABRT on the second request with an identical
    prompt.
  • Added Model::supports_seq_copy() backed by llama_memory_can_shift(): returns true
    only for full KV cache backends (standard attention-only transformers).
  • InferenceEngine::new() stores the result as supports_prefix_cache and logs it at
    startup.
  • do_prefill now guards llama_memory_seq_cp with a can_shift check as a second safety
    net.
  • Prefix caching is automatically disabled for incompatible models; all other features
    (stop sequences, metrics, streaming usage) remain fully functional.

• CUDA build (build.rs, Cargo.toml)

  • Removed the optional cudarc dependency (only used to query GPU memory, but its
    build.rs requires a CUDA-version feature flag that caused --features cuda to fail with
    a compile error). Replaced with a nvidia-smi subprocess call — no extra dependencies.
  • build.rs now links ggml-cuda, libcuda (driver API), libcudart, libcublas, and
    libcublasLt, searching both /cuda/lib64 and /cuda/targets/x86_64-linux/lib to
    support different CUDA installation layouts.

• Prefix-cache boundary token position (src/engine/model.rs)

  • do_prefill was copying positions 0..skip_prefix_tokens via seq_cp and then
    submitting the last prompt token at the wrong position (context_len instead of
    skip_prefix_tokens - 1). Changed to copy 0..skip_prefix_tokens-1 and always
    re-submit the boundary token in the batch at the correct position, ensuring valid
    positional encodings and correct logits.

---

[0.3.0] - 2026-03-10

Added

• PageTable — explicit logical→physical block mapping (src/kv_cache/mod.rs)

  • Replaced the flat kv_block_ids: Vec<BlockId> field in InferenceRequest with a named
    PageTable struct. The struct encapsulates the entries vector
    (logical_block_index → physical_block_id) and exposes block_ids(), len(),
    is_empty(), clear(), and extend().
  • Added ref_count: Vec<AtomicUsize> to KVCacheManager (one entry per physical block).
    allocate sets ref_count = 1; free_blocks decrements and only returns the block to the
    free list when the count reaches zero.
  • retain_block(id) — increments ref_count (used when a block is shared for prefix caching).
  • copy_on_write(id) -> Option<BlockId> — allocates a new exclusive block and decrements the
    shared one's ref_count; foundational for future true memory-sharing CoW.

• Prefix caching — skip re-prefill for identical prompts (src/scheduler/mod.rs, src/engine/mod.rs, src/engine/model.rs)

  • Scheduler now embeds a PrefixCache: HashMap<u64, PrefixCacheEntry> keyed by
    hash(prompt_tokens). Max capacity = max_batch_size / 4 entries.
  • When a request finishes (EOS, Length, or StopSequence), InferenceEngine calls
    Scheduler::try_insert_prefix which atomically transfers the request's kv_seq_id and
    page_table blocks into the cache (the KV data in llama.cpp is preserved — clear_sequence
    is skipped for cached entries).
  • On the next admission of a request with the same prompt hash, schedule_step detects the
    hit, transfers the cached blocks to the new request's PageTable, allocates only the
    generation blocks, sets skip_prefix_tokens and prefix_seq_id on the request.
  • do_prefill calls llama_memory_seq_cp(mem, prefix_seq_id, new_seq_id, 0, skip_tokens)
    inside the blocking task before building the batch, then submits only
    prompt_tokens[skip_prefix_tokens..] starting at the correct absolute position.
    After prefill, the engine clears the now-redundant prefix sequence and returns its ID to
    the pool via Scheduler::return_prefix_seq_id.
  • New Model trait method: copy_sequence_range(src, dst, token_count), backed by
    llama_memory_seq_cp; no-op in the stub.
  • Counters prefix_hits / prefix_misses (atomic) on Scheduler; exposed on InferenceEngine.

• Stop sequences (src/scheduler/batch.rs, src/engine/mod.rs, src/api/types.rs)

  • SamplingParams gains stop: Option<Vec<String>>.
  • ChatCompletionRequest.stop accepts both a JSON string and an array (OpenAI spec). Uses a
    custom deserialize_stop Serde helper.
  • StopReason::StopSequence variant added.
  • handle_logits now runs a rolling-buffer stop-sequence check (last 2 × max_stop_len chars)
    per request. The stop string is not emitted in the output; only the prefix before the
    match is sent to the client (OpenAI behaviour). Detection works across token boundaries.
  • Output filtering (<think> suppression, special token stripping) and stop sequence detection
    now share a single lock acquisition on per_request_state to prevent deadlocks.

• Prometheus /metrics endpoint (src/metrics.rs, src/api/routes.rs, src/main.rs)

  • New GET /metrics route returning Prometheus text exposition format (version 0.0.4).
  • Metrics registered at startup via Metrics::new():
    • ferrum_requests_total{finish_reason} — counter
    • ferrum_tokens_generated_total — counter
    • ferrum_request_latency_seconds — histogram (10 buckets: 0.05 s … 60 s)
    • ferrum_kv_cache_usage_ratio — gauge
    • ferrum_queue_depth — gauge
    • ferrum_active_requests — gauge
    • ferrum_prefix_cache_hits_total — counter
    • ferrum_prefix_cache_misses_total — counter
  • Dependency added: prometheus = "0.14".
  • Gauges and counter deltas are refreshed on every engine scheduling step.

• Streaming usage in the final chunk (src/api/routes.rs, src/api/types.rs)

  • The last SSE chunk (the one that carries finish_reason) now includes a usage object
    with prompt_tokens, completion_tokens, and total_tokens, matching the OpenAI
    streaming spec. Intermediate chunks omit the field (skip_serializing_if = "Option::is_none").

Changed

• InferenceRequest fields: kv_block_ids: Vec<BlockId> → page_table: PageTable;
  new fields skip_prefix_tokens: usize, prefix_seq_id: Option<i32>,
  submitted_at: Instant (for latency metrics).
• InferenceEngine::new now accepts metrics: Option<Arc<Metrics>>.
• OutputFilterState renamed to PerRequestState; its text_buffer field drives stop
  sequence detection.
• PLAN.md updated: Phase 1 marked completed with v0.1.0/v0.2.0 summaries; Phase 2
  progress tracked.