erllama 0.9.0

Added

• erllama_chat:set_observer/1' / clear_observer/0' hook.
  Lets a separate module (typically the server's metrics module)
  observe wall-time of every chat-NIF call (apply / parse) without the runtime taking a compile-time
  dependency on the metrics module. Registered via persistent_term;
  unset = no-op. Drives the new
  `erllama_chat_*_duration_seconds' Prometheus histograms
  on the server side.
• erllama_app:start/2' enables erlang:system_flag(scheduler_wall_time, true)' at boot so callers
  (the server's metrics module among them) can read per-scheduler
  busy ratios for the autoparser dirty-pool monitoring.

Changed

• chat_apply/2' runs upstream's common_chat_templates_apply' once per
  request (prompt + parser). The per-tools params cache and the
  render-only NIF are gone; only the per-model templates ref is
  cached.
• Vendored llama.cpp is b10068, upstream and unmodified (no local
  patches). scripts/vendor_llama.sh <tag> performs bumps and fails
  if any vendored file differs from the tarball. Unused vendor/
  libraries are pruned and LLAMA_OPENSSL is off (no OpenSSL link).
• Hex package manifest now includes c_src/erllama_chat_nif.{cpp,h}
  and c_src/erllama_resources.h (previous tarballs could not build).
• Bump vendored llama.cpp from b9334 to b9585. No API-breaking changes
  on our touchpoints; brings common/chat* bug fixes (LFM2 reasoning,
  tool-parser unification). UPDATE_LLAMA.md refreshed to document
  the common/ + vendor/ sync the prior procedure left implicit.

Removed (BREAKING)

• Streaming API: {erllama_token, _, {tool_call_delta, _}}' and erllama_tool_call_end' messages, plus the matching
  step result variants. The engine no longer classifies tool-call
  bytes mid-stream. Callers using erllama:infer/4' directly buffer tokens and call erllama:chat_parse/3'
  at done for structured extraction. HTTP wire format on the
  server is unchanged.
• Backend `tool_call_end_is_eos/1' callback.

Changed

• erllama:chat_apply/3' becomes chat_apply/2' (drops the
  ToolsHash' parameter). erllama_chat_cache' shrinks to
  caching only the heavy common_chat_templates_init' ref per model; each request invokes common_chat_templates_apply' fresh because
  the synthesized parser is sensitive to tool_choice' and parallel_tool_calls' (now folded into the NIF Inputs map).

Added

• Public erllama:chat_apply/2 + chat_parse/3 that delegate
  to erllama_chat via the model gen_statem so callers can
  build a chat_params_ref' and parse model output without touching the underlying NIF model resource. Backend gains an optional get_model_ref/1' callback; the stub backend returns
  {error, chat_not_supported}'. chat_purge/1' drops cached entries
  on demand for a given model id.

Fixed

• EOS-bounded tool-call flush: don't capture the EOS token's bytes
  before flushing the span. The previous in-span EogFlag path called
  req_tool_call_emit/3 unconditionally; for Granite / Phi-4 the EOS
  is a special token that detokenizes empty under special=false,
  so the buffer stayed clean in practice, but the path was fragile
  for any future model whose EOS detokenises to visible bytes. Now
  the EOS-end branch skips the emit and calls req_tool_call_end/2
  directly, matching the byte-string-end-marker semantics.

Added

• Complete the chat-autoparser NIF wiring. nif_chat_templates_apply
  now translates an Erlang inputs map (messages + tools JSON
  binaries, optional tool_choice) into common_chat_templates_inputs
  via common_chat_msgs_parse_oaicompat / common_chat_tools_parse_oaicompat,
  calls upstream, and returns the synthesized chat_params_ref plus
  the rendered prompt bytes. nif_chat_parse deserialises the
  per-template PEG arena from the cached params string and dispatches
  to common_chat_parse. The parsed common_chat_msg is marshalled
  to #{role, content, reasoning_content, tool_calls}; tool-call
  arguments come back as raw JSON binaries and are decoded to maps
  at the Erlang facade boundary (erllama_chat:parse/3).
  New erllama_chat_SUITE real-model CT (gated on
  LLAMA_TEST_MODEL) covers init / apply / parse round-trip and a
  partial-then-full streaming case.
• NIF wrapper for llama.cpp's common_chat_* autoparser. Vendors
  common/ + vendor/nlohmann + vendor/cpp-httplib from the
  pinned llama.cpp tree, flips LLAMA_BUILD_COMMON=ON, and links
  llama-common into the NIF .so. Three new entry points
  (nif_chat_templates_init/2, nif_chat_templates_apply/2,
  nif_chat_parse/3) run on dirty CPU; two new resources
  (chat_templates_ref, chat_params_ref) wrap
  common_chat_templates_ptr and common_chat_params. New Erlang
  facade erllama_chat (raw NIF shim) and
  erllama_chat_cache (LRU cache keyed on
  {ModelIdBin, ToolsHash}, with purge/1 for model unload). The
  refactored erllama_resources.h exposes the existing
  C resource type pointers + erllama_model_t to C++
  TUs behind an extern "C" guard. templates_init works
  end-to-end; templates_apply and parse ship as
  {error, not_implemented} placeholders (the Erlang-term marshalling
  lands in the follow-up). Dormant capability: the chat / messages /
  responses handlers do not consume this surface yet; Phase 3.C
  wires them up.
• EOS-bounded tool-call end-marker capture. Models configured with
  tool_call_markers => #{start => Bytes, 'end' => <<"$eos">>} opt
  into the new path: when the scheduler is inside an open tool-call
  span and the model samples a token with EogFlag = 1, the
  accumulated tool_call_bytes buffer is flushed via the existing
  erllama_tool_call_end message before the request
  finishes. Previously the buffer was silently dropped in that
  branch (the only emission site was the byte-string end-marker
  match in erllama_model_llama:map_marker/2). The
  byte-string-end families (Mistral </s>, Qwen </tool_call>,
  DeepSeek <｜tool▁call▁end｜>, Llama 3.1 <|eom_id|>) are
  byte-exact unchanged; the new path is opt-in via the sentinel
  binary <<"$eos">> on the end' key. Backend behaviour gains one optional tool_call_end_is_eos/1callback (the scheduler defaults tofalse` for backends that have not been updated).
  Targets the IBM Granite-3.x and Microsoft Phi-4-mini families
  whose wire shape bounds the call at EOS rather than a
  byte-string end marker; the server-side migrations land in
  follow-up PRs.

Changed

• The native tool-call capture splits spans on a repeated start marker. Families like
  Mistral tekken delimit parallel calls by emitting the start marker again
  ([TOOL_CALLS]n1[ARGS]a1[TOOL_CALLS]n2[ARGS]a2</s>) with no per-call end between
  them; the previous single-span capture concatenated every call into one
  erllama_tool_call_end message, which the server parsed as one garbled
  call. apply_step_results/2 now finalises the current span before opening the next
  when a start marker fires while already in-span, so each call yields its own end
  message and reuses the upstream per-call accumulation. Behaviour-preserving for
  families with explicit per-call end markers (qwen-xml, qwen3-coder, dsml): a start
  never fires while in-span because the end fires first. The stub backend gains an
  optional tool_call_script :: [start | body | end_tok] config so tests can drive
  exact decode-step sequences (the Mistral tekken [start, body, body, start, body, body, end_tok] shape and the qwen-style [start, body, end_tok, start, body, end_tok] regression guard run from the same harness).
• The tool-call end marker now carries the source token's eog flag.
  erllama_model_llama:map_marker/2 was discarding EogFlag on the end
  marker, so a model whose end marker IS the eos token (Mistral tekken uses </s> as
  both the per-span end marker and the assistant turn's eos) kept decoding past the
  close and spammed repeated tool calls under the greedy continuation. Backend step
  results emit {tool_call_end, Eog} instead of the bare atom; the scheduler sets
  Req.finishing = true on eog so the turn ends cleanly. Backend type spec updated
  accordingly.
• erllama_model_stub gains an opt-in step_delay_ms :: non_neg_integer()
  test knob that timer:sleeps for the configured number of milliseconds at the top
  of every step/2 call. Lets server-side concurrency tests deterministically keep a
  holder request in-flight while other requests race for the queue (used by the e2e
  suite's chat_busy_returns_429 to fix a pre-existing CI timing flake where the
  holder's stream could finish before the racing requests arrived). Default 0 is a
  true no-op for the cache / integration tests that already use the stub. Invalid
  configs coerce to 0.

Fixed

• Idle sticky-session seq pins are now reclaimed under seq-pool pressure, so the pool no
  longer permanently exhausts. A completed sticky turn keeps its sequence pinned for warm
  continuation, but those pins were never released (only end_session/2 or model stop
  freed them) - so after n_seq_max distinct sessions, every new session got
  {error, seq_capacity} (529) with retries never recovering. Admission now reclaims the
  least-recently-used idle pin (a session_seq entry whose seq has no in-flight
  request) when the pool is full, on both the fresh-admit (admit_normal/2) and queued-
  dispatch (dispatch_pending_admits/2) paths, while never reclaiming an in-flight
  session. seq_capacity now only fires when every seq is genuinely active. A reclaimed
  session re-admits cold (or warm-restores from the tiered cache) on its next turn.
  model_info/1 gains pinned_idle_seqs (reclaimable headroom; available_seqs stays
  ~0 since an admitted request immediately re-pins).
• Native tool-call capture now keeps the body between the markers. The marker scanner
  (erllama_model_llama:map_marker/2) is stateless - it tags only the start/end
  marker tokens - so for a model whose tool-call body is ordinary tokens (e.g.
  Qwen3-Coder's <tool_call><function=NAME><parameter=P>v</parameter></function></tool_call>)
  the body was streamed as content and the captured tool_call_bytes held only the
  start marker, yielding an empty call (name "unknown", arguments {}). The decode
  loop now accumulates any token sampled while a tool-call span is open
  (active_sampler = tool_call_syntax / tool_call_payload) into the tool-call bytes
  instead of the content stream, so the full call reaches the parser. Models that emit
  their call without the marker tokens (e.g. Qwen2.5 qwen-xml JSON) are unaffected -
  they never open a span.

Added

• Scheduler can proactively unload an idle model under sustained memory pressure
  (opt-in scheduler.unload_models_under_pressure => true). After cache eviction runs,
  if it could not free the target and pressure is still high, the scheduler calls the
  configured model_evictor (new erllama_model_evictor behaviour,
  evict_one/0) to unload the least-recently-active idle model. Cache slabs are always
  freed first; at most one model is unloaded per tick; the callback's return is
  validated so a missing or misbehaving evictor degrades to no-op. status/0 reports
  models_unloaded_total / last_model_unloaded. The engine names the evictor module
  via config only - no compile-time dependency on the server app that implements it.

Changed

• Vendored llama.cpp bumped from b9222 to b9334 (ggml 0.12.0 -> 0.13.0). No erllama
  source changes required; the C ABI surfaces the NIF wraps
  (erllama_safe.cpp: model load/free, context, sampler chain, llama_decode,
  llama_memory_seq_*, llama_state_seq_*) are unchanged.
• Cache eviction is now frecency-scored, not pure LRU. Byte-targeted eviction
  (evict_bytes, scheduler memory pressure) drops the lowest-scoring rows first, where
  the score is recency biased forward by the row's hit count with a 6 h half-life decay
  (erllama_cache_meta_srv:eviction_score/3). Pinned static-prefix rows and the
  currently-live session key (set_live_key/1, ds4 protected_sha) sort in a higher
  rank so they survive gc/0 and are evicted by evict_bytes only as a last resort
  when nothing else frees enough. The old one-shot last_used += hits*1s install bias
  is removed (hits now feed the score directly, so they are no longer double-counted).

Added

• Pinned static-prefix (agent_prefix) checkpoints. When a caller supplies a
  verified prefix_checkpoint_len (the end-of-tools token offset), the cold prefill
  writes an agent_prefix KV checkpoint at exactly that boundary - independent of the
  cold_min/cold_max band - and pins it so the hot, shared system+tools prefix is not
  evicted under churn. The pin is bounded to one row per namespace
  (erllama_cache_key:namespace/3), survives restart (recovered from the
  persisted save reason on the disk scan), and is re-applied on a warm resume that
  lands on the boundary. New erllama_cache_meta_srv:pin_row/2, ?POS_PINNED
  row field, ?C_SAVES_AGENT_PREFIX counter (saves_agent_prefix), and KVC save
  reason 6 (mirrors ds4 AGENT_SYSTEM). Cold-prefill segments now carry a per-boundary
  save reason.

Changed

• KV cache is now keyed by the rendered prompt bytes
  (detokenize(tokens)), not the token-id list (ds4-style,
  content-addressed). The same logical prompt now hits across turns
  even when it retokenises (chat-template wrapping, tool rendering,
  generated ids vs re-tokenised assistant text), where the old
  token-keyed cache went cold every turn. The longest-prefix lookup
  (erllama_cache_meta_srv:lookup_longest_text_prefix/2)
  scans stored byte-prefix lengths and resumes from the longest match.
  The uncovered suffix reuses the caller's original tokens when the
  byte boundary lands on a token boundary (token-exact resume - so a
  re-sent prompt still reproduces its reply); only a mid-token boundary
  re-tokenises the byte remainder (checkpoint_tokens ++ tokenize(byte_suffix), identical byte stream, sound). Hits stay
  exact (SHA-256 over the bytes; no fuzzy match). The KVC file format
  version is bumped to v2; old v1 (token-keyed) files are rejected
  on the startup disk scan, so the cache refills under the new scheme.
  erllama:list_cached_prefixes/2 now reports the matched
  byte length.

Added

• Per-context compiled-grammar cache. An identical GBNF (agentic clients like
  Claude Code resend the same tool grammar every turn) is now parsed once and
  cloned per request via llama_sampler_clone instead of re-parsed, which
  dominated infer admission for large tool grammars. The cache is a small
  byte-verified LRU on the context resource (a hash is only a pre-filter; identity
  is confirmed by length + memcmp), freed with the context. New
  erllama_nif:grammar_cache_stats/1 (#{hits, misses}) exposes whether
  the cache is taking effect.