kinfer v0.2.6

Added — function-xml tool calls held to their form and their schema

A GGUF model that writes its calls in the XML form (Qwen3.5 and later, and
what is built on them) now writes each <tool_call> outside its thinking under
a grammar made from the request's tools: a well-formed, closed call; a declared
parameter that says integer, number, boolean, enum, object or array reads as
one; nothing after the last call but the end of the turn. Other values, a
file's contents among them, are free as long as they do not close the
parameter early, and so are functions and parameters the request did not
declare, which the caller goes on reporting: a grammar of declared names only
would have turned the 4 calls to a missing bash or shell, in the 19,813 the
LocalKin swarm logged, into calls to something else. Every one of the swarm's
241 skills gets through with values of each kind its schema allows. Tokens are
drawn as before and drawn again under the grammar only when it rules one out,
as llama-server does: 0.1 ms a token, 9–25 ms for one drawn again; making the
grammar costs a request 4 ms with 57 tools. A value may not run into
</function> or </tool_call> either, so a model that leaves a parameter
open is led back to </parameter> rather than writing past its turn.

Played the way the box's 22 swarm seats play it — their prompts and tools,
214 tools for the family seats, a canned answer to each call, up to four
rounds — on the box and the laptop, the grammar on and off:

	calls	invalid, on / off	tokens drawn again	decode, on / off
ornith-1.5:35b, box	78 / 72	6 / 7	0	120 / 112 tok/s
ornith-1.5:35b, laptop	82 / 87	3 / 8	0	138.4 / 138.6
ornith, laptop, 8 at once	80	5	0
qwen3.8-27b, box	107 / 105	0 / 2	2	52.2 / 53.1
Flash-Next, laptop	95 / 108	4 / 10	7	9.5 / 9.8
qwen2.5-0.5b (Hermes), box	8 / 8	0 / 0	not held	311 / 309

What is left invalid with the grammar on is a call to a function the seat
does not have (bash, weather, lean_verify), passed through as before.
What it changed: qwen3.8-27b's and Flash-Next's "hourly": True and False,
read as strings, became booleans; told to break the schema, ornith wrote
"unit": "K", "days": "到月底", "hourly": "要" and a comma-separated
string for an array, and under the grammar celsius, 14, true and a JSON
array. Two runs with the grammar off differ from each other as much as on
from off (11 of 13 cases on fresh servers, greedy), so replies that differ
are not the grammar's doing; the tokens it drew again are. Hermes, DSML and
MLX models are unchanged; KINFER_NO_TOOL_GRAMMAR=1 turns it off.

Added — kinfer launch pi, hermes and openclaw, untested

Written from each agent's documentation (2026-11-07) and not run against kinfer.
Pi and Hermes Agent take their provider only from files, so each is pointed, by
PI_CODING_AGENT_DIR and HERMES_HOME, at a directory of kinfer's under
~/.kinfer/agents holding kinfer as its one provider, the model and its context
window; their own ~/.pi and ~/.hermes are neither read nor changed. Pi picks
the model by a name without colons, which its --model reads as a thinking
level. OpenClaw runs as a background service of its own, so launch openclaw
prints the configuration to merge and runs nothing. launch installs none of
them.

Added — what agents read to fit themselves to a server

• /api/show lists tools, thinking and vision among a model's
  capabilities, from its chat template (now read from the GGUF header) and its
  projector; model_info gives its architecture and trained context; and while
  it is loaded parameters holds num_ctx, the window each conversation has,
  which Hermes sizes its history by.
• /props, as llama-server answers it, with that window, the slots and the
  template's capabilities.
• /v1/models gives the loaded model's context_length.
• /metrics: kinfer_prompt_reused_tokens_total, the prompt tokens the prefix
  pool gave, and kinfer_prefix_entries, the pool's entries now — 0 reads every
  prompt whole, as the box's Flash-Next did for an hour with nothing to say so.

Fixed — -host-experts without -moe-cache loads again

v0.2.5 gives experts in host memory passes of 4096, and with no GPU cache that
pass asks llama.cpp for buffers the GPU may not have: Flash-Next with
-host-experts 36 on the MacBook Pro asked for 36.9 GiB of its 40 and failed to
load, where v0.2.4 served it in passes of 2048. A context that cannot be made
with a pass kinfer chose is now tried with a smaller one, 2048 first, before
the load fails; that one loaded with 4 slots of 32768 and 6.2 GiB to spare.

Fixed — experts placed by hand that leave a little room are not refused

-host-experts 24 -moe-cache 8192 -batch 2048 and -host-experts 36 -moe-cache 20000 on the MacBook Pro left 1.7 and 2.2 GiB after the weights
and were refused ("no room for a context"), while the same flags leaving 0.0
GiB were loaded with a warning, as chosen on 11-06. Experts placed by hand are
now sized like that whatever is left, measured down to what fits, and loaded
with the warning: both answer.

Fixed — a pool lost to the larger pass for experts in host memory

v0.2.5 reads prompts in passes of 4096 when experts are in host memory, and a
prefix pool needs the KV cache unified, whose pass reads every conversation's
cells. On the box, Flash-Next under the service's -slots 4 -prefix auto lost
its pool to that: every turn of an agent's conversation was read whole again,
19–41 s to a first token with 11–21k-token prompts. Before the pool's last
entry gives way — after the pool has shared the slots' cells and the expert
cache has given up what it may, as in v0.2.5 — the pass now goes back to 2048.
The same box with its services running, back to back with the same flags:
v0.2.5 kept no pool and read the 7,813-token second turn in 9.6 s; now one entry
stays and the turn is read in 0.4 s. Where v0.2.5 kept a pool, nothing changes:
Flash-Next with -slots 4 on the MacBook Pro still reads an 11,000-token prompt
at 761–846 tok/s (v0.2.5 804–844), and DeepSeek-V4.1 placed on the box at 222.
Models whose experts are all on the GPU are untouched.

Fixed — an Ollama MLX model served without its chat template

kinfer serves Ollama's MLX models from a directory of links to their blobs,
with the chat template written out from tokenizer_config.json, and stamps the
directory with the manifest. One made while that blob was still being copied
into Ollama's store read no template, was stamped all the same, and stayed so:
qwen3.8:27b-mlx, copied to the box over Thunderbolt, was served with chatml,
which does not know Qwen's tool calls — empty replies, and a call returned as
text. A directory that lacks the template its tokenizer_config.json carries is
now made again; on the box the same model parses its tool calls.

kinfer-v0.2.6-darwin-arm64.tar.gz sha256 93b7ada1c5ebda57e1e5150cc214c8d3ead6cadf05e94275e4c2472093b73e9c

kinfer v0.2.5

Faster — drafting on MLX: one wait a step, five drafts, a newer MLX

Ollama's qwen3.8:27b-mlx on a MacBook Pro (M5 Max), six greedy replies: 62.2 tok/s
on average before, 72.5 now (40–107), Ollama 0.40.15 78. Two changes:

• A step waits on the GPU once. It waited four times: for the drafts, for the
  pass that checks them, for the tokens taken from that pass, and for what was
  taken back after a draft that did not hold — and the GPU sat idle for the 4 ms
  the 64-layer pass took to build. Now the drafts are worked out while the pass is
  built, and the pass is evaluated with them, with what was taken back and with
  the tokens the reply takes: a step went from 48 ms to 43–46, and the six replies
  to 67.5, word for word the same.
• MLX 0.32.5-0f4740d, and five drafts a step. MLX 0.32.4 multiplies a
  quantized matrix by five rows or more at a higher cost per row than by four, so
  three drafts were the most that paid. The MLX after it has kernels for six to
  sixteen rows (ml-explore/mlx#4712): a pass checking five drafts costs little more
  than one checking three, and five drafts give 72.5 against three's 67.4. That
  MLX is not released; the libraries are what MLX's own CI built of commit
  0f4740d, the MLX Ollama 0.40.15 ships, with mlx-c 0.8.0 and a one-line patch
  (patches/mlx-c). kinfer fetches them once, 79 MB, the first time it serves an
  MLX model, into ~/.kinfer/mlx/0.32.5-0f4740d; the 0.32.4 directory can go. On
  the M3 Ultra they are no slower anywhere: Qwen3.5-0.8B alone 117 → 145 tok/s,
  at 64 the same 2,850; Qwen3.8-27B at 4 bits alone 29 → 32, at 16 the same 105.

Measured and left as it is: choosing how many tokens to draft from the time each
number takes and how often each position holds, as Ollama does. It gained
nothing on 0.32.4 and less than a fixed five on the newer MLX, and a reply then
depends on the drafts before it: run twice, the same prompt could give two
replies. With the number fixed it gives one.

Faster — many requests at once, with -prefix -1

With the prefix pool off, each conversation has a stream of the cache to itself,
and llama.cpp reads the sequences of separate streams together only while their
ids run on without a gap. As replies ended they left gaps, and every gap split
the step: Qwen2.5-0.5B on the M3 Ultra took 21 ms a step with 128 slots busy,
then 36, then 60. Now the highest busy slot moves into each one that frees up —
llama.cpp copies its stream, 12 MB for a 1024-token slot of that model — and the
step stays at 21 ms. 128 at once went from 2,467 tok/s to 4,803, 81% of what
llama-batched-bench gets on that machine; 64 from 1,983 to 3,405. The default,
with the pool, is unchanged at 3,029 and 2,328: the pool needs llama.cpp's unified
buffer, in which each conversation's attention reads every conversation's cells
(llama-batched-bench -kvu gets 3,436 at 128). docs/performance.md says when to
turn the pool off.

Faster — a prompt read with experts in host memory, in passes of 4096

Every pass of a prompt reads the experts kept in host memory again, so the pass
those models get unasked is now 4096 tokens, not 2048: half as many passes, half
as many reads. Its compute buffer takes 2–3 GiB from the expert cache. On the
MacBook Pro (M5 Max, the GPU at 40 GiB), a 7,753-token prompt and three replies,
the replies the same word for word:

model	prompt, 2048 → 4096	decode, 2048 → 4096
Qwen3.8-Flash-Next	715 → 917 tok/s	31.5–32.8 → 30.4–31.6
DeepSeek-V4.1	129 → 235	13.8–14.1 → 13.4–13.9
Ornith-1.5-397B	58 → 139	3.04–3.56 → 3.13–3.61

An agent rereads long prompts more than it writes, so the 3% of decode is worth
it; -batch 2048 takes it back. Placement keeps room for the larger pass. 8192
read Ornith's prompt at 328 tok/s but left a 9 GiB cache, which missed 98% of
the time: decode 27% slower, and 18 GiB of swap.

Checked — a mixture of experts on MLX, end to end

Ollama's qwen3.5:35b-a3b-nvfp4 (Qwen3.5's mixture of experts, 21.9 GB) on a
MacBook Pro (M5 Max): 101–105 tok/s alone against Ollama's 123–127, a 7,753-token
prompt read at 3,772 tok/s against 3,989, and four replies at once 188 tok/s in
all against Ollama's 112, which serves one at a time on MLX. One reply of six
was Ollama's word for word; the others were for a few words to a few sentences,
then went their own way where a word was a near tie. The package has no MTP head, so nothing drafts.

kinfer-v0.2.5-darwin-arm64.tar.gz sha256 67f7802bdcb3778d5831991ca6bc2347e64933d91c329c350a2c5db190db2e96

kinfer v0.2.4

A mixture of experts read from the SSD generates sooner: DeepSeek-V4.1 on a MacBook Pro 14.5–18.7 tok/s instead of 11.5–16.6, with the same replies.

curl -fsSL https://raw.githubusercontent.com/LocalKinAI/kinfer/main/scripts/install.sh | sh

• A miss is read straight into its slot. On a Mac the experts' GPU cache is memory the host shares, so a miss is now read from the SSD into it directly, not into host memory and then copied: 5 ms of V4.1's 78 a token on the laptop.
• The GPU kept busy while a layer's misses are read. A GPU left idle most of a millisecond lets its clock down and runs the next layer slower; V4.1's misses did that forty times a token. A small kernel on the graphs' own queue now fills most of the wait, and the GPU's part of a token went from 37.6 ms to 25.5. LLAMA_MOE_FILL=0 turns it off.
• LLAMA_MOE_TIMING=1 (with GGML_METAL_GPU_TIME=1) logs where generation's time goes at each cached layer when the model unloads.

DeepSeek-V4.1, four 256-token replies	v0.2.3	v0.2.4
MacBook Pro (M5 Max, 40 GiB GPU), every layer's experts on the host	12.0 / 16.6 / 11.5 / 11.5 tok/s	15.0 / 18.7 / 14.8 / 14.5 tok/s
M3 Ultra, 31 layers' experts on the host	15.5 / 17.5 / 15.4 / 14.8 tok/s	16.5 / 18.0 / 15.2 / 15.5 tok/s

Models that fit on the GPU are unchanged (Flash-Next on the M3 Ultra: the same replies, md5 for md5). The patched libraries are now darwin_arm64_b11998k7 (patch 0007). How the time was found and what else was tried: docs/big-models.md. CHANGELOG.md

kinfer-v0.2.4-darwin-arm64.tar.gz sha256 872a7c4e9876edf8fa282461779ce5d0bcefc174f89deb161349ef68d9b9c59c

kinfer v0.2.3

MLX models draft with their MTP head, many replies at once go faster, and a load short of memory says which settings did it.

curl -fsSL https://raw.githubusercontent.com/LocalKinAI/kinfer/main/scripts/install.sh | sh

• MTP drafting on MLX. A Qwen3.5 or Qwen3.8 model that keeps its MTP head (Ollama's NVFP4 models, Hugging Face's releases) drafts three tokens a step while a reply writes alone, and one pass checks them. A model under 4 GB of weights does not draft unless -draft N says so, since its steps are too quick for drafts to pay.
• Many replies at once. From eight slots up, each step samples its slots on every core instead of one: Qwen2.5-0.5B at 128 at once went from 2,439 to 2,929 tok/s on an M3 Ultra, and to 3,836 on an M5 Max.
• Prompts where memory is left. The prompt pass grows past Ollama's rule when the context leaves room for it: ornith-1.5:35b read an 11,000-token prompt 21–24% faster on an M5 Max.
• Settings that leave no room. A load that leaves under 256 MiB of the GPU says so, naming the flags set by hand, and a first pass that does run out of memory says the same in its error. A model that will not load is now a 500, not a 404.
• -slots: use 8, 16, 64 or 128, never 17–48 — past sixteen sequences llama.cpp b11998's step time more than doubles.
• MLX also loads the mixtures of experts of Qwen3.5's architecture (Ollama's qwen3.5:35b-a3b-nvfp4, ornith-1.5:122b-nvfp4), checked so far only in parts.

measured	before	v0.2.3
qwen3.8:27b-mlx alone, M5 Max	30 tok/s	38–75 tok/s
Qwen2.5-0.5B, 128 at once, M3 Ultra (temperature 0.7 / 0)	2,439 / 2,597 tok/s	2,929 / 3,207 tok/s
ornith-1.5:35b, an 11,000-token prompt, M5 Max	2,526–2,635 tok/s	3,135–3,206 tok/s

With drafting, five of six MLX replies were the same word for word as without, and the sixth parted at a near tie. The patched libraries are unchanged (darwin_arm64_b11998k6). CHANGELOG.md

kinfer-v0.2.3-darwin-arm64.tar.gz sha256 a193c945ea2916c75bb76fe28572346cf29507487f800239ebba03d25226962e

kinfer v0.2.2

Ollama's MLX models — NVFP4, as Ollama 0.40 pulls them — are listed and served where they lie.

curl -fsSL https://raw.githubusercontent.com/LocalKinAI/kinfer/main/scripts/install.sh | sh

• Ollama's MLX models. Ollama 0.40 keeps an MLX model as a blob per tensor, so kinfer list showed only Ollama's GGUFs. Now it shows qwen3.8:27b-mlx … ollama, MLX too, and serves it. kinfer keeps a directory of symbolic links to the blobs, in ~/Library/Caches/kinfer/ollama-mlx, with the chat template written out. Nothing is copied, nothing of Ollama's is touched, and kinfer rm points at ollama rm.
• NVFP4. The MLX engine reads NVIDIA's NVFP4 as ModelOpt exports it — 4-bit floats, an FP8 scale for each 16, a scale for the whole weight — beside bf16 and MLX 4-bit.

qwen3.8:27b-mlx on a MacBook Pro (M5 Max), the same blobs	Ollama 0.40.14	kinfer
decode	53–99 tok/s, with MTP drafting	30.2–31.0 tok/s
a 7,753-token prompt	794 tok/s	662 tok/s

The replies give the same answers as Ollama's, word for word at first and then in other words, as two engines' rounding makes greedy replies part. kinfer's MLX engine does not draft with the MTP head yet, which is most of the decode difference.

The patched libraries are unchanged (darwin_arm64_b11998k6). CHANGELOG.md

kinfer-v0.2.2-darwin-arm64.tar.gz sha256 f36ad5f1923d51539df9ca3e39536e51001452bddf296882b91f06fa0d57ded4

kinfer v0.2.1

Two models whose names differ only in case are two models again.

curl -fsSL https://raw.githubusercontent.com/LocalKinAI/kinfer/main/scripts/install.sh | sh

• A name means the model of exactly that name. An MLX directory and a GGUF of the same weights are often named but for case: kinfer pull mlx-community/Qwen3.5-0.8B-bf16 and kinfer pull ggml-org/Qwen3.5-0.8B-GGUF:BF16 give Qwen3.5-0.8B-bf16 and Qwen3.5-0.8B-BF16. Names were matched without regard to case, newest first, so with both in the store both names ran the newer one, the other could not be run, and kinfer rm of the older deleted the newer. Now an exact name comes first, and a name but for case only when it is one model.
• A name that is two models but for case, and neither exactly, is refused as ambiguous, as a prefix already was.
• A pull of exactly the name, unfinished, is that model: asked for, it says it has not finished, and rm takes what arrived of it, not the finished model beside it.
• A name that matches one model but for case finds it as before.

The patched libraries are unchanged (darwin_arm64_b11998k6). CHANGELOG.md

kinfer-v0.2.1-darwin-arm64.tar.gz sha256 458d6904b53a8c516d03c395231bf9b2b8ec15b75e2878f065926ab5f4f3110d

kinfer v0.2.0

Added — MLX: a directory of safetensors served in Go, no Python

A model can now be a directory as Hugging Face publishes it: config.json,
tokenizer.json and safetensors. kinfer serves such a directory on Apple's MLX,
and a GGUF on llama.cpp as before. The server makes no difference between them:
the same four dialects, the same chat template read from the model, and the same
thinking kept apart from the answer. The MLX engine is Go over mlx-c, through
purego, as llama.cpp is bound: no cgo and no Python.

• What it runs. Qwen3.5 dense (model_type qwen3_5), at bf16 or MLX 4-bit,
  such as the mlx-community conversions. Checked against mlx-lm layer by layer
  on Qwen3.5-0.8B (bf16 and 4-bit) and Qwen3.8-27B 4-bit: the same hidden states
  at every layer, the same greedy tokens. The tokenizer reads tokenizer.json; it
  matched the tokenizers library on 537 texts, 36,390 tokens.

• Getting one. kinfer pull mlx-community/Qwen3.8-27B-4bit pulls a repo
  with no GGUF and a model_type MLX serves as a directory of the store, named
  for the repo. It takes the weights and the files that describe them, goes on
  where it stopped, and checks the weights against Hugging Face's sha256.
  kinfer list shows it as "kinfer, MLX". It resolves by name or prefix, and
  kinfer rm deletes it whole. kinfer can-run sizes such a repo against the
  machine, and for a safetensors repo MLX does not serve, it now gets as far as
  suggesting GGUF conversions; it used to stop with an error.

• The libraries. MLX's libraries (78 MB) are not inside kinfer. The first
  MLX model fetches them once from this repository's mlx-0.32.4 release into
  ~/.kinfer/mlx/0.32.4, checked against a pinned sha256. They need macOS 26.2.

• Serving. Replies run batched, one forward pass for all. They are sampled
  on the GPU in llama.cpp's order, repeat penalty to temperature, and the next
  step starts before the host reads the last one's tokens. A finished reply's
  row is reused in place. Each row's attention reads its own columns only, so
  one long reply does not make the others read its length. Recent prompts are
  pooled as the GGUF engine pools a model with recurrent state: an agent's next
  turn and its next conversation skip what they share. MLX's buffer cache is
  capped at 4 GB; unbounded, it had held 76 GB.

• Measured on the M3 Ultra. Qwen3.5-0.8B, the same bf16 weights both ways:

  	kinfer on llama.cpp	kinfer on MLX
  1 / 4 / 16 / 64 replies at once	201 / 576 / 1,160 / 1,031 tok/s	217 / 671 / 1,270 / 2,833
  memory	15 GB	8.3 GB

  Qwen3.8-27B, Q4_K_M on llama.cpp against MLX 4-bit:

  	llama.cpp, no drafts	llama.cpp, MTP	MLX
  greedy decode	31.0 tok/s	35.7–51.3	36.0
  a prompt of 11,245 tokens	36.8 s	39.2 s	35.0 s
  1 / 4 / 16 at once	31 / 63 / 130	40 / 87 / 121	35 / 83 / 105

  With a 9,000-token system prompt, a 27B agent's second and third turns, and
  its next conversation, read their prompts in 0.28–0.37 s instead of 28 s.

• Not yet on MLX: a mixture of experts, MTP drafting, pictures, embeddings,
  and architectures besides Qwen3.5. A GGUF of any of them runs on llama.cpp.

Changed — a Qwen model thinks as its template says, as in Ollama

Qwen's templates have the model think by ending the prompt in <think>\n.
Qwen3.8's do it unless told not to; Qwen3.5's only when told to. kinfer renders
templates through llama.cpp's C API, which runs no Jinja, and the opener was
left out. So qwen3.8-27b answered without thinking even when a request asked
for it, and Qwen3.5 on MLX almost never thought.

Requests are now rendered as Ollama renders them, from the model's default:

• a request that says nothing gets what the template does by default;
• one that asks to think gets the opener: think: true or a level, OpenAI's
  reasoning_effort, Anthropic's thinking, the Responses API's
  reasoning.effort;
• think: false gets the empty think block, as before.

On the box, characters of thinking with think absent / true / false:

• qwen3.8-27b: 360 / 360 / 0, where it was 0 / 0 / 0;
• ornith-1.5-35b: 508 / 508 / 0, unchanged (it writes its own <think>);
• Qwen3.5-0.8B on MLX: 0 / 1,474 / 0.

Ollama 0.40.11 defaults qwen3.8 to thinking (medium), so this matches it. A
client of a Qwen3.8 model that sent no think now gets thinking, and with it
longer replies; it should send think: false (or Anthropic's thinking disabled,
or a reasoning effort of none) to answer straight away as before.

Fixed — a kinfer built without cgo died on its first BF16 GGUF

Metal compiles llama.cpp's kernels at the language version of the SDK the
executable says it was linked against, not the one the Mac has. Go's own
linker, which links kinfer when cgo is off (CGO_ENABLED=0, or a Mac without a C
compiler), says macOS 12, and that gets Metal 2.4. Below 3.1 llama.cpp's kernels
leave out their bfloat versions while the GPU still reports bfloat, so the first
BF16 matrix product found no kernel (kernel not found in any metal library)
and the server crashed. The released binaries come from the system linker and
were not affected. The patched libraries (scripts/build-libs.sh, now
darwin_arm64_b11998k6) ask for what a current SDK gets, 4.0 on macOS 26,
however kinfer was linked. On the box, Qwen3.5-0.8B BF16 now gives the same
reply from either build. The release set that scripts/fetch-libs.sh takes is
not patched.

MLX libraries 0.32.5-0f4740d

The MLX libraries kinfer's MLX backend runs on. kinfer downloads this once, the first time it serves an MLX model (a directory of safetensors), checks it against the sha256 it pins, and keeps it in ~/.kinfer/mlx/0.32.5-0f4740d. Nothing here is needed for GGUF models, and nobody needs to download it by hand.

This MLX is not a release of MLX's: it is commit 0f4740d (2026-10-27), as MLX's own CI built it that night (Release build, artifact backend-metal-macOS-ARM64). It has the kernels for quantized matrix products of six to sixteen rows (ml-explore/mlx#4712) that a pass checking MTP drafts is, which 0.32.4 has not; on a MacBook Pro (M5 Max), Ollama's qwen3.8:27b-mlx drafting five tokens a step went from 67.4 to 72.5 tok/s on average. It is the MLX Ollama 0.40.15 ships.

file	from
libmlx.dylib, libjaccl.dylib, mlx.metallib	the macOS 26 wheel of that build, mlx_metal-0.32.5-py3-none-macosx_26_0_arm64.whl
libmlxc.dylib	ml-explore/mlx-c 0.8.0 (a26f2d5), built against it with a macOS 26.2 deployment target and rpath @loader_path, with one line changed: MLX took a sliding window in scaled_dot_product_attention before mlx-c followed, so mlx-c passes -1, none, and its own API is as it was
LICENSE-mlx, LICENSE-mlx-c	both MIT

Needs macOS 26.2 or later on Apple silicon. Built by scripts/mlx-libs.sh.

kinfer-mlx-0.32.5-0f4740d-darwin-arm64.tar.gz sha256 561234c600f4dc81cf13a44f13ca7b1354d6316abd5631cbf3992159191c3d45, 78,835,850 bytes

kinfer v0.1.10

kinfer launch starts Claude Code or Codex on a local model, and kinfer now answers the other requests those agents make, besides the ones for a reply.

curl -fsSL https://raw.githubusercontent.com/LocalKinAI/kinfer/main/scripts/install.sh | sh

kinfer launch claude -model qwen3.8                 # Claude Code
kinfer launch codex -model ornith-1.5:35b           # Codex
kinfer launch -host 192.168.0.21:11590 claude       # a kinfer on another machine; asks which model
kinfer launch claude -model qwen3.8 -- --continue   # the agent's own arguments go after --

• kinfer launch. It starts the agent pointed at kinfer.
  • Where it connects: to the daemon kinfer run uses, started if nothing answers. With -host, it connects to a kinfer on another machine instead.
  • Menus: with no agent or no model given, it asks. It lists only the agents installed, leaves out embedding models, and puts the loaded model first.
  • Before the agent starts:
    • the model is loaded, so the first turn doesn't wait for it;
    • the agent is told how long one conversation can grow, so it summarizes its history before kinfer would have to drop turns;
    • Claude Code gets the model under every name it asks for one, so no background request asks for a claude-haiku the server doesn't have;
    • Codex gets a provider on its command line.
  • Nothing written: neither agent's configuration is changed, so there's nothing to restore afterwards.
• /api/ps reports the context a loaded model gives each conversation (context_length, as Ollama's does) and the most one conversation can have (context_max).
• The other requests agents make:
  • POST /v1/messages/count_tokens counts a request as /v1/messages renders it. On ornith-1.5:35b it gave 288 and 284 for requests whose usage.input_tokens came back 288 and 284.
  • GET /v1/models/{id} answers with both OpenAI's and Anthropic's fields.
  • HEAD /api/hello, Claude Code's first request in every session, now gets an answer. It used to count as a rejected request each time.
  • /v1/models now lists the models borrowed from Ollama's store as well as kinfer's own.

From a Mac mini, against ornith-1.5:35b on an M3 Ultra	Claude Code 2.1.318	Codex 0.155.1
write a word counter and three unittest cases, run them, fix what fails	37 s, model load included; tests pass	15 s; tests pass
requests kinfer refused in a session	none	none

The patched libraries are unchanged (darwin_arm64_b11998k5). CHANGELOG.md

kinfer-v0.1.10-darwin-arm64.tar.gz sha256 690e05449e09cdca907e851216682b407c22afe09a3267731fc351a52f44f0d4

kinfer v0.1.9

A model with experts in host memory keeps its prefix pool when memory is tight, so a conversation's next turn is no longer read whole again.

curl -fsSL https://raw.githubusercontent.com/LocalKinAI/kinfer/main/scripts/install.sh | sh

• The prefix pool stays when memory is tight. A conversation's next turn reuses the one before it through the prefix pool. A pool needs llama.cpp's KV cache unified, and a pass over a unified cache is sized for all of it. On the box, Flash-Next's four slots of 32768 took a pass of 5.8 GiB, against 2.2 for the conversations' own caches. Placement had already given that memory to the expert cache. So when the box's other services held memory, the pool was measured out, and every turn of a conversation was read whole again. Now the expert cache placement chose gives up as much as half of itself before the pool loses an entry.

Flash-Next on an M3 Ultra, -slots 4, with 26 GiB held by another process	v0.1.8	v0.1.9
prefix pool	none	one entry, with 2.5–3.1 GiB less expert cache
turns 2 and 3 of a 7,800-token conversation	12.6–13.1 s	1.0–1.2 s
decode, a 113–116-token reply	24.2–25.1 tok/s	21.3–21.8 tok/s
the three-turn conversation	47 s	26 s

With a 45,000-token agent prompt, that saves about 70 s on each turn.

• A cache set with -moe-cache stays as given.
• Where the pool already fit, nothing changes: on a MacBook Pro, Flash-Next and Ornith-397B load exactly as they did.

The patched libraries are unchanged (darwin_arm64_b11998k5). CHANGELOG.md

kinfer-v0.1.9-darwin-arm64.tar.gz sha256 a0e6b15c5afeea71f1fa8200fdce368dbf6f7a8515f48f4a56efd8bc16f858ba