llamacpp-jev — TypeSafe Jev's /v1/systemone (typed questions → probabilities) over an unmodified llama-server

[NakliTechie]

A tweet claimed any local model could answer TypeSafe Jev's structured-decision API with no engine changes. I built and verified it against llama.cpp — no fork, no patch — using only primitives llama-server already exposes: /apply-template, n_probs logprobs, GBNF grammar, and prompt-cache checkpoints.

You POST a batch of typed questions (yes/no · single-choice · ordinal-score) about a conversation or an image, and get a normalized probability distribution per question back — from any GGUF, locally, no API key.

Verified (Qwen3.5-2B / 4B-Q8_0):

• Accuracy scales cleanly on the public Open-Jev benchmark (444 items, 12 task families): 0.459 → 0.552 (2B → 4B).
• On in-domain routing, the 4B's probabilities are essentially calibrated (ECE 0.054); temperature scaling would close the rest.
• Text + vision: ~526 ms per fresh 448×448 image + 4 questions on an M4 Pro, ~236 ms on an L4 (uncontended).

Repo (MIT), design, benchmark and an honest competitor comparison: https://github.com/NakliTechie/llamacpp-jev

Honest limits: 2–64 single-token-label choices, raw-softmax probabilities (not trained-calibrated — 4B is ~calibrated in-domain only), small models collapse at 64-way, and it's a local-first server (front it with a gateway before untrusted exposure).

It leans entirely on llama-server's existing grammar + n_probs, so it's really a thin packaging of what the server already does. I'd be curious whether a native /v1/systemone-style typed-decision endpoint in tools/server would be of interest — happy to open a separate Ideas thread if so.

[cfournel]

would change my life :D

[samuel-koob]

I just got this integrated to my existing llamacpp server on docker, works well. The only changes I did to the code were to accept a model name (cause of server mode) and creating a new section in models.ini for qwen3.5 that looks like this:

[Qwen3.5-4B-jev]
load-on-startup = true
parallel = 4
model = /models/Qwen3.5-4B/Qwen3.5-4B-Q8_0.gguf
mmproj = /models/Qwen3.5-4B/mmproj-F16.gguf
ctx-size = 32768
reasoning-budget = 0
ctx-checkpoints = 32
checkpoint-min-step = 0
cache-ram = 0

So essentially I have a llamacpp-jev container that accesses the llamacpp server container for the model and does its magic on top of it. No llamacpp build needed or anything. Literally just wrapped the repo with a dockerfile

[ericcurtin]

It's been difficult in recent time to introduce new api's #28540 . Standing upon the shoulder's of giants, llmman brings Jev's /v1/systemone API to any model via ggml:

https://github.com/llmmanorg/llmman
https://llmmanorg.github.io/blog/llmman-now-speaks-system-one/