Jev-alternative Laya in GGUF

[monatis]

Hi all,
I just converted Laya, an open-source Jev-alternative system-1 model for typed decisions, to GGUF for GGML execution. I also built a no-dep executable to serve it through Typesafe-compatible API. It also allows stdin/stdout communication with JSON-formatted lines as well as one-shot launch from the command line, and it comes with a built-in web UI for easy testing. You can download the pre-built executable and GGUF files here.

I've converted it using ggmlc, a new project I'm developing as an ML compiler for GGML. It automatically lowers and compiles PyTorch/JAX/Flax/Keras models for GGML execution, without any need for hand-crafting the execution graph in C++. Then the ggmlc runtime can execute the graph baked in the GGUF file itself, just like ONNX. If you want to learn more about it, you can see some initial comparative benchmarks against llama.cpp here

[espetro]

Hi, If you want this without Python: I added native Kev support to my llama.cpp fork, server can run with llama-server -m kev-0.8b.gguf. Then target POST /v1/systemone for typed, calibrated answers (~100 ms, pointer head embedded in the GGUF), it's compatible with TypeSafe's SystemOne API.

Prebuilt binaries for mac/Linux/Windows on the release page.

I'll take some time to get this work ready for upstream llama.cpp since it's AI-assisted work and will need of a thorough review 👍

[ericcurtin]

It's been difficult in recent time to introduce new api's #28540 . Standing upon the shoulder's of giants, llmman brings Jev's /v1/systemone API to any model via ggml:

https://github.com/llmmanorg/llmman
https://llmmanorg.github.io/blog/llmman-now-speaks-system-one/