Skip to content

Release v0.1.0

Choose a tag to compare

@homer6 homer6 released this 16 Nov 07:29
· 7 commits to main since this release
0bcfb7a

Released 2026-11-16. The first release. A C++ library for training and running a model behind one interface, with a native backend per platform. This release has the interface, a deterministic stub engine that runs everywhere, and the MLX backend for Apple Silicon, which loads and runs a real Gemma-2 model with no Python. Because it is the first release, this page lists everything in it.

What it includes

The library: orama::model-runtime

  • model::Engine (include/model/engine.hpp), one interface for every backend:
    • infer(prompt, on_token): a token-streaming decode. The on_token callback receives each token and may return text to add to the context before the next token, so a caller can fold search or tool results into the decode as it runs.
    • train_step(samples): one LoRA-style training step over a few (prompt, target) pairs, returning the loss.
    • name(), backend(), is_stub().
  • model::make_engine(backend) builds the engine for a backend; model::default_backend() picks the platform's native one (MLX on macOS, libtorch elsewhere).
  • model::Backend: MLX, LibTorch, LlamaCpp, CUDA, each with label().
  • The stub engine: deterministic, streaming and steerable, with a training loss that falls step by step. It is what every backend without a real implementation returns, so the library builds and its callers run with no machine-learning toolchain.
  • Errors are values: methods return model::Result<T> (include/model/error.hpp), and no exception crosses the interface.
  • model::kName and model::kVersion (include/model/version.hpp) name the library and its version, 0.1.0.

The MLX backend (-DORAMA_BACKEND_MLX=ON, macOS on Apple Silicon)

  • A real MLX engine on Metal: an on-device LoRA training step that lowers the loss, and inference that still streams and accepts injected context.
  • Gemma-2 in C++ (include/model/gemma.hpp): loads mlx-community 4-bit weights and generates, with a cumulative repetition penalty over the generated tokens and a cycle backstop against greedy loops.
  • A SentencePiece tokenizer (model::Tokenizer) and a safetensors loader (model::Weights), both in C++.
  • gemma-cli <model-dir> [prompt]: loads a Gemma-2 model and generates.
  • gemma-bench: a performance harness, run with sudo, that reports prefill and decode tokens per second, GPU residency, power and frequency, and peak memory. GEMMA_STATS prints the same speeds from any run.
  • gemma-runbook.md: downloading the weights, building, running, checking the output and benchmarking.

Build

  • C++23, CMake 3.28 or later with Ninja, Catch2 fetched with FetchContent.
  • Backends are CMake options, all off by default: ORAMA_BACKEND_MLX, ORAMA_BACKEND_LIBTORCH, ORAMA_BACKEND_LLAMACPP, ORAMA_BACKEND_CUDA.
  • The version lives in project(VERSION) in CMakeLists.txt; a test checks the header says the same, and two more check this page exists and is indexed.

Licence and project files

  • MIT LICENSE and COPYRIGHT, an SPDX-License-Identifier: MIT line in every source file, THIRD_PARTY_LICENSES (generated by tools/third-party-licenses.py), CONTRIBUTING.md, CLA.md and CODE_OF_CONDUCT.md (#5).
  • SECURITY.md: how to report a vulnerability privately (#2).
  • The public tree carries nothing internal (#4).
  • A software bill of materials, sbom.spdx.json and sbom.cdx.json, regenerated by tools/sbom.py.

Changes

Change What
scaffold the C++23 library, CMake, FetchContent and Catch2
engine the Engine interface and the deterministic stub backend
MLX the MLX backend: on-device LoRA training step and inference
Gemma-2 the tokenizer and safetensors loader, then Gemma-2 4-bit generation, its fixes for greedy loops, GEMMA_STATS, gemma-bench, and the runbook
#2 SECURITY.md
#4 the tree prepared for the public export
#5 MIT licence, SPDX headers, third-party notices and contributor documents
this release the version 0.1.0, held by tests, and these notes

What it does not include

  • No libtorch, llama.cpp or CUDA backend yet. Their CMake options exist, but each returns the stub engine. On Linux and Windows, make_engine() gives the stub.
  • No server and no image. Serving a model over an OpenAI-compatible HTTP API, as an image, is #3.
  • No pull request was open at the cut, so nothing was deferred.

Known issues

  • None filed. The test that round-trips text through a real tokenizer model runs only when ORAMA_TOKENIZER_MODEL names a SentencePiece model; without one it passes without checking anything, and it did not run for this release (below).

Verification

Proof Result
Default build (no backend), ctest, macOS arm64 11 tests, all pass
MLX build (-DORAMA_BACKEND_MLX=ON, mlx 0.31.2), ctest, macOS arm64 18 tests, all pass
tools/spdx-headers.py --check passes
tools/sbom.py --check passes, regenerated after the last change

Not run: the real-tokenizer round trip and a Gemma-2 generation (no model weights on the release machine), and any Linux build.