Repository navigation
relm 0.2.0 — vision
relm 0.2.0 adds vision. A handle loaded with a model's mmproj projector takes image input across generation and embeddings — on the same vendored engine as 0.1.0, with the text-only paths byte-identical.
Install (macOS Apple Silicon + Intel + Linux binaries, no toolchain)
install.packages("relm", repos = c("https://vadale.r-universe.dev", getOption("repos")))Seeing machines
m <- llm(llm_download("qwen2-vl-2b-instruct-q4_k_m"),
projector = llm_download("qwen2-vl-2b-instruct-mmproj-f16"))
llm_generate(m, "What color is the shape in this image?", images = "circle.png")
#> "Blue"
llm_embed(m, "", images = "circle.png") # image and text in one vector spacellm(projector=)loads the vision tower alongside the model.llm_generate(images=)asks about an image.llm_embed(images=)puts images and text in one space.- Registry aliases pin an Apache-2.0 default pair (
qwen2-vl-2b-instruct-q4_k_m+qwen2-vl-2b-instruct-mmproj-f16), SHA256-verified against an immutable revision, fail-closed. vignette("vision", "relm")walks the whole path and is self-contained — it draws its own test images with base graphics, so it renders with or without a local model.
How it is tested
The image encoder's raw output is checked value-for-value against an unpatched upstream llama.cpp reference — bit-exact (max |Δ| = 0.0 over 98,304 values) on both macOS arm64 and Linux x86_64. Each runner builds that reference itself, because a float reference belongs to the machine that produced it: the same pristine upstream build disagrees with its own arm64 self across ISAs by far more than any tolerance worth setting.
What that gate does not cover is stated plainly in DECISIONS.md (D-026 fourth addendum): the encoder path carries none of relm's own patch, so bit-exactness there is the expected result. What supports vision across architectures is the byte-exact generation golden and the token-ids pin, both green on Linux against a reference recorded on arm.
Notes
- Images: JPEG, PNG, BMP. The
projectorfile is engine-trusted input, exactly like the model GGUF — prefer the SHA256-verified registry aliases. - Audio is not supported in this release, though the engine's multimodal library carries it.
- Binaries: macOS arm64 + macOS Intel + Linux. Windows is a later phase — build from source meanwhile (needs Rust + CMake).
Full changes: NEWS.md