Skip to content

relm 0.2.0 — vision

Choose a tag to compare

@Vadale Vadale released this 15 Jul 21:32
· 128 commits to main since this release
b8bd7de

relm 0.2.0 adds vision. A handle loaded with a model's mmproj projector takes image input across generation and embeddings — on the same vendored engine as 0.1.0, with the text-only paths byte-identical.

Install (macOS Apple Silicon + Intel + Linux binaries, no toolchain)

install.packages("relm", repos = c("https://vadale.r-universe.dev", getOption("repos")))

Seeing machines

m <- llm(llm_download("qwen2-vl-2b-instruct-q4_k_m"),
         projector = llm_download("qwen2-vl-2b-instruct-mmproj-f16"))

llm_generate(m, "What color is the shape in this image?", images = "circle.png")
#> "Blue"

llm_embed(m, "", images = "circle.png")   # image and text in one vector space
  • llm(projector=) loads the vision tower alongside the model.
  • llm_generate(images=) asks about an image.
  • llm_embed(images=) puts images and text in one space.
  • Registry aliases pin an Apache-2.0 default pair (qwen2-vl-2b-instruct-q4_k_m + qwen2-vl-2b-instruct-mmproj-f16), SHA256-verified against an immutable revision, fail-closed.
  • vignette("vision", "relm") walks the whole path and is self-contained — it draws its own test images with base graphics, so it renders with or without a local model.

How it is tested

The image encoder's raw output is checked value-for-value against an unpatched upstream llama.cpp reference — bit-exact (max |Δ| = 0.0 over 98,304 values) on both macOS arm64 and Linux x86_64. Each runner builds that reference itself, because a float reference belongs to the machine that produced it: the same pristine upstream build disagrees with its own arm64 self across ISAs by far more than any tolerance worth setting.

What that gate does not cover is stated plainly in DECISIONS.md (D-026 fourth addendum): the encoder path carries none of relm's own patch, so bit-exactness there is the expected result. What supports vision across architectures is the byte-exact generation golden and the token-ids pin, both green on Linux against a reference recorded on arm.

Notes

  • Images: JPEG, PNG, BMP. The projector file is engine-trusted input, exactly like the model GGUF — prefer the SHA256-verified registry aliases.
  • Audio is not supported in this release, though the engine's multimodal library carries it.
  • Binaries: macOS arm64 + macOS Intel + Linux. Windows is a later phase — build from source meanwhile (needs Rust + CMake).

Full changes: NEWS.md