Laya Vision

Typed, calibrated decisions about an image plus optional text, in one forward pass with no text generation. You give it a picture, some context and a set of questions. It answers each one as a multiple choice (choice), a yes/no probability (noul) or a level on a rubric you write (score), with probabilities you can act on at face value. The same model plays simple games from pixels: the screen is the image and the buttons are the options.

import laya
from PIL import Image

agent = laya.load_vlm("thaitea/laya-vision")
result = agent.predict(
    {"image": Image.open("photo.jpg"), "note": "customer says it arrived broken"},
    {
        "damage":   {"type": "score",  "instructions": "How much damage does the item show?",
                     "criteria": ["none", "cosmetic: scratches or dents", "functional: parts broken or missing", "destroyed"]},
        "category": {"type": "choice", "instructions": "What kind of item is this?",
                     "criteria": ["electronics", "clothing", "furniture", "food", "other"]},
        "outdoors": {"type": "noul",   "instructions": "Was the photo taken outdoors?"},
    },
)
a = result["answers"]
a["damage"]["score"], a["category"]["choice"], a["outdoors"]["noul"]   # expected level 0-3, top option, P(true)

Inputs are cut to fit the checkpoint's token budgets: each option to 48 tokens (less when many options share head_max_len, 256), the instructions to what the options leave, the state text to what max_len leaves after the images. An answer whose question was cut carries a truncated field that says what was dropped; predict(..., strict=True) raises instead.

Laya Vision is an independent fork of Laya that replaces its ModernBERT text encoder with a small vision-language model. Laya's predict(state, questions) API, output schema, RLCD training objective and temperature calibration are unchanged. It is an experimental research project, not affiliated with Convai Innovations, the authors of Laya.

• Try it: thaitea/laya-vision-demo, a Space on free CPU, about 3 s per image. Source in space/.
• Install: pip install -e . plus torchvision, which the image processor needs. ModernVBERT needs transformers >= 5.3.
• Documentation: https://r33drichards.github.io/laya-vision/, built from site-docs/.
• Community: join the Discord for questions, results and discussion.

• Accuracies are calibrated answers on the official validation splits. VQAv2 yes/no is a re-split of the official val set by image, so it is not comparable to published VQAv2 numbers.
• Evidence: the first two rows are backed by committed per-question predictions in results/raw/. python benchmarks/verify_published.py recomputes them and checks them against this table and each checkpoint's metrics JSON (recommended, previous). The rules for adding or changing numbers are in AGENTS.md.
• Latency is the median predict call on an L4, preprocessing included. Raw milliseconds vary by about 50% between L4 hosts; timed on the same GPU, the recommended checkpoint takes 0.83× the time of the previous one.

The recommended checkpoint against the previous one. On the full evaluation suite (59,427 questions over 34 validation sets), it answers 69.1% correctly against 69.3%. It is within a point on 23 sets, ahead on 3 and behind on 8, mostly small ones (VQA-RAD, 62 questions, 80.6% against 88.7%). It is better calibrated (ECE 0.041 against 0.064), 15% smaller, and plays games: 0.35 on the autoresearch games benchmark (0 = random play, 1 = expert) against −0.04. Scorecards: recommended, previous.

score answers mean something only on the first two rows, which were trained on rubric data. On held-out rubric data the previous checkpoint scores 54% over 5 levels on VLFeedback response grading (prior-only baseline 27.5%): Score head results.

The recommended checkpoint on the autoresearch games benchmark (greedy play, one forward pass per move, seeds never used in training; 0 = random play, 1 = expert):

These are from the run's result. A variant that also trains the vision tower plays better (0.54, with 4×4 mazes 0.71 and Snake 0.80), but it loses several points on visual-reasoning sets (its result). To watch a checkpoint play, or run the longer games suite: Play games.

autoresearch/ adapts karpathy/autoresearch to this model, and the recommended checkpoint came out of it. An agent edits one file (experiment.py), the fixed harness trains it for 15 minutes on an H100 and measures it, and a Pareto rule decides keep or discard.

• Four objectives: quality (accuracy over 34 eval sets minus calibration error), games, parameter count, and latency relative to a reference on the same L4. Noise margins come from repeat runs.
• The agent's instructions and what the tags so far found are in autoresearch/program.md, with the lessons carried over from autogo and KataGo.
• Long runs: modal run --detach autoresearch/full_run.py --commit <sha> --name <run> --minutes 120 trains a frontier recipe for longer and measures it the same way. It is resumable, and --attach takes over a run whose orchestrator died.

modal run autoresearch/harness.py --tag <tag>                    # one experiment: the committed experiment.py
python autoresearch/pareto.py show --tsv autoresearch/runs/<tag>/results.tsv

The documentation site, https://r33drichards.github.io/laya-vision/, is built from site-docs/ with MkDocs:

• Install and Tutorials: the demo Space, and a first prediction in Python.
• How-to guides: calibrate on your own data, run jobs on Modal, evaluate a checkpoint, play games.
• Concepts: how it works, the architecture, calibration, the training data.
• Reference: predict() and the answer schema, checkpoints, and every result, from typed answers vs generated JSON to what didn't work.

To build it: nix build .#docs (the site lands in ./result), or pip install mkdocs mkdocs-mermaid2-plugin && mkdocs build --strict. Pull requests run the build, a link check and a headless-browser check (docs-check.yml); pushes to main deploy it to GitHub Pages (deploy-docs.yml).

• Code: Apache 2.0, inherited from Laya. See LICENSE.
• Weights: CC BY-NC-SA 4.0, because the training data includes ScienceQA and CrisisMMD, which carry that license.

Laya Vision is an independent fork of Laya by Convai Innovations, Apache 2.0, and is not affiliated with them. The text decision model, its typed-question API (choice / score / noul), the RLCD training objective and the temperature calibration are theirs. This fork adds image input, the SmolVLM and ModernVBERT backbones, the rubric data, the game work and the autoresearch loop. The design is described in the author's write-up. The original text model and its Colab/Kaggle notebooks live in the upstream repo. ModernVBERT is by its authors under MIT; SmolVLM by Hugging Face under Apache 2.0.