OmniJev

Read the full v0.8 → v1.1 changelog · v1.1 release and downloads

OmniJev fine-tunes Qwen3.5 vision-language backbones with LoRA, decision heads and an ordinal head. v1.1 covers web and phone operation, robot scenes, video, games, board games, audio spectrograms and text. The current training manifest is still being audited; the earlier 270,000-record figure is not an exact count for this release.

A 4B model that answers decision questions about images, video, phone and computer screens, games and robot scenes. The released demonstrations are offline replays; reliable closed-loop gameplay and robot control have not been established. It reads the picture together with your question and any text context, answers with calibrated probabilities instead of generated text, and can say none of the above.

OmniJev is a System One decision model. You do not prompt it for text; you give it a state (images, video frames, screenshots, a camera feed, optional context) and a set of typed questions, and it returns a probability distribution for every question in one forward pass with 0 generated tokens:

Twelve questions about the same screenshot cost about as much as one. Options may also be regions of the image (box: [x1, y1, x2, y2], coordinates in 0–1000), so "which element / which cell / where is the hazard" are ordinary choice questions. A video is sampled into 16 timestamped frames, which is exactly what the model sees.

Three sizes, one code base (mso.infer), one API, all Apache-2.0, all on Qwen3.5 backbones:

The separate generative SFT baseline is tinnel123/OmniJev-SFT-0.8B. It uses PEFT text generation rather than MSO1.

Every example below works for all three: MSO1("ckpt", "base") picks the right path from the checkpoint. pip install fla-core is optional and turns on the fast linear-attention kernels.

Download all four weight packages and SHA-256 checksums from the GitHub v1.1 release.

7 short videos + a 35-second compilation, 181 states and 527 questions, all with real v1.1 4B inference. Games, phones, web tasks, two-view robot scenes and video understanding. Purple shows model probabilities; green REF shows source annotations.

All 7 demos · Watch/download the compilation · Download the full pack

These are recorded-trajectory replays, not live model-controlled runs. Some inputs may overlap training; they are illustrations, not new benchmark results.

Arcade: Enduro, Skiing and Pong · 22.5s · MP4	Board games: chess and Gomoku · 14.4s · MP4
Phone: weather, timer and drawing tutorial · 26.2s · MP4	Video: action, start frame and completion · 20.5s · MP4

The clips below illustrate tasks and the interface using historical releases. They are not newly generated v1.1 demonstrations or latency measurements.

A 21-step web task, page by page — on every page the model picks the element, the operation, whether this is the last step and how far along the task is.	Four robot tasks at once — four LIBERO-10 episodes at video speed; every frame decides the sub-task, the gripper's next direction, grasp now, holding, first stage done.
Playing a game in real time — paddle direction, next ball cell, bomb about to hit, score. One pass per frame, no search, no planner.	Operating a phone, step by step — which action to perform, how irreversible it is, whether the screen shows an error dialog.
A long-horizon robot episode — 276 frames of a two-stage manipulation task: instruction, sub-task, direction, grasp, holding, first stage done.	Watching a robot, judging every frame — five decisions per frame; the amber tick on the rail is the real grasp.
Pointing on a grid, no boxes drawn by anyone — a 96-cell grid over screenshots and photos; the model picks the cell, zooms in, picks again.	Navigating link by link toward a target page — which link to follow at each hop, a probability on every candidate, stop when arrived.
Watching a camera feed for danger — fire or smoke, what kind of hazard, how urgent, which region. The alarm bar is the model's own probability.	Gesture control — a hand in front of a camera becomes a device command, fired only when the model is confident enough.
Judging what happened in a video — did the event happen, in which frame it starts, is the person still in view, what is being done.	Ten held-out items, one stopwatch — a photo, a phone screen, a chess board, a robot view, a whole video; the clock runs exactly as long as the model took.
Listening by looking — five seconds of sound drawn as a spectrogram and a waveform: which sound it is, whether a person made it, whether it contains a sharp burst, how loud. The clip with the actual audio is next to it in the repository.

Score audit: these are capped task-family aggregates, not complete per-dataset benchmark results. Read the dataset/subset scores and missing coverage. Mario next-action accuracy is 43.52% (193 questions), not the mixed 56.05%; OK-VQA 93.47% uses supplied candidates including the gold answer, not official open-ended VQA scoring. The old safety report covers HaGRID only.

Full per-family metrics and counts · Aggregate source reports · Release manifest

Macro accuracy gives each family equal weight; micro accuracy weights each report by its own question count. Mean family ECE is an average of per-family ECE values, not pooled ECE.

The table reports existing evaluations, not a matched-sample ablation. Base models use candidate-answer probabilities (A1_raw), with at most 1,000 questions per family before dev/test splitting. OmniJev and SFT typically use about 1,500 questions per family. Sampling, option truncation and image inputs differ. The 30-family summary excludes music, for which only base results exist.

The released SFT baseline is 0.8B only. Its 41,951 generations were judged and audited: 97 incorrect explicit answer labels were accepted by the raw LLM judge. Audited accuracy is 48.0894% (equal to the parser), rather than the uncorrected judge's 48.3207%.

v1.1 OmniJev uses numbered multi-image panels. The historical SFT and base evaluations used the first still image only. Multi-image SFT retraining and inference over the newly frozen 272,561-question set are not completed. The 20-step distributed smoke checkpoint is not part of this release.

These are project evaluation families, not uniform official benchmark scores. POPE and LongVideoBench were excluded from training; Mind2Web uses official splits. Other families include custom row-level holdouts; the event-timing family uses videos from the Charades-STA training set. Row-level exclusion does not establish episode-level separation.

The main table contains raw evaluation reports, not a new evaluation of the calibrated serving outputs. Released temperature metadata comes from a separate calibration split. The 4B checkpoint also contains biases.noul=0.0531085661, which moves the yes/no boundary; therefore its raw POPE result must not be restated as a serving-path result.

4B has the highest 30-family macro accuracy, but POPE (84.73% vs 86.42% base) and LVB (56.80% vs 59.02% base) remain lower. The 0.8B and 2B models also have substantial POPE regressions. All rows are retained.

ECE measures the average confidence–accuracy gap across bins; lower is better. It does not guarantee that every individual confidence level is within that gap. Brier score and NLL are also reported in the full results. Generative SFT reports did not retain calibrated answer probabilities, so its ECE is unavailable.

The independent calibration set has 7,767 dev and 17,754 test questions. Temperature scaling preserves argmax accuracy; the additional 4B yes/no bias can change it. Calibration metadata reports temperature-only results.

The previous release measured 1/3/6/12-question requests on one idle A800-SXM4-40GB: 4B 294/292/344/436 ms; 2B 217/220/243/277 ms; 0.8B 216/216/218/236 ms. These are historical measurements, not new v1.1 serving benchmarks. The v1.1 source reports contain per-family latency, but do not form a controlled hardware-matched speed comparison. All Qwen3.5 sizes use the prefix-branch path.

git clone https://github.com/tinnel123666888/OmniJev && cd OmniJev
python -m venv venv && ./venv/bin/pip install -r requirements.txt     # torch, transformers>=5, pillow

hf download tinnel123/OmniJev --revision v1.1 --local-dir ckpt                       # the OmniJev weights
hf download Qwen/Qwen3.5-4B --local-dir base               # the backbone (or symlink a local copy)

An image. Questions are a map id -> question; the answers come back under the same ids.

from mso.infer import MSO1

m = MSO1("ckpt", "base")
answers = m.system_one(
    {"images": ["screen.png"]},
    {"op":   {"type": "choice", "instructions": "Which operation comes next?",
              "criteria": {"click": "tap an element", "type text": "", "scroll": ""}},
     "risk": {"type": "score",  "instructions": "How irreversible is the next action?",
              "levels": ["harmless", "needs care", "irreversible"]},
     "err":  {"type": "noul",   "instructions": "This screen shows an error dialog."}})
# answers["op"]   -> {"choice": "click", "probabilities": {"click": 0.81, ...}, "abstain": 0.02, "confidence": 0.81}
# answers["risk"] -> {"score": "needs care", "probabilities": {...}, "confidence": 0.7}
# answers["err"]  -> {"noul": 0.01}

A video. video_state samples 16 timestamped frames into one picture (ffmpeg and ffprobe must be on PATH); the model then judges the whole clip.

from mso.video import video_state

answers = m.system_one(
    video_state("clip.mp4"),                                  # writes clip.mosaic.jpg next to the clip
    {"happened": {"type": "noul",   "instructions": "The person opens the door."},
     "doing":    {"type": "choice", "instructions": "What is the person doing?",
                  "criteria": {"cooking": "", "cleaning": "", "reading": "", "eating": ""}},
     "progress": {"type": "score",  "instructions": "How much of the action is shown?",
                  "levels": ["none of it", "the beginning", "most of it", "all of it"]}})

Text context. Put it in front of the instructions; the model reads it together with the picture (the served API does the same with state.text).

task = "Task: book a table for two at 7 pm on the restaurant's website."
answers = m.system_one(
    {"images": ["page.png"]},
    {"op":   {"type": "choice", "instructions": task + "\nWhich operation comes next?",
              "criteria": {"click": "tap an element", "type text": "", "select": "", "scroll down": ""}},
     "done": {"type": "noul",   "instructions": task + "\nThe task is finished."}})

Regions. A choice option can be a region instead of a name: "options": [{"key": "a", "region": {"box": [120, 40, 380, 90]}}, ...] (0–1000). mso/templates.py has ready-made question sets for browsers, phones, robots, games and grids (templates.questions("libero", instruction=...)), and mso/infer.py is also a CLI (--ckpt --model --image --questions).

v1.1 uses Qwen3.5 0.8B, 2B and 4B backbones with rank-32 LoRA, decision and ordinal heads, LM features, prefix-branch inference and multi-image panels. This continuation trained for 5,000 / 4,000 / 3,000 steps respectively. Earlier curricula differ across sizes, so these are not equal cumulative training budgets. Training uses probabilistic scoring rules, followed by temperature calibration on a separate holdout.

The 0.8B generative SFT baseline trained for 12,500 steps, learning rate 1e-4 and accumulation 4. It generates answer text, has no OmniJev decision heads and does not use MSO1. Its historical training path reads the first still image only. Training record manifests and cumulative budgets are not aligned enough to call this a strict same-data comparison.

Apache-2.0 (see LICENSE) for the code and the weights; the backbone keeps its own license.

@misc{omnijev2026,
  title  = {OmniJev: an omni-modal System One decision model},
  author = {Xu, Tianrun and Fan, Hongbang and Lin, Jiahao and Zhu, Zilin and Diao, Zhenxin and Guo, Longteng and Liu, Jing},
  note   = {Beijing Zhongguancun Academy; Institute of Automation, Chinese Academy of Sciences; Zevo},
  year   = {2026},
  url    = {https://github.com/tinnel123666888/OmniJev}
}

OmniJev is developed jointly by Beijing Zhongguancun Academy, the Institute of Automation, Chinese Academy of Sciences and Zevo.

Main contributors

• Tianrun Xu (徐添润) · Core Developer
• Hongbang Fan (范红榜)
• Jiahao Lin (林佳豪)
• Zilin Zhu (朱子林)
• Zhenxin Diao (刁镇薪)
• Longteng Guo (郭龙腾) · Project Lead
• Jing Liu (刘静) · Corresponding Author

Contact — academic exchange and collaboration: s-xtr24@bza.edu.cn

A-OKVQA correction / 更正： the old genmcq aggregate contains incorrectly generated yes/no labels and is withdrawn as an understanding score. The saved 4B multiple-choice subset scores 77.07% (750). See the dataset and question-type audit.