Xiaomi-Lens-0

Xiaomi-Lens-0 is a 4B multimodal large language model that learns from pixel-level supervision through a shared visual backbone. Starting from Qwen3.5-4B, it decodes each generated <|mask|> token into a mask with a lightweight decoder, so the same model that answers in text can point at the regions it refers to, in a final answer or in the middle of its reasoning. It improves referring comprehension over Qwen3.5-4B by 5.6 points on average across eight RefCOCO-family splits and improves scores on all eight fine-grained perception benchmarks.

98-second overview. Also available on the project page (full-resolution MP4).

Recorded outputs across the six capability domains: Referring Expression Comprehension, Referring Expression Segmentation, Multi-Instance Grounding, Reasoning Segmentation, Embodied Referring, and Fine-Grained Perception.

• The Xiaomi-Lens-0 training code is now open source, including both training stages, launch scripts, and example data. See Training to get started.
• The Xiaomi-Lens-0 online demo is now live on Hugging Face Spaces.
• The Xiaomi-Lens-0 model weights and the LensBench benchmark dataset are now publicly available on Hugging Face.
• The Xiaomi-Lens-0 Technical Report is now publicly available on arXiv.

1. Referring Expression Comprehension: finds the object a phrase describes and returns its bounding box. First on seven of eight RefCOCO, RefCOCO+, and RefCOCOg splits, 5.6 points above its Qwen3.5-4B backbone on average.
2. Referring Expression Segmentation: returns a pixel mask for the referred object. 81.4% mean cIoU across the eight splits, leading the compared models on every split.
3. Multi-Instance Grounding: lists every instance of a category, each with its own box and mask. 63.9 mean F1 on COCO 2017, highest among the compared models and 0.9 points above GroundingPI.
4. Reasoning Segmentation: resolves functional and implicit requests after a short reasoning trace. Best on all four ReasonSeg scores.
5. Embodied Referring: picks the object a person asks to be handed in cluttered scenes. Best on all four RoboRefIt scores.
6. Fine-Grained Perception: thinks with visual latents, grounding each region it refers to while counting or answering high-resolution questions. Improves over its Qwen3.5-4B backbone on all eight perception benchmarks, including +6.8 points on CountQA.

We also release LensBench, a segmentation benchmark with 4,800 instruction-level samples from 1,200 paired evaluation units, covering General, Aerial, Driving, and Panoramic images. Each domain contains 300 units, each with four instructions crossing Single/Multiple targets with Explicit/Implicit wording.

Each generated <|mask|> token has a contextual state, or visual latent, that represents a target region given the image, instruction, and preceding response. A lightweight decoder combines this state with a single feature grid from the vision encoder before spatial merging to predict a mask. The model needs no separate segmentation encoder or multi-scale feature fusion. Joint language and mask training supervises the shared representation used for both spatial predictions and textual answers.

The same decoder supports thinking with visual latents: <|mask|> tokens placed inside the reasoning trace decode to masks for the regions the trace discusses, such as each instance being counted. Grounded reasoning traces supervise these intermediate regions. Mask decoding is an output branch, so no crops or region features are fed back to the language model.

Training runs in two stages: Stage 1 pretrains the mask decoder, and Stage 2 performs full-parameter supervised fine-tuning. The project page shows the training data mix and the referring expression segmentation, VQA, and LensBench data pipelines.

Referring Expression Segmentation: “second wedge grapefruit from left to right”	Multi-Instance Grounding: “all oranges”
Reasoning Segmentation: “What can be squeezed onto seafood as a tangy flavor enhancer?”	Fine-Grained Perception: “How many climbing stones are there?” → 5

All 24 recorded cases, four per domain, can be replayed on the project page.

All scores are percentages; higher is better. Bold marks the best score and underline the second best.

Recall, precision, and F1 at IoU 0.50 and 0.95; the mIoU group averages each metric over IoU thresholds from 0.50 to 0.95.

Box-grounding F1 on the RefCOCO family is on the project page.

LensBench contains 1,200 human-reviewed paired units and 4,800 instructions across general, aerial, driving, and panoramic imagery (300 units per domain). Each unit combines single/multiple targets with explicit/implicit wording. The two wordings share the same target mask; multiple targets use the union mask and its enclosing box. Targets and masks are fixed before instruction synthesis, followed by automated consistency checks and human review of each complete unit.

GPT-6-Astra leads cIoU on explicit and implicit instructions and in aerial, driving, and panoramic imagery. Qwen3.8-Max leads cIoU in general imagery (74.2). Xiaomi-Lens-0 ranks second on both explicit-instruction metrics and on cIoU in aerial, driving, and panoramic imagery, and leads aerial Prec@0.5 (79.6). Implicit instructions remain challenging: cIoU drops from 71.4 to 54.7, while Prec@0.5 drops from 85.0 to 70.0. The gap is largest for multiple targets in panoramic scenes, where cIoU falls from 71.8 to 48.9 and Prec@0.5 from 81.3 to 48.7.

Marginal results pool single and multiple targets. Each cell reports cIoU / Prec@0.5 (%, higher is better).

git clone https://github.com/SeerRay-Lab/Xiaomi-Lens-0.git
cd Xiaomi-Lens-0
pip install -r requirements.txt   # torch, transformers 5.4, flash-attn
pip install -e .

Inference needs a CUDA GPU (bf16, FlashAttention-2). The commands below load the weights from Hugging Face as SeerRay-Lab/Xiaomi-Lens-0; pass a local directory to use a downloaded copy.

python infer.py \
  --checkpoint SeerRay-Lab/Xiaomi-Lens-0 \
  --image /path/to/image.jpg \
  --text 'Locate all instances in the image that match the description "the cat on the left". Report bbox coordinates and masks.' \
  --output result.jpg --draw-bbox

The script prints the generated text and saves the masks (and, with --draw-bbox, the boxes) overlaid on the image. Useful options:

• --per-mask-output: save one image per predicted mask instead of a single composite.
• --mask-threshold: sigmoid threshold for binarizing masks (default 0.5; 0 draws a soft heatmap).
• --temperature, --top-p: sample instead of decoding greedily.

Each grounded instance in the answer is written as <bbox>[x1, y1, x2, y2]</bbox><mask><|mask|></mask>, with box coordinates normalized to 0–1000. Every <|mask|> token is decoded into a pixel mask at the resolution the vision encoder sees. Masks emitted inside <think> ground the reasoning and are not part of the final answer.

import torch
from PIL import Image
from infer import (
    build_inputs, generate_completion, load_model_and_processor,
    run_forward, split_answer_content, parse_answer_bboxes,
)

device = torch.device("cuda")
model, processor, mask_token_id = load_model_and_processor("SeerRay-Lab/Xiaomi-Lens-0", device)

image = Image.open("/path/to/image.jpg").convert("RGB")
text = ('Locate all instances in the image that match the description '
        '"the cat on the left". Report bbox coordinates and masks.')
inputs, _ = build_inputs(processor, image, text, device)

# Phase 1: generate text with <bbox> and <|mask|> tokens.
completion_ids, completion, num_masks = generate_completion(
    model, processor, inputs, max_new_tokens=2048, mask_token_id=mask_token_id)
print(completion)

# Phase 2: decode each <|mask|> into a mask; keep only those in the final answer.
if num_masks > 0:
    _, num_thought_masks = split_answer_content(completion)
    mask_logits = run_forward(model, inputs, completion_ids, mask_token_id, device)
    masks = mask_logits[:, num_thought_masks:].sigmoid() > 0.5   # (1, K, H, W)
    boxes = parse_answer_bboxes(completion)                       # [[x1, y1, x2, y2], ...] in 0–1000

Serving uses two vLLM processes on the same checkpoint: one generates text, and one runs the mask decoder through vLLM's pooling API. Install the vLLM extra in a separate environment (tested with vLLM 0.19.1):

pip install -e ".[vllm]"

# Terminal 1: text generation on :8000
bash src/xiaomi_lens/integrations/vllm/run_vllm.sh SeerRay-Lab/Xiaomi-Lens-0

# Terminal 2: mask decoding on :8001
bash src/xiaomi_lens/integrations/vllm/run_vllm_mask.sh SeerRay-Lab/Xiaomi-Lens-0

# Terminal 3: query both and save the overlay
python -m xiaomi_lens.integrations.vllm.client \
  --image /path/to/image.jpg --query "the cat on the left" --output result.png

The client sends the image path to the mask server, so it must run on the same machine as the servers. Ports and the served model name can be changed with PORT and SERVED_NAME; if you override MAX_PIXELS, set the same value for both servers.

LensBench is available on Hugging Face Datasets. Download it with:

hf download SeerRay-Lab/LensBench --repo-type dataset --local-dir LensBench

Evaluate with eval/run.py. By default it loads SeerRay-Lab/Xiaomi-Lens-0 and SeerRay-Lab/LensBench from Hugging Face; --model-path and --dataset-path also accept a local checkpoint directory and a local test.parquet (or its directory).

pip install -e ".[eval]"

# Single GPU
python eval/run.py --output-dir outputs/lensbench

# One shard per GPU, merged at the end
python eval/run.py --devices cuda:0 cuda:1 cuda:2 cuda:3 --output-dir outputs/lensbench

# Local copies
python eval/run.py --model-path /path/to/Xiaomi-Lens-0 --dataset-path LensBench --output-dir outputs/lensbench

# vLLM backend (text and mask engines on two GPUs; requires ".[vllm]")
python eval/run.py --backend vllm --vllm-text-device cuda:0 --vllm-mask-device cuda:1 --output-dir outputs/lensbench_vllm

Results are written to metrics.json (Prec@0.5, cIoU and gIoU, overall and per domain, explicitness and cardinality) and per-sample predictions to predictions.jsonl. Add --resume to continue an interrupted run and --limit N for a quick check.

Training starts from Qwen3.5 in two stages. Stage 1 trains the mask decoder and the <|mask|> embedding with mask loss alone, keeping the rest of the model frozen. Stage 2 fine-tunes all parameters with the LM loss plus the mask loss (dice + BCE). The launchers use DeepSpeed (ZeRO-1 for Stage 1, ZeRO-2 for Stage 2) and expect CUDA GPUs.

pip install -e ".[train]"        # transformers 5.4, deepspeed, accelerate
pip install -e ".[train-fast]"   # optional: liger-kernel and Qwen3.5 linear-attention kernels

# Prepare the base model: rename Qwen's box tokens to <bbox>/</bbox> (IDs and weights unchanged)
hf download Qwen/Qwen3.5-4B --local-dir checkpoints/Qwen3.5-4B
python tools/prepare_base_model.py checkpoints/Qwen3.5-4B --in-place

# Optional: synthetic toy data to check the setup end to end
python examples/train/make_toy_data.py --output examples/train/toy

# Stage 1 → checkpoints/xiaomi-lens-stage1
MODEL_PATH=checkpoints/Qwen3.5-4B DATA_CONFIG=examples/train/stage1.yaml \
  bash scripts/train_stage1.sh

# Stage 2 → checkpoints/xiaomi-lens-stage2 (starts from the Stage 1 output)
DATA_CONFIG=examples/train/stage2.yaml bash scripts/train_stage2.sh

Training data is JSONL listed in a YAML manifest. Each <|mask|> in an assistant turn is paired with one COCO RLE mask; examples/train/README.md describes the format. Arguments appended to a launcher override its defaults (python train.py --help lists them all). For multi-node training, run the launcher on every node with NNODES, NODE_RANK, and MASTER_ADDR set. Both launchers resume from the latest checkpoint-* in OUTPUT_DIR with the same world size and ZeRO stage. The output loads with infer.py, eval/run.py, and the vLLM servers.

If you find Xiaomi-Lens-0 useful, please cite:

@misc{xiaomilens0,
  title         = {Xiaomi-Lens-0 Technical Report},
  author        = {Pei Fu and Yike Liu and Yue Sun and Hanmo Wang and
                   Xiuwen Xi and Ruoceng Zhang and Shaojie Zhang and Tingwei Huang and
                   Chao Jiang and Hang Li and Bingrui Liu and Zhenbo Luo and
                   Heng Qu and Changjian Xu and Ming Xu and Jian Luan},
  year          = {2026},
  eprint        = {2610.23279},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  url           = {https://arxiv.org/abs/2610.23279}
}

Xiaomi-Lens-0 is built on Qwen3.5.

We thank InstructSAM for their open-source contributions.

Code is licensed under Apache-2.0.