Official implementation of Xiaomi-SpatialEmbodied, the released codebase for the paper XEmbodied: A Foundation Model with Enhanced Geometric and Physical Cues for Large-Scale Embodied Environments (arXiv 2604.18484).

The model is designed for cloud-side autonomous-driving and robotics VQA in data closed-loop systems. It improves domain semantics and 3D understanding while preserving general VLM capabilities. The implementation follows the paper's two complementary enhancement paths: a geometric adapter for intrinsic spatial representations and an embodied adapter for compact physical cues.

Key Contributions:

• Present Xiaomi-SpatialEmbodied, a cloud-side embodied closed-loop VQA generalist fusing intrinsic geometric representations with physical cue interaction.
• Propose 3DA and EIEA-powered implicit physical cue alignment for adaptive geometric priors injection and efficient physical-augmented reasoning.
• Develop a progressive domain curriculum for robust adaptation with reduced forgetting, validated on 18 benchmarks to demonstrate consistent embodied understanding gains.

Xiaomi-SpatialEmbodied uses Qwen3-VL-30B-A3B-Instruct as its multimodal language backbone and adds trainable spatial and embodied interfaces. Images are processed by the native Qwen vision tower and a VGGT 3D foundation model; the resulting features are aligned before language reasoning, so spatial evidence participates in the model's internal representation rather than being appended as text or an auxiliary depth map.

• 3D Adapter (3DA): VGGT extracts multi-layer geometric features from the same camera image. VGGTEmbeddingMerger resamples the VGGT patch grid to Qwen's image-token grid, applies token merging, normalization, and MLP projection, then injects the result through a cross-attention residual (image tokens query VGGT tokens). This preserves the original sequence length while adding depth, layout, and relative-position cues.
• Efficient Image-Embodied Adapter (EIEA): Outputs from embodied tools such as occupancy, 3D boxes, and map cues are distilled into a compact set of TOR embeddings. These embeddings are inserted at designated positions in the MLLM context, providing physical affordance priors without exposing raw tool traces to the language model. This repository ships the inference consumer and precomputed caches.
• Progressive domain curriculum: Training proceeds from general multimodal capability to spatial/3D supervision and then embodied driving and manipulation data. The staged recipe reduces catastrophic forgetting and yields consistent gains across the 18 benchmarks reported in the paper.

The released main checkpoint is the 3DA variant under xiaomi_spatialembodied/3DA/my_qwen3_vggt_xattnv2/; the connector variants in xiaomi_spatialembodied/3DA/variants/ are provided for ablation studies.

Architecture of Xiaomi-SpatialEmbodied.

Performance across spatial and 3D understanding, semantic reasoning, and embodied affordance benchmarks.

• 2026/09/14: Xiaomi-SpatialEmbodied code officially released
• 2026/04/20: Xiaomi-SpatialEmbodied paper available on arXiv

The best results among the listed models are bolded.

The best results among the listed models are bolded.

The best results among the listed models are bolded.

• Release model code
• Release model weights
• Release inference code
• Release training code
• Release evaluation code (tools/eval_tools/; the Table 3 embodied benchmarks that have no released scorer are marked in the table above)
• Release dataset download

The two experimental connector variants under xiaomi_spatialembodied/3DA/variants/ are ablation studies and have no released weights.

BaseModel-open/
├── xiaomi_spatialembodied/
│   ├── 3DA/                                 # 3D Adapter (see xiaomi_spatialembodied/3DA/README.md)
│   │   ├── my_qwen3_vggt_xattnv2/           # ★ MAIN: Qwen3-VL-MoE-VGGT (3DA geometric connector)
│   │   │   ├── modeling_qwen3_vl_moe_vggt.py  # Core geometric-enhanced model definition
│   │   │   ├── configuration_qwen3_vl_moe.py
│   │   │   ├── modular_qwen3_vl_moe.py
│   │   │   ├── plugin/
│   │   │   │   ├── qwen3_vl_moe_vggt_register.py  # MS-Swift custom model registration
│   │   │   │   └── save_qwen3_vl_moe_vggt.py      # Merge VGGT weights into fused checkpoint
│   │   │   ├── scripts/                     # 4-stage SFT + inference script family
│   │   │   └── vggt/                        # VGGT backbone code
│   │   └── variants/                        # Experimental connector variants (not released)
│   │       ├── my_qwen3_moe_vl_vggt_xattn_mlp/    # Configurable fusion + deepstack connectors
│   │       └── my_qwen3_moe_vl_vggt_qformer/      # Q-Former main connector + deepstack MLP
│   ├── EIEA/                                # Inference side only; see xiaomi_spatialembodied/EIEA/README.md
│   │   ├── README.md                        # Scope, quick start, known limitations
│   │   ├── infer_with_EIEA_3D.py            # Loads the shipped TOR cache and runs inference
│   │   ├── models/
│   │   │   ├── qwen3_vl_vggt_register.py    # Registration with TOR physical embedding auto-injection
│   │   │   ├── qwen3_vl_moe_vggt.py         # Model definition with physical-geometric dual enhancement
│   │   │   └── ...
│   │   ├── tor_embeds_cache_fb.pt           # Pre-cached TOR embeddings (loaded by qwen3_vl_vggt_register.py)
│   │   └── tor_embeds_cache.pt              # Second, different TOR cache (see note below)
│   └── infer_demo.py                        # Basic VLM inference entry
├── demo/                                    # Synthetic sample image for the quick start
├── model_tools/                             # VGGT <-> Qwen grid-alignment debug tooling
├── requirements.txt / .gitignore
└── tools/                                   # Data processing & evaluation toolkit (see tools/README_TOOLS.md)
    ├── convert_data_tools/                 # Dataset format conversion scripts
    ├── parse_io_tools/                     # Raw dataset parsing (Parquet/Arrow/PKL)
    ├── infer_tools/                        # Inference helpers and output post-processing
    │   └── infer_demo_show.py              # Visual inference demo script
    ├── eval_tools/                         # Benchmark evaluation and score aggregation
    ├── vis/                                # Result visualization
    ├── data_statistics/                    # Data statistics, filtering, and sampling
    ├── build_model/                        # Model parameter counting and checkpoint utilities
    ├── reward_tools/                       # Reward model utilities for GRPO/RL training
    ├── trainer_tools/                      # Training debug and launch scripts
    └── planningQA/                         # Planning QA data preparation

The 4-stage training scripts live in xiaomi_spatialembodied/3DA/my_qwen3_vggt_xattnv2/scripts/ (see xiaomi_spatialembodied/3DA/README.md for what each one does).

Documentation map:

Xiaomi-SpatialEmbodied is a unified foundation model framework for autonomous driving and embodied intelligence, where the geometric enhancement backbone (3DA) and physical cue injection paradigm (EIEA) work in deep synergy. Both are built on Qwen3-VL-30B-A3B-Instruct:

The 3D Adapter (3DA) is a geometric connector that injects representations from a 3D foundation model (VGGT) into the MLLM, aligning 3D structure with language reasoning so the VLM can understand 3D geometry without relying on external depth models. Unlike approaches that merely append depth or point clouds as auxiliary inputs, 3DA fuses the VGGT features directly into the native visual token stream.

Key components:

• VGGTSpatialEncoder: Loads the VGGT aggregator to extract multi-layer spatial features from images. Unnecessary heads (camera/point/depth/track) are removed to reduce memory consumption.
• VGGTEmbeddingMerger: Aligns and fuses 3D spatial tokens with native 2D visual tokens through interpolation, token merging, RMSNorm + MLP projection, and Cross-Attention, injecting 3D geometric cues into the vision-language model.

Registered with MS-Swift for one-stop fine-tuning and inference, serving as the foundational carrier of Xiaomi-SpatialEmbodied's geometric perception capability.

Note on token bookkeeping. The released main model does not add any new tokens to the sequence: the connector resamples the VGGT patch grid onto Qwen3's image-token grid and adds a cross-attention residual back into the existing image embeddings. Only the experimental Q-Former variant (under variants/, not released) introduces dedicated 3D tokens.

The two streams are resampled onto a common grid, so how the VGGT input is framed matters. The released checkpoints were trained with the VGGT stream letterboxed to a white 518×518 square while the Qwen stream keeps the image's aspect ratio, and the connector resamples with bilinear interpolation. For non-square images the two grids therefore agree only near the image centre. Both settings are configurable and default to the released behaviour:

vggt_connector_config = {
    "letterbox": True,          # or export VGGT_LETTERBOX=0
    "interp_mode": "bilinear",  # or export VGGT_INTERP_MODE=nearest-exact
}

Changing either one invalidates weights trained with the other, so only do it when retraining. See xiaomi_spatialembodied/3DA/README.md for details.

xiaomi_spatialembodied/3DA/ contains the released main model plus two experimental connector variants, kept for ablation comparison:

Only one variant can be mounted per swift run via --custom_register_path. See xiaomi_spatialembodied/3DA/README.md for the full layout, per-variant scripts, and VGGT path resolution (VGGT_REPO_DIR).

While prior work leverages embodied-specific tools (e.g., occupancy grids, 3D boxes, HD map cues) to explicitly guide VQA reasoning, such explicit tool invocation suffers from low efficiency and poor alignment between tool outputs and language reasoning processes. To address these limitations, EIEA distills raw tool outputs into compact token summaries via TOR (Textual Object Rationale) embeddings, which are then seamlessly reinserted into the MLLM context via masked_scatter, circumventing the interpretability burden on LLMs.

EIEA shares the model's underlying structure with 3DA, together forming Xiaomi-SpatialEmbodied's complete embodied perception capability.

Scope of this release: the repository ships a pre-computed TOR embedding cache (xiaomi_spatialembodied/EIEA/tor_embeds_cache*.pt) together with the inference code that loads and injects it. The EIEA pipeline that produces such a cache is not open-sourced at this time.

conda create -n xiaomi_spatialembodied python=3.10
conda activate xiaomi_spatialembodied

# Install PyTorch
pip install torch>=2.4.0 torchvision>=0.19.0

pip install ms-swift
pip install qwen_vl_utils==0.0.14
pip install trl -U
pip install deepspeed
pip install transformers>=4.57.1
pip install decord

⚠️ transformers must be ≥ 4.57. Qwen3-VL (including the MoE variant) was introduced in transformers 4.57, and the model code imports transformers.models.qwen3_vl_moe.modeling_qwen3_vl_moe. The MS-Swift registration files declare the same constraint (requires=["transformers>=4.57", ...]).

deepspeed is required by every script under xiaomi_spatialembodied/3DA/*/scripts/ (they all pass --deepspeed zero2|zero3), and decord is needed to decode video samples.

vLLM is optional and only used for the evaluation tooling. It ships its own torch build, so installing it here would silently override the pins above — use a separate environment.

See requirements.txt for the full pinned list.

pip install peft==0.14.0
pip install aenum sentencepiece protobuf

Before running any inference script, configure the paths to your local model checkpoints. The repository uses relative paths by default, so you only need to update the specified placeholder paths.

1. Download the base VLM: Download Qwen3-VL-30B-A3B-Instruct or let MS-Swift auto-download it.
2. Download VGGT-1B and export the two environment variables the code reads at load time:

   export VGGT_REPO_DIR=/path/to/xiaomi_spatialembodied/3DA/my_qwen3_vggt_xattnv2  # parent of the vggt/ package
   export VGGT_DEVICE=cuda:0     # default is cpu, which is unusable for a 30B model
   export VGGT_DTYPE=bfloat16    # default is float32

3. Build a fused checkpoint containing the VGGT weights (required — the model code never loads VGGT weights on its own):

   python xiaomi_spatialembodied/3DA/my_qwen3_vggt_xattnv2/plugin/save_qwen3_vl_moe_vggt.py \
     --base_model_dir  /path/to/Qwen3-VL-30B-A3B-Instruct \
     --vggt_ckpt_path  /path/to/VGGT-1B/model.pt \
     --vggt_repo_dir   /path/to/xiaomi_spatialembodied/3DA/my_qwen3_vggt_xattnv2 \
     --output_dir      xiaomi_spatialembodied/3DA/my_qwen3_vggt_xattnv2/base_ckpt

   Alternatively, point CUSTOM_MODEL_CFG["model_path"] in xiaomi_spatialembodied/infer_demo.py at the checkpoint you downloaded from Kaggle — that one is already fused.
4. EIEA physical embedding: The repository includes pre-cached TOR embedding files — no extra pretrained model downloads required.
5. Training: the 4-stage recipe is encoded in xiaomi_spatialembodied/3DA/my_qwen3_vggt_xattnv2/scripts/sft_ddp_s{1..4}*.sh; each stage's --model points at the previous stage's merged checkpoint.

Use xiaomi_spatialembodied/infer_demo.py to run Xiaomi-SpatialEmbodied's geometric enhancement backbone via MS-Swift. The script ships with a synthetic sample image (demo/sample.jpg) so it runs without any dataset download:

# From the repository root
export VGGT_REPO_DIR=$PWD/xiaomi_spatialembodied/3DA/my_qwen3_vggt_xattnv2   # parent of the vggt/ package
export VGGT_DEVICE=cuda:0
export VGGT_DTYPE=bfloat16

python xiaomi_spatialembodied/infer_demo.py

Script configuration:

# Enable Xiaomi-SpatialEmbodied geometric enhancement model
USE_CUSTOM_MODEL = True

# Custom model config (use Kaggle-downloaded trained weights, or a locally fused base_ckpt)
CUSTOM_MODEL_CFG = {
    "model_path": "Path to the *fused* checkpoint (VGGT weights already merged in)",
    "model_type": "qwen3_vl_moe_vggt",
    "register_file": "<repo>/xiaomi_spatialembodied/3DA/my_qwen3_vggt_xattnv2/plugin/qwen3_vl_moe_vggt_register.py",
}

# Native model config
NATIVE_MODEL_CFG = {
    "model_path": "Qwen/Qwen3-VL-30B-A3B-Instruct",
    "model_type": "qwen3_vl_moe",
}

# Inference parameters
INFER_CFG = {
    "max_tokens": 512,
    "temperature": 0,
    "user_prompt": "Your prompt here",
    "image_paths": ["<image_path>"]
}

Use xiaomi_spatialembodied/EIEA/infer_with_EIEA_3D.py to load the shipped TOR cache and run inference with physical cue injection:

cd xiaomi_spatialembodied/EIEA
python infer_with_EIEA_3D.py

This script shows how to load the TOR embedding cache shipped with the repository and inject it, combining the physical branch with the VGGT geometric branch for dual-cue inference. No extra pretrained models need to be downloaded.

⚠️ The repository provides one pre-computed TOR embedding cache and the inference code that loads and injects it; the EIEA pipeline that generates such a cache is not open-sourced at this time. The demo therefore injects a fixed soft prompt rather than deriving physical priors per input image.

See tools/README_TOOLS.md for detailed documentation of each tool, including functionality descriptions and usage examples.

Kangan Qian^*, ChuChu Xie^*, Yang Zhong^✉,*, Jingrui Pang, Siwen Jiao, Sicong Jiang, Zilin Huang, Yunlong Wang, Kun Jiang^†, Mengmeng Yang, Hao Ye^✉,†, Guanghao Zhang, Hangjun Ye, Guang Chen, Long Chen, Diange Yang^†

Affiliations: Automotive and Robotics, Xiaomi Corporation   Tsinghua University   National University of Singapore   McGill University   University of Wisconsin-Madison

✉ Project Leader   * Equal contribution   † Corresponding author

If this project is useful in your work, we'd appreciate a citation:

@misc{qian2026xembodiedfoundationmodelenhanced,
      title={XEmbodied: A Foundation Model with Enhanced Geometric and Physical Cues for Large-Scale Embodied Environments},
      author={Kangan Qian and ChuChu Xie and Yang Zhong and Jingrui Pang and Siwen Jiao and Sicong Jiang and Zilin Huang and Yunlong Wang and Kun Jiang and Mengmeng Yang and Hao Ye and Guanghao Zhang and Hangjun Ye and Guang Chen and Long Chen and Diange Yang},
      year={2026},
      eprint={2604.18484},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2604.18484},
}