Embodied.cpp 🤖

Embodied.cpp is an inference runtime for embodied AI models: Vision-Language-Action (VLA) models and World-Action Models (WAMs) for robotic perception and control. It runs these models efficiently on heterogeneous hardware (CPU / CUDA GPU / NPU) using GGUF weights, and ships with ready-to-use servers and evaluation clients.

---

Table of Contents

• Embodied.cpp 🤖
  • Table of Contents
  • 1. 🧭 Current Support and Roadmap
    • 1.1 Model Support Roadmap
    • 1.2 Runtime Roadmap
  • 2. 🚀 Quick Start
    • 2.1 Clone the Repo
    • 2.2 Get GGUF Weights
    • 2.3 Install System Dependencies
    • 2.4 Build by Model and Backend
    • 2.5 Start a Server
    • 2.6 Evaluate in Simulation
  • 3. 🧪 Evaluate in Simulation
    • 3.1 LIBERO
    • 3.2 RoboTwin
  • 4. 🔧 Convert Your Own Model
  • 5. 🗂️ Project Structure
  • 6. 📄 Citation
  • 7. ⚖️ License
  • 8. 🙏 Acknowledgements

---

1. 🧭 Current Support and Roadmap

1.1 Model Support Roadmap

The table below summarizes the embodied AI model families that Embodied.cpp already supports and the ones we plan to support next. For a more detailed taxonomy and architectural discussion, please refer to our technical report.

Family	Subtype	Support ✅	Planned 🚧
VLA	AR-Token VLA	-	OpenVLA
VLA	VLM-Backboned VLA	pi0.5, HY-VLA	Octo
VLA	Hierarchical VLA	-	Hi Robot, GeneralVLA
VLA	Asynchronous VLA	-	GR00T N1, Fast-in-Slow
WAM	Predict-then-Act WAM	-	UniPi
WAM	Unified AR-Modeling WAM	LingBot-VA	WorldVLA
WAM	Shared-Backbone WAM	-	DreamZero, FastWAM, Cosmos Policy, UWM
WAM	Latent-space WAM	-	Being-H0.7

1.2 Runtime Roadmap

• This part of the project is still under active construction 🚧
• A more modular and maintainable runtime architecture for Embodied.cpp
• Additional inference optimizations, such as real-time chunking and VLA caching
• More hardware backends, including Metal on macOS

---

2. 🚀 Quick Start

2.1 Clone the Repo

git clone <repo-url> && cd embodied.cpp
./patches/init_third_party.sh

2.2 Get GGUF Weights

Pre-converted GGUF releases for Embodied.cpp are available on Hugging Face:

• https://huggingface.co/SEU-PAISys/Embodied.cpp

The repository currently hosts GGUF artifacts prepared for the current Embodied.cpp runtime, including:

• pi0.5: main policy GGUF plus multimodal projector GGUF
• HY-VLA-0.5: combined VLA GGUF for RoboTwin and related runtime paths
• LingBot-VA: transformer GGUF and companion artifacts used by the LingBot path

Recommended local layout:

checkpoints/
  pi05/
    pi05.gguf
    pi05-mmproj.gguf
  Hy-Embodied-0.5-VLA-RoboTwin/
    Hy-Embodied-0.5-VLA-RoboTwin_bf16.gguf
    Hy-Embodied-0.5-VLA-RoboTwin_q4_K.gguf
  lingbot_va/
    lingbot_transformer.gguf
    ...

You can also convert upstream checkpoints yourself with the scripts in scripts/, but for most users the Hugging Face GGUF releases are the fastest way to get started.

2.3 Install System Dependencies

Install the required system packages for your platform before building.

Linux: Make sure cmake, protobuf, zeromq, and cppzmq are available from your package manager before building.

macOS (Apple Silicon, CPU-only verified for pi0.5):

brew install cmake protobuf zeromq cppzmq

2.4 Build by Model and Backend

Model switches default to OFF. Enable only the runtimes you need.

pi0.5, CPU-only:

cmake -S . -B build \
  -DCMAKE_BUILD_TYPE=Release \
  -DMODEL_BUILD_VLA_PI05=ON
cmake --build build --target vla-pi05-server -j$(nproc)

pi0.5 on macOS / Apple Silicon, CPU-only:

cmake -S . -B build \
  -DCMAKE_BUILD_TYPE=Release \
  -DMODEL_BUILD_VLA_PI05=ON
cmake --build build --target vla-pi05-server -j$(sysctl -n hw.logicalcpu)

HY-VLA, CPU-only:

cmake -S . -B build \
  -DCMAKE_BUILD_TYPE=Release \
  -DMODEL_BUILD_VLA_HY_VLA=ON
cmake --build build --target vla-hy-vla-server -j$(nproc)

LingBot-VA, CPU-only:

cmake -S . -B build \
  -DCMAKE_BUILD_TYPE=Release \
  -DMODEL_BUILD_WAM_LINGBOT_VA=ON
cmake --build build --target wam-lingbot-server -j$(nproc)

HY-VLA + LingBot-VA, CUDA GPU:

cmake -S . -B build \
  -DCMAKE_BUILD_TYPE=Release \
  -DMODEL_BUILD_VLA_HY_VLA=ON \
  -DMODEL_BUILD_WAM_LINGBOT_VA=ON \
  -DGGML_CUDA=ON \
  -DCMAKE_CUDA_COMPILER=/usr/local/cuda/bin/nvcc \
  -DCMAKE_CUDA_ARCHITECTURES=<your-arch> \
  -DProtobuf_PROTOC_EXECUTABLE=/usr/bin/protoc
cmake --build build --target vla-hy-vla-server wam-lingbot-server -j$(nproc)

If you want a single build with all currently supported runtimes enabled:

cmake -S . -B build \
  -DCMAKE_BUILD_TYPE=Release \
  -DMODEL_BUILD_VLA_PI05=ON \
  -DMODEL_BUILD_VLA_HY_VLA=ON \
  -DMODEL_BUILD_WAM_LINGBOT_VA=ON
cmake --build build --target vla-pi05-server vla-hy-vla-server wam-lingbot-server -j$(nproc)

2.5 Start a Server

# VLA server (pi0.5)
./build/vla-pi05-server \
  checkpoints/pi05/pi05-mmproj.gguf \
  checkpoints/pi05/pi05.gguf

# VLA server (HY-VLA)
./build/vla-hy-vla-server \
  checkpoints/Hy-Embodied-0.5-VLA-RoboTwin/Hy-Embodied-0.5-VLA-RoboTwin_bf16.gguf

# LingBot world-action server (bind to 5555 to match the client example below)
./build/wam-lingbot-server \
  --bind tcp://*:5555 \
  checkpoints/lingbot_va/lingbot_transformer.gguf

Each executable is generated only when its corresponding MODEL_BUILD_* switch is enabled at configure time.

2.6 Evaluate in Simulation

pi0.5 on LIBERO:

# Start the pi0.5 server in another shell first.
./build/vla-pi05-server \
  checkpoints/pi05/pi05-mmproj.gguf \
  checkpoints/pi05/pi05.gguf

# Install the LIBERO runtime once
bash eval/sim/libero/setup_libero.sh

# Run one LIBERO smoke-test episode
eval/sim/libero/libero_uv/.venv/bin/python eval/client/run_sim_client_direct.py \
  --arch pi05 \
  --libero-suite object \
  --task-id 0 \
  --n-episodes 1 \
  --max-steps 80 \
  --seed 42 \
  --tokenizer lerobot/pi05_libero \
  --vla-addr tcp://127.0.0.1:5555

HY-VLA on RoboTwin:

# Build with MODEL_BUILD_VLA_HY_VLA=ON first, and use a HY-VLA GGUF from the
# released Embodied.cpp checkpoint set.

# Install the RoboTwin runtime once
bash eval/sim/robotwin/setup_robotwin.sh

# Run one RoboTwin episode
GGML_CUDA_DISABLE_GRAPHS=1 \
eval/sim/robotwin/robotwin_uv/.venv/bin/python \
  eval/client/run_robotwin_native_hy_vla.py \
  --model checkpoints/Hy-Embodied-0.5-VLA-RoboTwin/Hy-Embodied-0.5-VLA-RoboTwin_bf16.gguf \
  --task-name place_empty_cup \
  --episodes 1

LingBot-VA on LIBERO:

# Install the LIBERO runtime once
bash eval/sim/libero/setup_libero.sh

# Run a test episode
eval/sim/libero/libero_uv/.venv/bin/python eval/client/run_sim_client_direct.py \
  --arch lingbot_va \
  --libero-suite object \
  --task-id 0 \
  --n-episodes 1 \
  --tokenizer /path/to/lingbot-va-tokenizer \
  --vla-addr tcp://localhost:5555

---

3. 🧪 Evaluate in Simulation

3.1 LIBERO

LIBERO tests robotic manipulation skills on four task suites: spatial, object, goal, and 10. A fifth suite long (90 tasks) is also available.

--libero-suite spatial  → libero_spatial
--libero-suite object   → libero_object
--libero-suite goal     → libero_goal
--libero-suite 10       → libero_10
--libero-suite long     → libero_90

Use --task-id 0..9 (or 0..89 for long) to select an individual task.

The direct simulation client currently supports:

• --arch pi05
• --arch lingbot_va

Example pi0.5 smoke test:

eval/sim/libero/libero_uv/.venv/bin/python eval/client/run_sim_client_direct.py \
  --arch pi05 \
  --libero-suite object \
  --task-id 0 \
  --n-episodes 1 \
  --max-steps 80 \
  --tokenizer lerobot/pi05_libero \
  --vla-addr tcp://127.0.0.1:5555 \
  --output-dir outputs/pi05_libero_smoke

3.2 RoboTwin

RoboTwin is a dual-arm robot benchmark with real-world-style manipulation tasks. Run HY-VLA natively in C++:

bash eval/sim/robotwin/setup_robotwin.sh   # one-time setup

GGML_CUDA_DISABLE_GRAPHS=1 \
eval/sim/robotwin/robotwin_uv/.venv/bin/python \
  eval/client/run_robotwin_native_hy_vla.py \
  --model <path-to-hy-vla.gguf> \
  --task-name place_empty_cup \
  --episodes 1

See eval/sim/robotwin/README.md for detailed setup modes and troubleshooting.

4. 🔧 Convert Your Own Model

GGUF conversion scripts are in scripts/:

Script	Converts
convert_pi05_to_gguf.py	pi0.5 model weights
convert_pi05_mmproj_to_gguf.py	pi0.5 multimodal projector
convert_hy_vla_to_gguf.py	HY-VLA combined vision+action
convert_lingbot_va_to_gguf.py	LingBot-VA transformer + companion GGUFs

Quantization helpers:

Script	Quantizes
quantize_hy_vla_gguf.py	HY-VLA models
quantize_lingbot_wan_gguf.py	LingBot-VA models

If you do not need a custom conversion, prefer the prebuilt GGUF releases at:

• https://huggingface.co/SEU-PAISys/Embodied.cpp

5. 🗂️ Project Structure

What lives where, in plain language:

Directory	What it contains
models/	C++ model implementations (pi0.5, HY-VLA, LingBot-VA)
runtime/	Model registry, architecture detection, shared utilities
adapter/	I/O boundary — translates sensor/simulator data into typed inputs the models understand
serving/	Server code (ZeroMQ/Protobuf) for VLA and LingBot APIs
kernels/	Custom CUDA kernels (used when building with GPU support)
scripts/	GGUF conversion, quantization, and evaluation helpers
patches/	Third-party code patches applied during setup
eval/	Evaluation clients and simulation setups (LIBERO, RoboTwin)

6. 📄 Citation

If you find Embodied.cpp useful in your research, please consider citing:

@article{xu2026embodiedcpp,
  title={Embodied.cpp: A Portable Inference Runtime of Embodied AI Models on Heterogeneous Robots},
  author={Xu, Ling and Han, Chuyu and Li, Borui and Wu, Hao and Jiang, Shiqi and Cao, Ting and Li, Chuanyou and Zhong, Sheng and Wang, Shuai},
  journal={arXiv preprint arXiv:2607.02501},
  year={2026},
  doi={10.48550/arXiv.2607.02501},
  url={https://arxiv.org/abs/2607.02501}
}

7. ⚖️ License

This project is released under the Apache License 2.0. Third-party dependencies, model checkpoints, datasets, and upstream reference implementations are distributed under their own licenses.

8. 🙏 Acknowledgements

Supported models:

• pi0.5 / OpenPI
• HY-VLA
• LingBot-VA

Foundational projects this build depends on:

• llama.cpp (LLM inference engine)
• vla.cpp (unified VLA runtime)
• LIBERO (manipulation benchmark)
• RoboTwin (dual-arm robot benchmark)