Zero-WAM:
In-Context World-Action Modeling from Human Videos for Open-Ended Task Generalization

Jiaming Zhou   Qihang Zhang^*   Gangwei Xu   Cunxin Fan   Yujie Zhao   Ruilin Wang   Yiming Luo   Shuai Yang   Xing Zhu   Yujun Shen   Junwei Liang^†   Yinghao Xu^†
Robbyant  ·  HKUST (GZ)  ·  HKUST
^*Project Lead    ^†Corresponding Authors

Zero-WAM targets zero-shot cross-task robotic manipulation, where a policy must execute tasks that were never practiced during training using only deployment-time context. It brings in-context learning to robotics by treating human demonstration videos as visual task specifications, enabling a causal video-action policy to predict future robot observations and executable actions from either language instructions or human video prompts.

• Paper release: arXiv:2608.26103
• Code: expected before Sep 15, 2026
• Model: expected before Sep 15, 2026
• Data: expected before Sep 15, 2026

• We formulate zero-shot robotic task generalization as in-context world-action modeling, where one causal policy supports both language and human videos as task instructions.
• We introduce a scalable HumanGen pipeline that converts task-sampled robot trajectories into semantically matched human video instructions, yielding 74.2K human-robot ICL pairs over 8.6K tasks.
• We propose Zero-WAM with an in-context future chunk prediction objective to reduce shortcut learning and strengthen the use of human video prompts.
• We demonstrate zero-shot cross-task generalization in RoboTwin 2.0 and real-world unseen task configurations without collecting corresponding robot data or updating model parameters.

Zero-WAM builds its training data around task diversity rather than raw trajectory count. Task-diverse VA re-samples five public robotic video-action datasets at the task level, yielding more than 6K manipulation tasks and about 400K robot trajectories per training epoch. HumanGen complements this robot-domain corpus with 74.2K human-robot ICL pairs over 8.6K tasks, spanning external, in-house, simulation, and real-world sources.

The in-context human video generation pipeline starts from task-sampled robot trajectories that retain executable actions. A VLM parses task semantics and object-state changes, an image editor converts the first robot frame into a human observation, and a video generation model synthesizes the corresponding human manipulation video. Each generated video is filtered for semantic preservation and physical plausibility before being paired back with the original robot trajectory as an ICL sample.

Zero-WAM achieves 46.95% average zero-shot success on seven unseen RoboTwin 2.0 tasks, outperforming LingBot-VA by 29.50 percentage points.

If you find Zero-WAM useful, please cite:

@misc{zhou2026zerowam,
  title = {Zero-WAM: In-Context World-Action Modeling from Human Videos for Open-Ended Task Generalization},
  author = {Zhou, Jiaming and Zhang, Qihang and Xu, Gangwei and Fan, Cunxin and Zhao, Yujie and Wang, Ruilin and Luo, Yiming and Yang, Shuai and Zhu, Xing and Shen, Yujun and Liang, Junwei and Xu, Yinghao},
  year = {2026},
  eprint = {2608.26103},
  archivePrefix = {arXiv},
  primaryClass = {cs.RO},
  url = {https://arxiv.org/abs/2608.26103}
}