PanoWorld: A Generative Spatial World Model for Consistent Whole-House Panorama Synthesis

Jinrang Jia, Zhenjia Li, Yijiang Hu, Yifeng Shi

Ke Holdings Inc.

PanoWorld is a generative spatial world model for consistent whole-house panorama synthesis. Given a floorplan and a style reference, it autoregressively generates node-based 360-degree panoramas that align with practical VR-tour navigation while preserving cross-view geometry and material consistency across an entire house.

This repository currently releases the PanoWorld-LRM inference code, together with model checkpoints and evaluation data links. More components of the full PanoWorld pipeline will be released progressively.

Overview

• Whole-house synthesis is formulated as autoregressive generation over discrete panorama viewpoints, matching real VR-tour navigation.
• A floorplan-derived 3D shell provides global structural guidance for multi-room layout consistency.
• A dynamic 3DGS cache serves as renderable spatial memory, preserving cross-node geometry and material identity.
• PanoWorld-LRM reconstructs metric-scale multi-room geometry from panoramic observations for high-quality whole-house rendering and evaluation.

News

• 2026-05-19: Paper released and project page launched.
• 2026-05-25: Open-sourced the PanoWorld-LRM inference code, checkpoints (including 1024x512 and 2048x1024 model weights), and evaluation data (50 RealSee3D scenes).
• Coming Soon: PanoWorld 2D generator inference code and checkpoints.
• Coming Soon: Full PanoWorld pipeline, visualization, and evaluation code.
• Coming Soon: Private scene data for evaluating PanoWorld panorama synthesis.
• Coming Soon: PanoWorld-LRM training code.
• Coming Soon: PanoWorld 2D generator training code.

Inference

Quick Start

PanoWorld-LRM

1. Install dependencies:

pip install -r requirements.txt

The released inference package is tested with Python 3.10.18, PyTorch 2.3.1, TorchVision 0.18.1, and CUDA 12.1.

2. Download the prepared RealSee3D inference and evaluation data (Download):

3. Check the selected config and update data.root_data_dir, data.data_path, inference.ckpt_path, and inference.out_dir if needed.

4. Launch inference with one of the provided scripts:

bash infer_1024_512.sh

or

bash infer_2048_1024.sh

You can also run inference directly with:

python inference.py --config configs/inference_1024_512.yaml

5. If you would like to run inference on your own data, please refer to the dataset format description (Here):

You may reorganize your own data into the same format. Inference only depends on the panoramic image panoImage_1600.jpg, the camera extrinsics extrinsics.txt, and the viewpoint-to-room grouping defined in map.json. Organize your data as follows:

<your_data_root>
  <scene_name1>
    map.json
    viewpoints
      <view_name1>
        panoImage_1600.jpg   # panorama image, w:h = 2:1; the resolution is not strictly limited to 1600x800
        extrinsics.txt       # 4x4 camera extrinsic matrix (c2w) for this viewpoint
      <view_name2>
        panoImage_1600.jpg   # panorama image, w:h = 2:1
        extrinsics.txt       # 4x4 camera extrinsic matrix (c2w) for this viewpoint
      <view_name3>
        panoImage_1600.jpg   # panorama image, w:h = 2:1
        extrinsics.txt       # 4x4 camera extrinsic matrix (c2w) for this viewpoint
      <view_name4>
        panoImage_1600.jpg   # panorama image, w:h = 2:1
        extrinsics.txt       # 4x4 camera extrinsic matrix (c2w) for this viewpoint
      ...
  <scene_name2>
  <scene_name3>
  ...

Create a TXT file listing the scenes to be processed in the same format as realsee3D_eval_8views.txt, then set config.data.root_data_dir and config.data.data_path accordingly.

6. The inference results will be saved in inference.out_dir. The output_ply directory can be directly visualized with SIBR_Viewer:

./SIBR_gaussianViewer_app -m /Path/to/output_ply

You may also use other viewers such as SuperSplat.

Inference GPU Memory Usage

	1024x512	2048x1024
8-views	27507MiB	108369MiB
12-views	40285MiB	OOM

_{Tested on NVIDIA H200. The paper uses 1024x512 for experiments and metric computation.}

PanoWorld 2D Generator

Coming Soon

PanoWorld

Coming Soon

Released Files

• inference.py: main inference entrypoint
• model.py, transformer.py, dpt_head.py, prope_custom.py: model definition
• dataset.py, utils.py, metric_utils.py: dataset loading and evaluation helpers
• configs/: released inference configs for 1024x512 and 2048x1024
• data_realsee3D/: released RealSee3D evaluation file lists

Model Checkpoints

Component	Resolution	Link	Notes
PanoWorld-LRM	1024x512	Checkpoint	Released
PanoWorld-LRM	2048x1024	Checkpoint	Released
PanoWorld 2D Generator	Coming Soon	Coming Soon	Coming Soon

Data

Split	Dataset	Usage	Link	Notes
Training	3D Front	Train LRM and 2D generator	Download	Data processing scripts: Coming Soon
Training	RealSee3D	Train LRM and 2D generator	Download	Data processing scripts: Coming Soon
Training	Private 2D panoramas	2D generator only	-	Private
Evaluation	RealSee3D	Evaluate LRM	Download	Released, including 50 RealSee3D scenes
Evaluation	Private scene data	Evaluate PanoWorld panorama synthesis	Coming Soon	Coming Soon

Citation

If you find this project useful, please cite:

@misc{jia2026panoworldgenerativespatialworld,
      title={PanoWorld: A Generative Spatial World Model for Consistent Whole-House Panorama Synthesis},
      author={Jinrang Jia and Zhenjia Li and Yijiang Hu and Yifeng Shi},
      year={2026},
      eprint={2605.17916},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2605.17916},
}

License

This project is released under the Apache 2.0 License. See LICENSE for details.

Third-party code included in this repository may retain its original license notices. For example, prope_custom.py preserves the upstream MIT license notice from its original authors.

Acknowledgements

We would like to thank Gynjn/MVP, QwenLM/Qwen-Image, realsee-developer/RealSee3D, and 3D Front for their inspiring open-source contributions.