HARMONY

Reconstruct and render a complete, editable 3D room from a single reference photo — room shell, furniture, wall-mounted objects and decorations as separate meshes — using VGGT depth/camera estimation, VLM scene reasoning, image-edit inpainting and Hunyuan3D object generation.

The HARMONY300 benchmark's scoring code ships here in evaluation/ — see Evaluation.

VGGT depth/camera + Manhattan estimation recover the room shell; each object is segmented, completed by inpainting and lifted to 3D; placement then proceeds hierarchically — 1. wall-mounted → 2. furniture & ceiling → 3. decorations → 4. lighting — producing the final scene_full.glb.

Three upstream repos are expected as sibling directories. Each has a PLACE_MODEL_HERE.md telling you what to clone and which checkpoints to fetch:

git clone <this-repo> harmony && cd harmony
# then, per the instructions in each placeholder directory:
#   vggt/                      — depth & camera estimation (imported off sys.path)
#   Grounded-Segment-Anything/ — grounding (imported off sys.path)
#   Hunyuan3D-2/               — 3D-generation server

Activate scenegen only when running main.py — the segmentation stages self-activate eagle-embodied / lang-sam via subprocess, and the servers run in vllm-env / hunyuan3d in their own terminals.

Only vllm-env has an exported spec (environment_yml/vllm-env.yml); requirements.txt lists the pip packages scenegen needs (Python 3.10; install the CUDA torch/torchvision wheels matching your driver first). eagle-embodied, lang-sam and hunyuan3d have no exported spec — see MODELS_AND_PATHS.md for the rest and for the first-run checklist.

MODELS_AND_PATHS.md lists every vendored model, conda env and path placeholder this copy of the repo expects you to fill in, and ends with a first-run checklist. Work through it before your first run.

Copy .env.example to .env and fill in whichever keys your chosen backends need (next section). Nothing sources .env automatically — export it yourself in each shell that runs the pipeline or a server:

set -a; source .env; set +a

Three roles are pluggable: the VLM, the image-edit model and the 3D generator. Each can run locally on your own GPU or against a hosted API, and they are chosen independently — the request/response shapes are identical, so no code changes are needed either way.

You need one of these two, not both.

Option A — local Qwen server (no API key, no per-call cost, needs a GPU):

conda activate vllm-env
CUDA_VISIBLE_DEVICES=1 vllm serve \
  Qwen/Qwen3-VL-30B-A3B-Instruct-FP8 \
  --port 8080 \
  --max-model-len 40000 \
  --enforce-eager \
  --served-model-name qwen3 \
  --generation-config vllm \
  --enable-auto-tool-choice \
  --tool-call-parser hermes \
  --gpu-memory-utilization 0.9

export SCENEWEAVE_VLM_BACKEND=qwen      # the default
export VLM_API_URL=http://localhost:8080/v1/chat/completions   # if not on :8080

Option B — a hosted GPT API (no GPU needed for this role):

export SCENEWEAVE_VLM_BACKEND=gpt
export OPENAI_API_KEY=sk-...
export SCENEWEAVE_VLM_MODEL=gpt-5.5                            # optional
export SCENEWEAVE_VLM_URL=https://api.openai.com/v1/chat/completions   # optional

Any OpenAI-compatible chat/completions endpoint works — set SCENEWEAVE_VLM_URL and SCENEWEAVE_VLM_MODEL to point elsewhere. If the key is missing the pipeline says so and falls back to the local Qwen endpoint rather than failing mid-run.

Reasoning models (gpt-5.x, o-series) are detected by model id: they reject an explicit temperature/top_p, and their max_tokens budget covers hidden reasoning as well as the answer, so both are adjusted automatically.

conda activate vllm-env
CUDA_VISIBLE_DEVICES=2 python -m uvicorn Qwen.img_server:app --host 0.0.0.0 --port 8000

or export SCENEWEAVE_IMG_EDIT=gemini with GEMINI_API_KEY.

Hunyuan3D is the only backend for all four object-generation stages. On a Hunyuan3D 404 — the known can't-mesh-this signature, typically a glass or transparent object deadlocking marching cubes — the pipeline re-inpaints that one object as an opaque material and retries once before giving up on it.

Windows, doors and framed art are never 3D-generated: they become flat textured plane meshes via object_placement/wall_mounted/generate_window_plane.py.

conda activate hunyuan3d
cd Hunyuan3D-2
python api_server.py --host 0.0.0.0 --port 8081 --enable_tex

Needed only for the four object-generation stages; you can skip those while iterating (see --skip below) and the rest of the pipeline still runs.

Copy .env.example for the complete list.

Run each local server in its own terminal / tmux pane. main.py runs in the scenegen env and reaches the servers over HTTP on localhost.

conda activate scenegen

# --image is required; output defaults to outputs/<timestamp>/
python main.py --image data/indoor_images/my_room.jpg

# Explicit output directory
python main.py --image data/indoor_images/my_room.jpg --output outputs/my_room

# Resume an existing run (each stage skips work already on disk)
python main.py --image data/indoor_images/my_room.jpg --rerun-dir outputs/my_room

# Skip the slow 3D-generation stages while iterating
python main.py --image data/indoor_images/my_room.jpg \
    --skip object_generation furniture_object_generation \
           decoration_object_generation ceiling_object_generation

# Run only specific stages
python main.py --image data/indoor_images/my_room.jpg \
    --stages floorplan segment_wall_objects inpaint_wall_objects

On start, main.py copies the reference photo into the output directory, so --rerun-dir is self-contained afterwards. A stage that raises is logged with a traceback and the pipeline continues to the next one.

harmony-release/
├── main.py                       — pipeline entry point, stage registry
├── MODELS_AND_PATHS.md           — what you must supply before first run
├── .env.example                  — every environment variable, documented
├── floorplan/                    — photo → room shell
│   ├── pipeline.py                 stage-1 driver
│   ├── agent/                      VLM scene parsing & prompt construction
│   ├── vggt_estimates/             VGGT depth, Manhattan frame, camera alignment
│   ├── wall_line/                  wall mesh, camera placement, rasteriser
│   ├── openings/                   door & window detection
│   ├── geometry_refine.py          depth-guided wall-extent refinement
│   ├── gen_empty_room_ref.py       furniture removal → empty_room_ref.png
│   └── texture_from_reference.py   backproject real textures onto the shell
├── object_placement/
│   ├── segmentation_la_sam2.py     LocateAnything→SAM2 driver (all 3 domains)
│   ├── vlm_backend.py              qwen / gpt55 VLM router
│   ├── wall_mounted/               windows, doors, art, mirrors, sconces
│   ├── furniture/                  sofas, beds, tables, chairs, rugs
│   ├── decorations/                pillows, books, plants, vases
│   ├── ceiling/                    pendants, chandeliers, fans
│   ├── missing_items/              GroundingDINO pass for unplaced items
│   └── assemble_scene_glb.py       merge shell + all objects → scene_full.glb
├── lighting_module/               — VLM light estimation, pyrender & Cycles renders
├── Qwen/                          — image-edit server + backend adapter
├── evaluation/                    — CLIP / photometric / GPT-judge harness
├── scripts/                       — batch runner + standalone debug runners
├── data/                          — input photos (see PLACE_DATA_HERE.md)
├── outputs/                       — per-run output directories
├── environment_yml/               — conda environment specs
├── vggt/                          — VGGT (vendored, required)
├── Grounded-Segment-Anything/     — GroundingDINO + SAM (vendored, required)
└── Hunyuan3D-2/                   — Hunyuan3D (vendored, required)

Each object domain follows the same module layout: segment_* (or the shared LocateAnything driver) → inpaint_* → object_generation_* → material_properties → place_*. The furniture package holds the shared implementations that the other domains import: material_properties.py (estimate_material / apply_to_glb) and the Hunyuan3D client in object_generation.py.

Reconstructions across HARMONY300 — each rendered from its own recovered camera. More on the project page.

evaluation/ is the scoring code for the HARMONY300 benchmark — the same harness that produces the numbers on the HARMONY300 leaderboard (linked from the project page). Get the 300 scenes, their reference photos and per-scene cameras from the dataset on Hugging Face.

It reports two families of metrics:

Appearance scoring is independent of the pipeline and driven by environment variables:

export REF_DIR=/path/to/harmony300/ref
export REGEN_DIR=/path/to/3D-RE-GEN/results
export GEN3DSR_DIR=/path/to/Gen3DSR/out
export HARMONY_ROOTS=/path/to/harmony/outputs
bash evaluation/evaluation.sh

Geometry scoring takes the method roots explicitly, since each baseline stores its meshes differently:

python evaluation/chamfer_eval.py \
    --gt-root   /path/to/front3d_gt/sceneobjgt \
    --harmony-root /path/to/outputs/harmony300 \
    --out       results_geometry.json

Each --<method>-root is optional; a method with no root given is skipped. The scene list is taken from --gt-root, so the same scenes are scored for every method. Alignment is Sim(3)-normalised + ICP, because the methods do not share the ground truth's world frame.

See evaluation/README.md for the per-method directory layouts expected by loaders.py.

This repository is released under the MIT License.

The vendored upstream repos (VGGT, Grounded-Segment-Anything, Hunyuan3D-2) and the model weights they fetch keep their own licenses — check each before any commercial use. HARMONY300's easy tier derives from 3D-FRONT, which is licensed for non-commercial research only; per-scene provenance is recorded in the dataset's attribution.csv.

@article{sun2026harmony,
  title   = {HARMONY: Hierarchical Agentic Reasoning for Monocular Image-to-Scene Synthesis},
  author  = {Sun, Shufan and Wang, Chen and Song, Enxin and Gu, Jiatao and Liu, Lingjie},
  url     = {https://arxiv.org/abs/2609.26793},
  year    = {2026}
}