Describe a room. Compile it. Step inside.
Holodeck is a VR-first prompt-to-world system for Meta Quest. A user describes an indoor environment from a Unity creation room; a staged language-model pipeline plans the scene, retrieves reviewed assets, authors the intended layout, and hands it to a deterministic compiler. The portal opens only after the room has passed physical checks and has a safe spawn.
Current scope: one indoor room per generation, built from approved Unity assets. Holodeck does not generate arbitrary meshes, multi-room buildings, or outdoor terrain.
The Creation Room: prompts begin here, and the destination portal opens after compilation.
flowchart TB
U["<b>CREATION ROOM CLIENT</b><br/><hr/><small>Unity / OpenXR client</small>"]
API["<b>API GATEWAY</b><br/><hr/><small>Prompts in, artifacts out</small>"]
P["<b>PLANNER ORCHESTRATOR</b><br/><hr/><small>Runs the planning stages</small>"]
L["<b>LLM CONTRACTS</b><br/><hr/><small>Intent, assets, layout, lighting</small>"]
R["<b>ASSET INTELLIGENCE</b><br/><hr/><small>Retrieves reviewed candidates</small>"]
W["<b>WORLDSPEC CONTRACT</b><br/><hr/><small>Validates planner handoff</small>"]
C["<b>PHASE-ZERO COMPILER</b><br/><hr/><small>Checks physical legality</small>"]
A["<b>RUNTIME CONTRACTS</b><br/><hr/><small>Publishes validated artifacts</small>"]
V["<b>DESTINATION RUNTIME</b><br/><hr/><small>Loads rooms for VR</small>"]
U --> API --> P
P --> L
P --> R
L --> W
R --> W
W --> C --> A --> V
classDef unity fill:#173b63,stroke:#93c5fd,color:#eff6ff;
classDef api fill:#334155,stroke:#cbd5e1,color:#f8fafc;
classDef planning fill:#553508,stroke:#fbbf24,color:#fffbeb;
classDef assets fill:#452366,stroke:#d8b4fe,color:#faf5ff;
classDef contract fill:#164e63,stroke:#67e8f9,color:#ecfeff;
classDef compiler fill:#123f32,stroke:#6ee7b7,color:#ecfdf5;
class U,V unity;
class API api;
class P,L planning;
class R assets;
class W,A contract;
class C compiler;
Each box above is a concrete source-level component. The four model decisions are implemented together by the LLM contract component; retrieval, physical compilation, and Unity realization remain separate components with different authority.
The system deliberately separates design authorship from physical authority:
| Layer | Responsibility |
|---|---|
| Language models | Interpret the prompt, define slots and relations, choose approved assets, and propose exact positions, yaw, support targets, and compact lighting intent. |
| Deterministic backend | Restrict candidates, validate contracts, resolve runtime assets, check bounds, supports, walls, overlaps, and safe spawn, then apply minimal legal repair. |
| Unity/OpenXR | Realize the compiled room, render lighting and materials, track controllers, provide locomotion and UI, and manage the portal transition. |
The compiler is not a second interior designer. It projects the model-authored composition into the nearest physically valid room and never sends raw model output directly to Unity.
The active planner uses four sequential structured calls because each stage narrows the next stage's problem:
- Design brief — extracts the room's purpose, atmosphere, palette direction, focal moment, and negative constraints.
- Semantic intent — converts the brief into required slots, counts, groups, typed relations, and support or wall requirements.
- Approved selection — binds those slots to reviewed asset, material, pack, and StyleKit IDs from a bounded shortlist.
- Layout and lighting — authors exact coordinates, yaw, support and wall targets, plus a compact key/fill/practical-light plan.
Deterministic validation runs between calls. This prevents an early missing slot, invented asset, unresolved support target, or malformed transform from silently reaching Unity. Earlier call-collapsing experiments reduced slot and group reliability, so the current pipeline favors coherence over minimum request count.
The planner's final handoff is a validated WorldSpec, not executable Unity instructions. The phase-zero compiler resolves registry metadata, converts local support relationships into world transforms, clamps objects to legal room surfaces, repairs or drops illegal optional placements, verifies overlap and support capacity, and proves a safe player spawn. A translation audit then compares every required model-authored placement with the compiled result so a bad proposal can be distinguished from a compiler miscompilation. Unity consumes only the resulting artifacts.
Holodeck does not expose the full Unity library to the selection model. Assets are scanned, enriched with geometry and semantic metadata, reviewed, and embedded offline. At generation time, the prompt and scene context retrieve a compact candidate set.
Holodeck uses a manually reviewed library of third-party 3D assets sourced from marketplace packs, including Unity Asset Store packages. Each asset remains governed by its original publisher and marketplace license; Holodeck does not relicense the underlying assets. The project's metadata and runtime code should not be interpreted as a license to redistribute the original asset packages.
flowchart TD
I["Reviewed Unity assets<br/>roles · bounds · pivots · affordances"] --> E["EmbeddingGemma-300M<br/>768-dimensional normalized vectors"]
Q["Prompt + semantic slots"] --> QE["Query embedding"]
E --> K["Cosine-similarity ranking"]
QE --> K
K --> RR["Reserve requested-role coverage"]
RR --> M["MMR diversification"]
M --> O["Up to 50 reviewed candidates"]
O --> X["Selection model"]
classDef source fill:#eef2ff,stroke:#6366f1,color:#1e1b4b;
classDef process fill:#f8fafc,stroke:#64748b,color:#0f172a;
classDef result fill:#ecfccb,stroke:#65a30d,color:#1a2e05;
class I,Q source;
class E,QE,K,RR,M process;
class O,X result;
- Cosine similarity ranks assets by semantic relevance.
- Requested-role reservation protects explicit needs such as a bed, desk, or lamp.
- Maximal Marginal Relevance (MMR) balances relevance with variety so the shortlist is not filled with near-duplicates.
- The model may select only known, reviewed asset IDs; invented assets fail validation.
The active index contains 342 approved assets, each represented by a normalized 768-dimensional vector. Query text combines the user prompt with normalized scene concepts, mood tags, style descriptors, and required roles. Because both query and asset vectors are normalized, cosine similarity can be computed efficiently as a dot product.
Embeddings are used only for retrieval. They do not choose final assets, infer coordinates, or judge physical validity. Their job is to reduce the full reviewed inventory to at most 50 semantically relevant and sufficiently varied candidates before the selection call.
Asset metadata remains as important as the vector itself. Each reviewed record can carry semantic roles, room affinity, footprint and bounds, pivot and front-axis corrections, wall-mount behavior, support requirements, and usable local support surfaces. That metadata connects a semantic match such as “side table” to correct physical realization.
Support surfaces are geometry, not labels. Tables and shelves expose bounded local surfaces with usable dimensions and heights; supported children retain explicit parent placement IDs. If overlap or wall repair moves a parent, its dependent props move with it instead of being left floating at stale world coordinates.
| System | Role in the architecture |
|---|---|
| StyleKits | Bind coherent shell materials, palette, lighting defaults, and Quest-safe runtime polish to the selected design direction. |
| Realization registry | Maps approved backend IDs to concrete Unity prefabs and their scale, pivot, forward-axis, and geometry corrections. |
| Room Narrator | Produces phase-aware chatter from a deterministic gold bank, uses an LLM only as fallback, and caches normalized ElevenLabs audio. |
| Failure forensics | Stores compact raw stage requests/responses and exact validator paths only when a stage fails; successful runs stay compact. |
| Translation audit | Proves required placements survive compilation and records displacement, yaw change, and authorized substitution before runtime handoff. |
A successful request produces two artifacts under build/<world_id>/:
phase0.json— room shell, compiled placements, material bindings, physical audit, and safe spawn.manifest.json— the runtime-facing contract consumed by Unity.
Selection or layout failures include compact forensic data captured from the structured request, raw response, and backend validation errors. Successful runs omit that verbose payload.
git clone https://github.com/lesprgm/JuniorIS.git
cd JuniorIS
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
cp .env.example .envAdd the provider credentials required by .env. The default configuration uses Gemini for planning and OpenRouter for layout; ElevenLabs is optional unless voice synthesis is needed.
Semantic retrieval also expects the gated EmbeddingGemma weights at models/embeddinggemma-300m and a compatible sentence-transformers installation. The 1.2 GB model is intentionally not committed to this repository.
set -a; source .env; set +a
python3 -m src.api.serverThe backend starts at http://127.0.0.1:8000.
curl -s http://127.0.0.1:8000/plan_and_compile \
-H 'Content-Type: application/json' \
-d '{
"prompt_text": "A calm reading room with an armchair, a side table, warm lighting, and a large rug",
"user_prefs": {"prompt_mode": "llm", "llm_required": true}
}' | python3 -m json.toolUseful endpoints:
| Endpoint | Purpose |
|---|---|
GET /healthz |
API health and contract version |
POST /plan_and_compile |
Generate and compile a room |
POST /voice/chatter_plan |
Build phase-aware Room Narrator chatter |
POST /voice/transcribe |
Transcribe a headset microphone recording |
POST /voice/tts |
Synthesize and cache ElevenLabs audio |
/build/... |
Serve compiled artifacts to Unity |
pytest -qThe Unity 2022.3/OpenXR client is maintained as a sibling project. It submits prompts to this API and consumes the generated manifest and phase-zero artifacts.
src/
api/ FastAPI endpoints and generation orchestration
llm/ Structured model contracts and provider transport
planning/ Scene planning, retrieval, selection, and layout normalization
compilation/ Physical validation, minimal repair, and artifact output
indexing/ Asset scanning, enrichment, review, and embedding generation
runtime/ Realization registry and safe-spawn logic
voice/ Room Narrator chatter, ElevenLabs synthesis, and audio caching
schemas/ WorldSpec, manifest, StyleKit, pack, and API contracts
stylekits/ Reviewed palette, shell, lighting, and runtime-polish presets
tests/ Planner, compiler, API, schema, runtime, and voice regressions
docs/ Architecture and implementation references
- Approved, pre-authored assets only
- One room shell per request
- Sequential model stages favor layout quality over minimum latency
- Final legality always overrides aesthetic intent
- Unsupported optional content may be repaired or omitted
- Portal readiness requires a valid compiled world and safe spawn