PartInteractor: Intent-Driven Part-Aware 3D Authoring for Continuous Co-Creation in XR

| 📄 Paper | 🌐 Project Page |

---

📌 Overview

PartInteractor rethinks generative XR authoring by treating semantic part hierarchies as first-class interactive representations, enabling users to move beyond one-shot object generation toward flexible, fine-grained human-AI co-creation.

The system supports multimodal authoring inputs, including:

• 🎙️ Speech
• ✏️ in-situ Sketch
• 📷 Headset-captured Image
• 🧩 Component-/Object-level Interaction
• 🥽 VR/MR Authoring Workflows

Note: The current release is preliminary and intended primarily for research and prototyping. We plan to improve functionality, usability, and robustness in future updates.

---

🗂️ Project Structure

PartInteractor/
├── src/                              # Mian project code
│  ├── 3DGen/                         # Unity XR project
│  │   ├── Assets/
│  │   │   ├── Scenes/                # Main VR/MR scenes
│  │   │   ├── Scripts/               # Unity client scripts
│  │   │   ├── Resources/             # Local prefabs and resources
│  │   │   └── StreamingAssets/       # Local 2D retrieval assets
│  │   │
│  ├── ForwardServer.py               # OpenAI forwarding server
│  ├── Model3DServer.py               # 3D generation server
│  ├── get_3d_mesh_and_gaussian.py    # Part-aware 3D generation script
│  ├── testAPI.py                     # OpenAI API connection test
│  ├── testS2I.py                     # 2D generation connection test
│  ├── requirements.txt               # Python dependencies
└── README.md

---

✅ Prerequisites

1. 🖥️ Hardware

   • Meta Quest 2/3/Pro (3 is recommended)
   • A PC capable of running Unity and local Python servers

2. 🧰 Software

   • Unity Hub Install
   • Unity 2022.3.58f1
   • Python 3.8.20
   • Meta Quest Space Setup
   • Meta Quest Setup Link for App Developement

3. 🔑 API Access

   • OpenAI API keys [Commend Interpretion & 2D Generation]
   • Hugging Face Token [3D Generation]

---

🚀 Installation

1️⃣ Clone or Download the Repository.

git clone https://github.com/SNeC-Lab-PSU/PartInteractor.git
cd PartInteractor

2️⃣ Configure the OpenAI API Key

Update the .env file:

OPENAI_API_KEY=your_openai_api_key_here

3️⃣ Configure the Hugging Face token:

Update the get_3d_mesh_and_gaussian.py file:

client = Client("omnipart/OmniPart",
                 token="your_hugging_face_token_here")

4️⃣ Create the Python Environment

Using Anaconda and install the required packages:

conda create -n partinteractor python=3.8.20
conda activate partinteractor
pip install -r requirements.txt

---

⚙️ Configuration

Before running the system, configure the local server addresses.

1️⃣ OpenAI Forwarding Server

In ForwardServer.py, set the TCP bind host and port_1 for the OpenAI bridge.

2️⃣ Unity/Headset Client Connection

In 3DGen/Assets/Scripts/Utils.cs, set ServerIP and ServerPort to match the TCP bind host and port_1 in ForwardServer.py.

3️⃣ PC 3D Generation Server

In Model3DServer.py, set PYTHON_EXE, host, and port_2 for your local 3D generation environment.

4️⃣ Unity/Headset 3D Model Loader

In 3DGen/Assets/Scripts/GenModelLoader.cs, set serverBaseUrl to match the host and port_2 in Model3DServer.py.

For Unity Editor with Quest Link, local addresses such as 127.0.0.1 may be sufficient. For standalone headset deployment, use the computer's IPv4 Address and make sure firewall rules allow the required ports.

---

🔗 Connection Tests

1️⃣ Test OpenAI API Connection

python testAPI.py

2️⃣ Test Hugging Face Token Connection

python testS2I.py

3️⃣ Unity Setup

Open 3DGen/ in Unity Hub using Unity 2022.3.58f1. Let Unity resolve packages and import TextMesh Pro essentials if prompted.

---

▶️ Running the Project

1. Start the OpenAI forwarding server:

   python ForwardServer.py
   

2. Start the 3D model generation server in a second terminal:

   python Model3DServer.py
   

3. Open one of the main Unity scenes:

   • 3DGen/Assets/Scenes/Main_VR.unity for VR authoring.
   • 3DGen/Assets/Scenes/Main_MR.unity for MR authoring.

4. Connect a Meta Quest headset and enable Quest Link.

5. You can either press Play directly in the Unity Editor through Quest Link, or Build and Deploy the project to the headset for standalone execution.

6. Speak a command. Voice recording can be triggered by:

   • Holding the right controller B button.
   • Holding the R key in the Unity Editor.
   • Pinching the left thumb and middle finger.

   Release the trigger to stop recording and submit the instruction.

7. Additional MR hand gestures:

   • Close all five fingertips of the right hand together to capture a screenshot/camera image （MR mode）.
   • Pinch the left thumb and ring finger to cancel multi-selection mode.

8. Example Commands:

   Create a object for me.
   Convert my drawing to a 3D object.
   Generate this object from the camera image.
   Make the object grabbable.
   Make the object partail grabbable.
   Explode the object.
   Assemble the object.
   Split this object.
   Enter selection mode.
   Merge the selected parts.
   Delete this component.
   Change this component's color to red.
   Move this object infront of me.
   

---

🧱 Adding Local Resources

1️⃣ Reusable Unity prefabs can be added under:

3DGen/Assets/Resources/

2️⃣ Add your 2D reference images for retrieval under:

3DGen/Assets/StreamingAssets/ObjectCandidates/<object-name>/

Supported image extensions are .png, .jpg, and .jpeg.

---

🙏 Acknowledgements

Those packages are already included in the source code. You are only required to install them if you want to use the latest version of the packages or meet some issues. Also, the sources are listed here for credit purposes.

1. First, install NuGet for Unity.

   • Open Window > Package Manager
   • Click + button, and select Add package from git URL...
   • Enter the following URL and click Add button

     https://github.com/GlitchEnzo/NuGetForUnity.git?path=/src/NuGetForUnity
     

2. Use plugin of OpenAI API Wrapper.

   • Navigate to NuGet > Manage NuGet Packages
   • Search for OpenAI
   • Install OpenAI wrapper for Unity project

3. The VR environment is from A.R.S|T. and robot model is from Zeuk Games.

---

📚 Citation

If you use this implementation, methodology, or any part of this repository in your research, please cite our paper. We truly appreciate your interest in our work!

@inproceedings{jiang2026partinteractor,
  title={PartInteractor: Intent-Driven Part-Aware 3D Authoring for Continuous Co-Creation in XR},
  author={Jiang, Jianan and Li, Bin},
  booktitle={Proceedings of the 39th Annual ACM Symposium on User Interface Software and Technology},
  pages={1--13},
  year={2026}
}

---

✉️ Contact

If you have any questions, suggestions, collaboration ideas, or would like to discuss future release plans, please feel free to reach out.

We are happy to connect with researchers and practitioners interested in 3D authoring, generative XR systems, and human-AI co-creation.