Ubiq-Genie is a framework that enables you to build server-assisted collaborative mixed reality applications with Unity using the Ubiq framework. This is particularly useful for building multi-user applications that require server-side processing such as generative models, conversational agents, and real-time transcription. For more information, please refer to the Ubiq-Genie paper.
Note: Ubiq-Genie currently uses the Ubiq v0.4.2 client code. An updated version of Ubiq-Genie supporting the latest versions of Ubiq will be released later in summer 2024.
These instructions will get you a copy of the project up and running to run the samples and to start building your own applications. Please note that Ubiq's server dependencies only support installation on Windows and Linux at the moment. An alternative for macOS users is to use GitHub Codespaces to run the server-side components (in this case, you may skip installing Node.js, step 1, 2, and 3). The client-side components can be run on practically any platform supported by Unity and Ubiq.
-
Clone this repository somewhere on your local PC.
-
Open a terminal in the
Server
folder and runnpm install
to install the dependencies. This includes the Node.js server of Ubiq. -
Create a virtual environment using
venv
orconda
. -
Install
torch
,torchvision
, andtorchaudio
version 1.13 using the installation instructions on the PyTorch website. Note: please ensure to install the correct version of PyTorch matching the CUDA version of your GPU. -
From the
Server
folder, install the Python dependencies by runningpip install -r samples/requirements.txt
orconda install --file samples/requirements.txt
, depending on your virtual environment. -
In Unity, open the
Unity
folder. To add Ubiq to theUnity Hub
, open theUnity Hub
, clickAdd
, then navigate to/Ubiq/Unity
and clickSelect Folder
. -
Read the README file in the corresponding folder in the
Server/samples/apps
folder for further setup instructions. For a list of available samples, see the Samples section below.
For more information on how to use Ubiq-Genie, please refer to the README file in the Server
folder.
The Server/samples
folder contains a number of samples that demonstrate how to use Ubiq-Genie. For more information on how to use these samples, please refer to the README files in the corresponding folders. Currently, the following collaborative samples are available:
- Texture Generation: generates a texture based on voice-based input and an optional ray to select target objects
- Multi-user Conversational Agent: a conversational agent that can be interacted with by multiple users
- Transcription: transcribes audio streams of users in a room
For a demo video of the samples, please refer to the Ubiq-Genie demo video.