Sigma How to Build

Building and Configuring SIGMA

SIGMA (Situated Interactive Guidance Monitoring and Assistance) is a distributed mixed-reality app that runs in a client-server configuration. The client app runs on the HoloLens device and streams sensor data live to a server app running on a desktop (or laptop) computer, which in turn streams back rendering instructions to the client app.

NOTE: To run the app without problems, it is important that you have a reasonably powerful server machine (with a modern CPU and GPU) and a very good network connection.

Prerequisites

1. Ensure that the prerequisites for building \psi on Windows are installed. See Building the Codebase for details.

2. Enable Developer Mode on the desktop. In the Windows search bar, type "Developer features" and turn on Developer Mode in the resulting Settings window.

3. Enable Developer Mode and Device Portal on the HoloLens. Then verify that you can connect to the HoloLens via the Device Portal. For more details see https://learn.microsoft.com/en-us/windows/mixed-reality/develop/advanced-concepts/using-the-windows-device-portal.

4. Enable Research Mode on the HoloLens. In the Device Portal, navigate to "System", then select "Research mode". Check the box that says "Allow access to sensor streams". Then reboot the HoloLens (you can do this from within the Device Portal by clicking on the "Power" button located at the upper-right of the page). For more information on Research Mode, see https://learn.microsoft.com/en-us/windows/mixed-reality/develop/advanced-concepts/research-mode.

5. SIGMA uses Azure Speech service, and optionally also Azure OpenAI and Azure Spatial Anchors. To run SIGMA, you will need an Azure subscription with a Speech resource and, optionally, an Azure OpenAI resource and a Spatial Anchors resource.

Build and Deploy SIGMA

1. Open the Applications\Sigma\Sigma.sln solution in Visual Studio.

2. From the Build menu, select Rebuild Solution.

3. Locate the SigmaApp project in the Solution Explorer. Right-click on it and select Properties.

4. In the Debug properties under Start options, select Remote Machine as the Target device, then enter the IP address of the HoloLens in the Remote machine field.

5. Choose Universal (Unencrypted Protocol) as the Authentication Mode.

6. From the Build menu, select Deploy SigmaApp. You can also deploy the app by right-clicking on the SigmaApp project in the Solution Explorer and selecting Deploy. The first time you do this, you may be prompted to enter a pairing PIN. On the HoloLens, open the Settings app, go to Update & Security, and select For Developers. Tap on the "Pair" button. This should display a PIN which you should then enter in Visual Studio. You only need to do this once. For more info, see https://learn.microsoft.com/en-us/windows/mixed-reality/develop/advanced-concepts/using-visual-studio

Configure SIGMA

The SIGMA client and server apps communicate over the network and they need to be configured appropriately. The apps are configured via XML configuration files. Initial configuration files are generated by the server and client apps on their first runs; they then must be manually edited by the user for their particular setup.

The server configuration file 'sigma.server.config' is located in the user's 'Documents\Sigma\<MachineName>' folder on the server, while the client configuration file 'sigma.client.config.xml' is stored under the 'Documents' folder on the HoloLens.

Both the server and the client configuration files can define multiple configurations, which are matched by name. The client will display the list of available configurations on the HoloLens on startup. The user selects the name of the configuration they want to run, and upon pressing the Run button, the client will communicate to the server the name of the configuration, which the server then picks by name from the list of available configurations on the server.

Server (SigmaComputeServer) Configuration

The server is implemented by the SigmaComputeServer project under Applications\Sigma. When you first install SIGMA, you will need to run this server once in order to create the server configuration file, which you will then manually edit.

1. Run the SigmaComputeServer project under 'Applications\Sigma' in the Solution Explorer. This will start the server in a console window. When you see the prompt "Press any key to exit", go ahead and press any key to exit.

2. Locate and open the 'sigma.server.config.xml' configuration file which the previous step should have created in a new folder under your 'Documents' folder named 'Sigma\<MachineName>' (e.g. 'C:\Users\<userprofile>\Documents\Sigma\<MachineName>\sigma.server.config.xml').

3. In the SigmaComputeServerPipelineConfiguration section under PipelineConfigurations, make the following changes:

   • Change OutputPath to the folder in which to save the output data stores.

   • Change ResourcesPath to the full path of the SIGMA Resources folder in your local repo, e.g. '<repo-root>\Applications\Sigma\Sigma\Resources'.

   • Change PersistentStateFilename to the prefix path of the '.tasklibrary.json' file located in your Resources folder, e.g. '<repo-root>\Applications\Sigma\Sigma\Resources\sigma.state.diamond'.

   • Optionally, to use LLM queries, change LLMQueryLibraryFilename to the full path of the LLM queries file in your local repo, e.g. '<repo-root>\Applications\Sigma\Sigma\Resources\llmqueries.json'. Otherwise, leave this field empty and change TaskGenerationPolicy to 'FromLibraryOnly'.

   • Change AutoStartTaskName to empty by deleting the placeholder text.

4. Save your changes and close the configuration file.

Additional Server Configuration

Environment Variables

The server uses the Azure Speech service for speech to text and, optionally, the Azure OpenAI service for performing LLM queries. Access to these services needs to be configured via environment variables.

In the Windows search bar, type "environment" and select "Edit the system environment variables". Create the following User environment variables (the AzureOpenAI variables are only necessary if you are configuring SIGMA to use LLM queries as outlined in the Server (SigmaComputeServer) Configuration section):

Variable	Value
CognitiveService.Speech	<your Azure Speech service key>
CognitiveServices.SpeechRegion	<your Azure Speech service region>
AzureOpenAI.Key	<your Azure OpenAI service key>
AzureOpenAI.Endpoint	<your Azure OpenAI service endpoint>

You can obtain the above values from the respective resources in your Azure subscription. Note that you will need to exit and restart Visual Studio in order for it to pick up the new environment variables.

Client (SigmaApp) Configuration

The client is implemented by the SigmaApp project under 'Applications\Sigma'. When you first run the app on the HoloLens, it will create a default configuration file on the device which you will then need to download, edit, and re-upload to the device by following the steps below.

1. Make sure that the HoloLens is powered on and you are logged in on the device.

2. Bring up the Start menu, click on All Apps, then scroll down to locate the Sigma app which was deployed to the device in the previous section Build and Deploy SIGMA. Tap on the Sigma app tile to launch the app.

3. If for some reason the app was not previously deployed, you can also deploy and launch SIGMA from Visual Studio by right-clicking on the SigmaApp project, then selecting 'Debug -> Start New Instance'. Note that the Debug properties for the project (specifically the Start options) must have been correctly configured as described in the Build and Deploy SIGMA section.

4. When the app launches, it will initially display a "Please wait. Initializing ..." message. When the main UI appears, click the 'Exit' button at the bottom to exit the app. This should create an initial configuration file in the 'Documents' folder on the HoloLens.

5. Connect to the HoloLens using the Device Portal.

6. Navigate to 'System -> File explorer', then select the 'Documents' folder. Locate the 'sigma.client.config.xml' configuration file and download it to your desktop machine.

7. Open the downloaded file and make the following changes in the SigmaAppConfiguration section:

   • Change the Name to Diamond (this name should match the name of a corresponding SigmaComputeServerPipelineConfiguration section in the server configuration).

   • Replace the ComputeServerNames section with the following:

     <ComputeServerNames>
       <string>server-hostname</string>
       <string>server-ip-address</string>
     </ComputeServerNames>

     substituting server-hostname and server-ip-address with the hostname and IP address of the desktop machine running the SigmaLiveComputeServer.

   • You may add more lines representing additional IP addresses or hostnames as needed (e.g. the IP address of the USB interface on your desktop machine).

   • You may also create additional SigmaAppConfiguration sections, changing the Name property to match the Name property of a corresponding SigmaComputeServerPipelineConfiguration section in the server configuration file. These names can be anything you want so long as they are unique and both the server and client configurations each contains a section with a matching name.

   • Once you have finished editing the 'sigma.client.config.xml' file, upload it back to the 'Documents' folder on the HoloLens using the Device Portal.

Additional Client Configuration

Azure Speech

The SIGMA client app also uses the Azure Speech service for text to speech (synthesis). To provide the service key to the app, you will need to create a text file named 'CognitiveServicesSpeechKey.txt' containing the key (as obtained from the Azure Speech resource in your Azure subscription). Upload this file to the HoloLens 'Documents' folder via the Device Portal.

World Spatial Anchors

The SIGMA app can use either the local spatial anchor store or cloud-based Azure Spatial Anchors to persist the world coordinate system, which keeps the world coordinates in the data consistent across runs. Each SigmaAppConfiguration section in the client configuration file may contain a WorldSpatialAnchorId property to specify the ID of the spatial anchor to use. If this property is not specified, the app will prompt the user to either create a new world spatial anchor whose ID will be saved to the app's configuration file, or alternatively continue with a temporary world spatial anchor that will not be persisted across runs.

By default, SIGMA will use the local spatial anchor store to save the world spatial anchor. This enables the app to use the same world spatial anchor on the device across runs, but it will not be possible to share the same spatial anchor across multiple devices (if anchoring the spatial data collected using different devices to a common coordinate system is important, for example). Additionally, if the SIGMA app is deleted from the device, the persisted world spatial anchor will be lost.

For the above reasons, users may want to use persistent world spatial anchors that can be shared across devices. To do this, you will need to use Azure Spatial Anchors instead of the local spatial anchor store.

To enable Azure Spatial Anchors, create a text file named 'AzureSpatialAnchorsAccountInfo.txt' containing the following lines:

AccountDomain: <spatial-anchors-account-domain>
AccountKey: <spatial-anchors-account-key>
AccountId: <spatial-anchors-account-id>

You can obtain these values from the Spatial Anchors Account resource in your Azure subscription. Upload this file to the 'Documents' folder on the HoloLens via the Device Portal. The existence of this file in the 'Documents' folder will cause the app to use Azure Spatial Anchors when attempting to locate or create the world spatial anchor.

When a world spatial anchor is first created using Azure Spatial Anchors, the app will save the ID of the anchor under the selected SigmaAppConfiguration in the configuration file. The ID will be a unique GUID saved under the WorldSpatialAnchorId property. If you wish to use the same world spatial anchor on multiple devices, you will need to copy the ID from the configuration file on one device to the configuration file on the other device(s).