OpenSpaceGuide: steering OpenSpace with natural language

Pictures taken in the Visualiseringscenter C dome theater.

---

This repo contains a script for piloting an OpenSpace instance with OpenAI's GPT4 and Whisper. More details about this project on OSF — https://osf.io/jcgzt/

Instructions

• Setup your OPENAI_API_KEY environment variable. You need a paid access to OpenAI's API, see https://platform.openai.com/api-keys.
• Install the Python dependencies — pip install -R requirements.txt (it's recommended to setup a virtual environment first, see this).
• Install OpenSpace — this fork contains a patch to display the text prompts on screen with --text-widget. Otherwise, install from the main repo.
• Launch OpenSpace, then start the script (python main.py --input keyboard).

CLI reference

• --address <addr>, --port <port>, --password <pwd>: OpenSpace server configuration. Default values will work for a locally running instance.
• --input <keyboard|speech>: keyboard or speech. Whether to use a text input in the console or speech through the microphone via Whisper.
• --trigger <key>: Set this to use a key press to start/stop listening (if --input is speech). Otherwise, it listens continuously.`
• --microphone <index>: Microphone index to use. Run once to see the list.
• --text-widget: Display prompts and responses in OpenSpace (requires using the fork)
• --targets <comma,separated,list>: List of OpenSpace targets (Scene Identifiers) that the LLM should be aware of. Defaults to all visible renderables when the script is started.

Acknowledgements

This work is supported by the Knut and Alice Wallenberg Foundation (KAW 2019.0024), the Marcus and Amalia Wallenberg Foundation (MAW 2023.0130), and efforts from Visualiseringscenter C (Norrköping, Sweden).