Currently this is just a VR viewer for StableDiffusion outputs, but I'm planning to make it into a fully fledged RPG maker
When in VR, press "A" button on your controller to record audio.
- this is convered using OpenAI's whisper into a text prompt
- this text prompt is rendered using StableDiffusion
- a depth map is generated using DPT
- and that is all rendered back for you in VR
To run, you need to start the server: flask_app.py and then go to the page /static/examples/whisper.html
The easiest way is probably just with colab
Check out the demo