Skip to content

Sync Blender mesh to Vision Pro with gesture handling

Notifications You must be signed in to change notification settings

daniloc/GeometryLink

Repository files navigation

Prototype: Sync Blender mesh to Vision Pro with gesture handling

Blender.plugin.example.mp4

This project uses WebSockets to transmit the selected mesh in Blender to Vision Pro, with interactive gestures.

Usage

  • Install Blender Plugin.zip as an add-on in Blender
  • Edit connect() in ContentView.swift with the hostname of the machine running Blender
  • Build/Run to Simulator or a Vision Pro device
  • Select a mesh in Blender. It should appear in the Vision Pro volume. You can grab and scale it with your hands.

⚠️☣️ The python code is bad!

I've shipped almost no python in my life. The add-in is the product of extended collaboration with an LLM and should be re-written by someone who knows how to write python network code.

Do not reuse the python code. Feel free to open a PR with replacement, non-toxic python code.

Pipline

Server

  • Mesh is selected in Blender
  • The mesh is converted locally to USD
  • USD file is converted to base64 string and sent over WebSocket

Client

  • base64 string is received and converted back into USD file
  • USD file is imported as a RealityKit entity and attached as a child to an anchor entity

Known issues

  • There's a bit of a hang on closing Blender, owing the bad python code
  • Can't figure out how to make Bonjour work, but partial implementations are included; the service is broadcasting, but none of the details seem to populate. PR's welcome there too.
  • There's an upper limit on mesh complexity; chunking or some other strategy should be implemented to allow bigger files to be transferred over WebSocket
  • I'm sure there's a better way to manage package dependencies in Blender's python; feel free to propose one!

Missing Apple code

Gesture handling borrows heavily from a file that used to exist at this URL but has since vanished:

https://developer.apple.com/documentation/realitykit/transforming-realitykit-entities-with-gestures

The docs/project had a bug around programmatically attaching the gesture, so maybe they're reworking it.