Maverick XR Agent for Unity 1.1.0

Version 1.1.0 adds support for Maverick 1.1, the model update that learned to take corrections, follow what people look and point at, and tell who is speaking.

What is new

• Undo. A twelfth tool, undo, reverses the agent's last one to three commands exactly: objects, rooms, states, positions and what the player holds. It refuses, and changes nothing, when something else changed an object in the meantime, when a command failed or cannot be reversed, or when there is nothing to undo.
• Attention and people. The scene can carry what each person is looking at or pointing at, who is speaking and whose hands the agent acts with. SceneAgentGaze can report gaze and pointing for several people, and Command and Answer take the speaker.
• Short memory. The model sees its recent commands and any open question, and tool results now say what changed.
• Changes made outside the agent. SceneWorld.ApplyExternalChange and the syncFromTransforms option tell the model and undo about objects that a person, physics or your own code moved.
• Cancel. SceneAgentSession.Cancel() stops a running command cleanly.

All of this is opt-in. Set SceneAgent.schemaVersion to V1 to use it with Maverick 1.1. The default, V0, behaves exactly as in 1.0, and both Maverick 1.0 and Maverick 1.1 work with it.

Also fixed since 1.0

The output check now rejects missing required arguments, arguments of the wrong type and numbers that are not finite, and calls inside the model's reasoning never run. A command sent while another one is running is no longer started, and a cancelled run no longer keeps acting on the scene. The full list is in the changelog.

Demo

A demo of Maverick 1.1 (1 min 44 s) is attached to this release as maverick-1-1-demo.mp4. It follows a technician and a remote expert in a three-room house, and includes a correction, a fetch from another room, a two-step plan and an undo that is refused. It was recorded live on a laptop with a 4 GB GPU; the script was rehearsed and the expert's voice is text-to-speech.

Install

In Unity, choose Window > Package Manager > + > Add package from git URL and enter:

https://github.com/mcbu-xrlab/Maverick-Unity.git#v1.1.0

The model guide lists the Maverick 1.1 and 1.0 files with their checksums. The schema 1.1 guide explains the new format and how to migrate.

Worth knowing

• A remote person's request can move objects or press buttons in the on-site scene. The model does not check permissions, so enforce them in your application.
• Maverick 1.1 still tends to act instead of asking in situations it has not seen. Its model card lists this and the other limits.
• Tested on Windows with Unity 2022.3. Linux, macOS and standalone headsets have not been validated.