#1734 actions based vr input #1735

richardTingle · 2022-01-07T16:26:28Z

This leaves all the old stuff there (the oculus, osvr & other openvr) but focusses on the LWJGL openvr for the short term future (I guess OpenXR later on as the more long term future). I have ended up with a bit of a leaky abstraction where VRInputAPI wants to talk about openVr specific stuff like action manifests but I couldn't avoid that without requiring users to cast to LWJGLOpenVRInput which feels equally as bad (especially as LWJGLOpenVRInput is probably the only one that's going to be usable till OpenXR comes along)

This allows Jmonkey VR to use the actions manifest style VR (which is the non deprecated openVr api for mapping button presses to "stuff" in game).

An example action manifest might look like

{
  "default_bindings": [
    {
      "controller_type": "oculus_touch",
      "binding_url": "oculusTouchDefaults.json"
    }
  ],
  "actions": [
    {
      "name": "/actions/main/in/OpenInventory",
      "requirement": "mandatory",
      "type": "boolean"
    },
    {
      "name": "/actions/main/in/scroll",
      "type": "vector2",
      "requirement": "mandatory"
    },
    {
      "name": "/actions/main/out/hapticTest",
      "type": "vibration",
      "requirement": "optional"
    },
    {
      "name": "/actions/main/in/leftTrigger",
      "type": "vector1",
      "requirement": "mandatory"
    }
  ],
  "action_sets": [
    {
      "name": "/actions/main",
      "usage": "leftright"
    }
  ],
  "localization" : [
    {
      "language_tag": "en_us",
      "/actions/main" : "My Game Actions",
      "/actions/main/in/OpenInventory" : "Open Inventory"
    }
  ]
}

With a default bindings looking like

{
  "action_manifest_version" : 0,
  "bindings": {
    "/actions/main": {
      "haptics" : [
        {
          "output" : "/actions/main/out/hapticTest",
          "path" : "/user/hand/left/output/haptic"
        }
      ],
      "sources" : [
        {
          "inputs" : {
            "click" : {
              "output" : "/actions/main/in/OpenInventory"
            }
          },
          "mode" : "button",
          "path" : "/user/hand/left/input/x"
        },
        {
          "inputs" : {
            "position" : {
              "output" : "/actions/main/in/scroll"
            }
          },
          "mode" : "joystick",
          "path" : "/user/hand/left/input/joystick"
        },
        {
          "inputs" : {
            "pull" : {
              "output" : "/actions/main/in/leftTrigger"
            }
          },
          "mode" : "trigger",
          "path" : "/user/hand/left/input/trigger"
        }
      ]
    }
  },
  "category" : "steamvr_input",
  "controller_type" : "oculus_touch",
  "description" : "Bindings for a oculusTouch controller",
  "name" : "Bindings for a oculusTouch controller",
  "options" : {},
  "simulated_actions" : []
}

Then it would be used like:

    public static void main(String[] args) {
        AppSettings settings = new AppSettings(true);
        VREnvironment env = new VREnvironment(settings);
        env.initialize();

        if (env.isInitialized()){
            VRAppState vrAppState = new VRAppState(settings, env);
            vrAppState.getVRinput().registerActionManifest("C:/Users/richa/Documents/Development/android/jmonkeyVrTest/src/main/resources/actionManifest.json", "/actions/main");

            Main app = new Main(vrAppState);
            app.setLostFocusBehavior(LostFocusBehavior.Disabled);
            app.setSettings(settings);
            app.setShowSettings(false);
            app.start();
        }
    }

And within the update loop like

    List<Geometry> handGeometries = new ArrayList<>();

    @Override
    public void simpleUpdate(float tpf) {

        VRAppState vrAppState = getStateManager().getState(VRAppState.class);
        int numberOfControllers = vrAppState.getVRinput().getTrackedControllerCount(); //almost certainly 2, one for each hand

        //build as many geometries as hands, as markers for the demo (Will only tigger on first loop or if number of controllers changes)
        while(handGeometries.size()<numberOfControllers){
            Box b = new Box(0.1f, 0.1f, 0.1f);
            Geometry handMarker = new Geometry("hand", b);
            Material mat = new Material(assetManager, "Common/MatDefs/Misc/Unshaded.j3md");
            mat.setColor("Color", ColorRGBA.Red);
            handMarker.setMaterial(mat);
            rootNode.attachChild(handMarker);
            handGeometries.add(handMarker);
        }

        VRInputAPI vrInput = vrAppState.getVRinput();

        for(int i=0;i<numberOfControllers;i++){
            if (vrInput.isInputDeviceTracking(i)){ //might not be active currently, avoid NPE if that's the case

                Vector3f position = vrInput.getFinalObserverPosition(i);
                Quaternion rotation = vrInput.getFinalObserverRotation(i);

                Geometry geometry = handGeometries.get(i);
                geometry.setLocalTranslation(position);
                geometry.setLocalRotation(rotation);
            }
        }

        if (vrInput.getDigitalActionState("/actions/main/in/OpenInventory").state){
            System.out.println("openInventory");
            vrInput.triggerHapticAction("/actions/main/out/hapticTest", 1, 5, 1 );
        }

        System.out.println("left trigger" + vrInput.getAnalogActionState("/actions/main/in/leftTrigger").x);
    }

Assuming people are happy with this I intend to expand the wiki to document how to write action manifests etc

richardTingle · 2022-01-07T16:33:45Z

jme3-vr/src/main/java/com/jme3/input/vr/lwjgl_openvr/LWJGLOpenVRInput.java

+            actionSetHandles.put(actionSet,actionSetHandle);
+        }
+
+        //Todo: this seems to imply that you could have multiple active action sets at once (Although I was not able to get that to work), allow multiple action sets


It annoyed me that I couldn't get multiple action sets working. But then I didn't really understand why you'd want more than one action set, or why you'd want to turn them off. So I haven't worried about it too much

The way multiple action sets could be used is, for example, if you want to have input per AppState. Or maybe one action set for interacting with the world and another for interacting with the user interface. There definitely are use cases, but I don't think it's a big deal if we don't support it, especially as the OpenVR documentation is a bit vague on how that is supposed to work.

…dern actions based input/vr/AnalogActionState.java All the other apis are vender specific or are alternative of versions OpenVR, so are deprecated - Deprecate everything but lwjgl openVr - Add support for restricting analog, digital and haptics to a particular hand - improve vr javadocs - Add haptics to the new openVR api - Add analogue inputs - Add action based digital controls into jme-vr

grizeldi

Wrote down a couple of documentation nitpicks, but overall from a code perspective looks good to me. I don't have any jme powered VR projects on hand, nor the time to make one from scratch, so I couldn't test if it actually runs though.

grizeldi · 2022-01-09T13:53:56Z

jme3-vr/src/main/java/com/jme3/input/vr/VRInputAPI.java

+     * {@link #registerActionManifest} must have been called before using this method.
+     *
+     * @param actionName The name of the action. E.g. /actions/main/in/openInventory
+     * @param restrictToInput the input to restrict the action to. E.g. /user/hand/right. Or null, which means "any input"


Maybe add the path for the left hand in here as well, just for the sake of completeness

grizeldi · 2022-01-09T13:56:54Z

jme3-vr/src/main/java/com/jme3/input/vr/VRInputAPI.java

    /**
     * Check if the given button is down (more generally if the given input type is activated).
+     *
+     * Deprecated as should use an actions manifest approach. See {@link #registerActionManifest}


I'd add a notice about action based approach only working on OpenVR on all places we tell people to use that instead, so the one in a million people that might still be using some other backend don't get too confused.

Fair point. I've made that clear on the deprecation notices

grizeldi · 2022-01-09T13:59:16Z

jme3-vr/src/main/java/com/jme3/input/vr/VRInputAPI.java

+     * Note that registering an actions manifest will deactivate legacy inputs (i.e. methods such as {@link #isButtonDown}
+     * will no longer work
+     *
+     * This option is only relevant to OpenVR


I'd link this somewhere in this javadoc comment.

Ooo, that's very useful. I ended up backwards engineering how action manifests worked from the examples I found but thats much more useful, thanks. I've added that

grizeldi · 2022-01-09T14:12:13Z

jme3-vr/src/main/java/com/jme3/input/vr/lwjgl_openvr/LWJGLOpenVRInput.java

+            actionSetHandles.put(actionSet,actionSetHandle);
+        }
+
+        //Todo: this seems to imply that you could have multiple active action sets at once (Although I was not able to get that to work), allow multiple action sets


The way multiple action sets could be used is, for example, if you want to have input per AppState. Or maybe one action set for interacting with the world and another for interacting with the user interface. There definitely are use cases, but I don't think it's a big deal if we don't support it, especially as the OpenVR documentation is a bit vague on how that is supposed to work.

…n based API is only for OpenVr

grizeldi

Looks good to me.

stephengold · 2022-01-09T18:56:25Z

Thanks for the review, @grizeldi.

I couldn't test if it actually runs though.

@richardTingle Have you tested the new code in an application?

richardTingle · 2022-01-09T20:39:51Z

Thanks for the review, @grizeldi.

I couldn't test if it actually runs though.

@richardTingle Have you tested the new code in an application?

@stephengold yes, I did an install to maven local, created a project (with code much like in my intro to this PR) in intelliJ and ran on an oculus quest 2 (via virtual desktop so it pretended to be a PCVR system)

stephengold · 2022-01-09T23:00:29Z

Sounds good, then. Thanks for your contributions to the project, @richardTingle !

richardTingle commented Jan 7, 2022

View reviewed changes

stephengold added this to the Future Release milestone Jan 7, 2022

richardTingle force-pushed the #1734-actions-based-vr-input branch from 909f250 to 120f333 Compare January 8, 2022 11:42

grizeldi self-requested a review January 9, 2022 13:40

grizeldi reviewed Jan 9, 2022

View reviewed changes

jMonkeyEngine#1734 Improve VR javadoc to make it clear that the actio…

b7dfff2

…n based API is only for OpenVr

grizeldi approved these changes Jan 9, 2022

View reviewed changes

stephengold merged commit 4af7aab into jMonkeyEngine:master Jan 9, 2022

richardTingle deleted the #1734-actions-based-vr-input branch January 10, 2022 10:53

Ali-RS modified the milestones: Future Release, v3.6.0 Dec 31, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

#1734 actions based vr input #1735

#1734 actions based vr input #1735

richardTingle commented Jan 7, 2022

richardTingle Jan 7, 2022

grizeldi Jan 9, 2022

grizeldi left a comment

grizeldi Jan 9, 2022

richardTingle Jan 9, 2022

grizeldi Jan 9, 2022

richardTingle Jan 9, 2022

grizeldi Jan 9, 2022

richardTingle Jan 9, 2022

grizeldi Jan 9, 2022

grizeldi left a comment

stephengold commented Jan 9, 2022

richardTingle commented Jan 9, 2022

stephengold commented Jan 9, 2022

#1734 actions based vr input #1735

#1734 actions based vr input #1735

Conversation

richardTingle commented Jan 7, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

grizeldi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

grizeldi left a comment

Choose a reason for hiding this comment

stephengold commented Jan 9, 2022

richardTingle commented Jan 9, 2022

stephengold commented Jan 9, 2022