Add support for input and gestures for hand controllers #552
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
.../InputSystem/Controllers/Hands/BaseMixedRealityHandControllerDataProviderProfileInspector.cs
Outdated
Show resolved
Hide resolved
...es/InputSystem/Controllers/Simulation/SimulatedHandControllerDataProviderProfileInspector.cs
Show resolved
Hide resolved
This was referenced May 1, 2020
...k.core/Runtime/Providers/Controllers/Simulation/Hands/SimulatedHandControllerDataProvider.cs
Show resolved
Hide resolved
XRTK-Core/Packages/manifest.json
Outdated
| @@ -11,7 +11,7 @@ | |||
| "dependencies": { | |||
| "com.unity.package-manager-ui": "2.1.2", | |||
| "com.unity.textmeshpro": "2.0.1", | |||
| "com.unity.xr.legacyinputhelpers": "2.0.8", | |||
| "com.unity.xr.legacyinputhelpers": "2.1.3", | |||
There was a problem hiding this comment.
We sure we want to do this?
Last time I updated this package we had some errors
There was a problem hiding this comment.
@StephenHodgson do you remember what happened? What the errors were, so I can verify.
…feature/hand-input-and-poses # Conflicts: # Submodules/Lumin # Submodules/Oculus # Submodules/SDK # Submodules/WindowsMixedReality
…github.com/XRTK/XRTK-Core into feature/hand-input-and-poses # Conflicts: # Submodules/SDK
…ision # Conflicts: # Submodules/SDK
…ision # Conflicts: # Submodules/SDK
…github.com/XRTK/XRTK-Core into feature/hand-input-and-poses # Conflicts: # Submodules/Lumin # Submodules/SDK
…K/XRTK-Core into feature/hand-input-and-poses
* Update ColliderExtensions.cs * Update CollisionExtensions.cs
|
(minor) Cannot grab both dumbbells in both hands. Only one hand works Replication Steps:
Expected Behaviour: |
|
(minor) Table material makes hands not very visible (both are mostly white) Ideally, the table should be a darker colour material to improve the visibility of hands
|
…feature/hand-input-and-poses
|
Tested the new updates, absolutely loving the redesigned stations, it's a lot more fluid. A side note, is it would be nice if the inner part of the button was out more, so the press is visibly more present. (big squishy buttons) |
… into feature/hand-input-and-poses
…feature/hand-input-and-poses # Conflicts: # Submodules/Lumin # XRTK-Core/Packages/manifest.json
…K/XRTK-Core into feature/hand-input-and-poses
SimonDarksideJ
approved these changes
Sep 22, 2020
XRTK-Build-Bot
pushed a commit
that referenced
this pull request
Dec 25, 2020
* Delete baked lighting settings for base * Fix ControllerPopupWindow NRE * Implement internal is grabbing update * Disable hand meshing by default and assign tracked poses to test profile * Add IsValidCollision extension to colliders * Feature/pni change requests (#577) * Update ColliderExtensions.cs * Update CollisionExtensions.cs * updated submodules * updated sdk * updated sdk * Cleanup namespace * Better fallback pointer pose calculation * Introduce hand rendering mode * Implement runtime rendering mode switch * Update submodules * Start new hands examples room * Add canvas UI and 3D Buttons station * Show button stations result * Implement set rendering mode station * Added more stations * Update submodules * updated wmr checkout * updated sdk * Update submodules * Add additive scene loader for hands demo scene * Restore base scene * Add simple physics demo station * Cleanup namespace * Cleanup hand examples a bit * Update stations * Implement runtime bounds mode selection * Update submodules * Update Ultraleap checkout * Update physics station * Update submodule * Update submodule * Fix UWP certficate error * Moved examples to examples repo * reset editor window options for hands * Implement pinch strength * Add pinch strength to hand controller interface * Update submoduules * Update SDK * Use array instead of list for tracked poses * Restructure pose recognition code * Improved variable naming * Add root pose to hand data * Set root pose for simulation * Fix joint rotations * Update SDK * Remove base converter and introduce Post Processor * Update data provider to use post processor * Optimize recognition * Introduce grip strength * Enable setters for post processor * Minor refactoring in post processor * Removed NONE joint * Rename joints * Update submodules * Extend default hand mappings * Sync simulation profiles * Reset popup window options for hands * Hand input and poses change request (#616) * Cleaned up RecordedHandJoint(s) to remove reference values removed null unity object lifetime bypass added better exceptions when starting simulated hand controller data provider and Simulated hand data converter * fixed missing line * updated sdk checkout * Implement grip pose * Adjust processor to joint enum renamings * updated sdk checkout * Introduce hand finger enum * Rename pose recognizer and implement frame throttle * Minor docs update * Implement finger curls * Add Oculus (Desktop) package back in for now * Disable debug log for grip calculations * updated tracked pose processor compare (#617) aesthetic and spell checking in hand controller Co-authored-by: Dino Fejzagic <dino.f@live.de> * Implement IMixedRealityHandController.TryGetFingerCurlStrength * Update SDK * Disable recognition for now * Add index curl constraint to isgripping detection * updated sdk checkout * updated SDK * updated sdk checkout * Implement recorded pose baking and update inspector * Update SDK * Remove handedness from hand data * Update Oculus * Fix baked data not persisted * Update SDK * Fix pose recognizer handedness issue * Improve pose processor * Fix IsPointing set to false when pinching * Update submodules * Update examples checkout * Do not pass tracked pose scriptable object outside of data provider * Update examples * Update examples * Fix potential NRE Co-authored-by: Stephen Hodgson <hodgson.designs@gmail.com> * Fix potential casting error Co-authored-by: Stephen Hodgson <hodgson.designs@gmail.com> * Fix formatting * Use for instead of foreach * Revert metro package version * Rename HandControllerPoseDefinition to HandControllerPoseProfile * Revert to ReordableList for tracked poses * Change HandData to be a struct * Update submodule * Update example module * Fix invalid simulated hand position when camera rotation is changed * Fix generic pointer pose invalid when camera rotated * Remove redundant properties in hand mesh data * Update submodules * Only update hand state if hand is tracked * Remove obsolete pose extensions * Introduce hand utilities for pose estimation * Update submodules * Implement TryGetHandMeshData * Remove HandData input event * Update submodule * Update submodules * Implement relativeTo paramter for TryGetJointPose * Fix IsPointing * Use TrackingState instead of bool for tracking state * Fix missed occurence of IsTracked * Update Oculus * Fix pointer pose * Update Oculus * Fix fallback pointer pose * Fix pose recognizer left/right issue * Update submodules * Remove tracked pose mapping (coming back later) * Fix hand velocity calculation * Fix local to world joint translation * Fix simulation mappings * Fix IsPointing after teleport * Update submodule * Fix simulated hand not reset on tracking loss * Fix simulated hand position after teleport * Update SDK * Update SDK * Update examples module * fixed some typos * updated oculus checkout * Update example checkout * Reset hand controller options window * Update submodules * fixed submodule checkouts * Remove tracked poses from hand data provider interface * Cache bounds arrays * Allow a bit more tolerance wit IsPointing * Update submodules * Update example * Update modules * Update submodules * Update oculus * Update WMR * Clean up gizmo meshes * Remove duplicate vertices in gizmo meshes * Update submodules and wmr package * Only apply default head height if not managed by device * Update submodules * Introduce IHandDataPostProcessor * Update submodules * Update WMR submodule Co-authored-by: Stephen Hodgson <hodgson.designs@gmail.com> Co-authored-by: Simon (Darkside) Jackson <darkside@zenithmoon.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
XRTK - Mixed Reality Toolkit Change Request
Overview
This pull request is the next iteration on the hand tracking feature, making it actually useful. It implements a lot of refactoring to improve the overall architecture and data flow based on recent learnings.
Data Flow / New Platforms
All hand data that platform APIs provide is now being processed in the respective hand data converter. Introduced
BaseHandDataConverterwhich contains base and fallback conversion logic for hands, e.g. it will provide a pointer pose if the platform did not provide it. The essential piece of data for hands is nowHandDatamore than ever, which the converters generate. So for each platform there is aPlatformHandControllerDataProvider + PlatformHandDataConverterpair taking care of preparing data for the agnostic hand controller.So for a new hand platform "all" that needs to be done is implementing the data provider and if applicable the converter.
Reduced simulation specifics
The only specifics
SimulatedMixedRealityHandControllernow has are the default interaction mappings it needs for the simulated hand to be controlled in editor. It now directly derives fromMixedRealityHandController.BaseHandControllergot removed sinceMixedRealityHandControlleressentially is the base for any hand. I'll look into ways to get rid ofSimulatedMixedRealityHandControllercompletely in a next iteration of hands.Near / Far Interaction
Near and far interaction using hands is now supported using
Also added new APIs for
Physics
The
HandPhysicsEnabledsetting can now be changed at runtime to enable or disable the physics components for hands using the new setter forIMixedRealityHandControllerDataProvider.HandPhysicsEnabled. This can be used to e.g. temporarily enable physics interactions and then disable them again. Also physics is now disabled by default.The
IMixedRealityHandControllerDataProvider.BoundsModeandIMixedRealityHandControllerDataProvider.UseTriggerssettings are now runtime settable as well.Tracked Poses
All hand profiles now have a new setting "Tracked Poses". These are for one the poses that are available for hand simulation and secondly these are the poses that are being tracked/recognized by the hands system. This is a work in progress feature that will enable user's to define their custom poses they recorded and have input actions raised when those poses are recognized. For now it drives simulation only. Of course each platform may override the global setting here.
Rendering Mode
The former "Hand Meshing Enabled" setting in hand profiles got replaced by a rendering mode setting. The setting defaults to
Joints, other options areNonefor when you just want "Ghost Hands" andMeshfor hand mesh visualization. If the platform does not provide hand mesh dataMeshwill fallback toJoints. Of course each platform may override the global setting here.Submodule Changes