Auto-enable capabilities needed for eye tracking #4194
Assignees
Comments
wiwei
added a commit
to wiwei/MixedRealityToolkit-Unity
that referenced
this issue
May 8, 2019
It's somewhat of a common thing that folks are hitting that enabling the 'Gaze Input' capability isn't clear (i.e. it's something you have to do as a separate step in VS, and while it's documented in our docs, you don't know it's necessary without reading that or debugging through a number of things). Adding this help text will hopefully get some additional eyes while we wait for Unity to expose this property in its own UI. I considered doing this automatically in our build process, but not everyone uses the UWP build window, and doing automagic stuff can also be really frustrating (because the user never learns that this is required and one day when the magic stops you have to deal with the pain anyway). We already have significant auto-running code complexity in the editor and I don't think that adding addiitional auto running things makes sense. microsoft#4194
|
See #4234 - I opted to make this a help text around the setting, rather than additional magic. I'm concerned that we already have a large amount of automagic in the toolkit, and additional stuff increases this automatic complexity. |
Railboy
pushed a commit
to Railboy/MixedRealityToolkit-Unity
that referenced
this issue
May 13, 2019
It's somewhat of a common thing that folks are hitting that enabling the 'Gaze Input' capability isn't clear (i.e. it's something you have to do as a separate step in VS, and while it's documented in our docs, you don't know it's necessary without reading that or debugging through a number of things). Adding this help text will hopefully get some additional eyes while we wait for Unity to expose this property in its own UI. I considered doing this automatically in our build process, but not everyone uses the UWP build window, and doing automagic stuff can also be really frustrating (because the user never learns that this is required and one day when the magic stops you have to deal with the pain anyway). We already have significant auto-running code complexity in the editor and I don't think that adding addiitional auto running things makes sense. microsoft#4194
|
Thanks for the work @wiwei. However, just adding help text isn't sufficient. We're in Berlin right now and partners are already struggling with this. In fact, one user created a PowerShell script that he could use as part of his automated build just to make sure this capability gets set. I don't feel that help text is enough. We either need to do something to modify the manifest or we need to push to get this surfaced in Unity so we can enable it as we do for all other capabilities. I'm going to open this issue again. We can discuss in next weeks MRTK shiproom (I won't be able to make it this week due to the hacks). Thanks! |
|
We have precedent elsewhere in MRTK, Kurtis pointed me here: We should resolve this one way or the other for all capabilities. |
gejohnst
added a commit
to gejohnst/MixedRealityToolkit-Unity
that referenced
this issue
May 21, 2019
* ColorChanger find mesh if not present, also add method to change to random matieral * Code cleanup, part 1 of N. I'm going through the code to better understand things. As part of reading the code, I'm seeing some occasional typos/issues/areas for cleanup (i.e. stale comments, dead code). Changes below include: - Adding some comments to things like Distorters (some of them aren't used and *could* be deleted, but they seem to be useful to have as a package) - Something are deleted that are not used (i.e. some Interlator* things, some utilites and CalibrationSpace things) - Some code cleanup standardizing on Mathf.Clamp (instead of doing manual if/else checks). * Update the "Use Eye Tracking" properties to be consistent with each other. As microsoft#4195 calls out, the inspector has a term called "Prefer Eye Tracking" but elsewhere it's referred to as "UseEyeTracking." Renaming everything to PreferEyeTracking would actually be more proper (because it's a more accurate description of how things work) However, renaming things to PreferEyeTracking would be a breaking change (where the main value here is consistency/term correctness). It's safer for us to keep things consistent by renaming the serialized property (which supports backwards compat and is not a breaking change) * Bold the fact that the 'Gaze Input' capability is only settable in Visual Studio. This issue comes up repeatedly, but unfortunately isn't something that we have too much control over - there needs to be Unity support added to check this from within Unity, and until that happens, we should try to do all we can to call out in our docs that this is a VS specific step. microsoft#4193 * Add explicit help text around the "Eye Tracking Enabled" property. It's somewhat of a common thing that folks are hitting that enabling the 'Gaze Input' capability isn't clear (i.e. it's something you have to do as a separate step in VS, and while it's documented in our docs, you don't know it's necessary without reading that or debugging through a number of things). Adding this help text will hopefully get some additional eyes while we wait for Unity to expose this property in its own UI. I considered doing this automatically in our build process, but not everyone uses the UWP build window, and doing automagic stuff can also be really frustrating (because the user never learns that this is required and one day when the magic stops you have to deal with the pain anyway). We already have significant auto-running code complexity in the editor and I don't think that adding addiitional auto running things makes sense. microsoft#4194 * Added Android as supportable platform * Update Assets/MixedRealityToolkit/Interfaces/InputSystem/IMixedRealityEyeGazeProvider.cs Co-Authored-By: wiwei <wiwei@microsoft.com> * Adding custom depth offset support and min near fade clamp. * Update the TryGetJointPose docs to accurately describe the set of inputs it can take. These functions only work if you pass in the specific Handedness.Left or Handedness.Right. They should be documented as such. * Adding support for vertex extrusion in world space. * Moving spherical harmonics to vertex shader. * created profile config for mouse -> moved speed setting to profile to detangle controller from pointer * cached mouse profile so we don't access the service on every update * Fix a mismerge in the previous changes * Updates to optimize window * Enable instancing to disable batching for local space triplanar mapping which utilizes object scale. * update teleport sdk components to use the service registry * Draft of updating inspectors * Restored build settings, project settings, quality settings to dev branch defaults * update core input system references to use the service registry * update provider input system references to use the service registry * update input sim service to use service registry * Minor improvements to the channel packer (now texture combiner). * Adding an upgrade path for the scriptable render pipeline. * input system updated to use registry * Adding mesh extrusion example. * sdk cursor scripts now use service registry * Updates to iteration * Added 'create' buttons to profile inspectors. * Added header drawing function to inspector utility, updated profiles * Initial gaze cursor state machine is working * First version of 'DebugStatus' hacked into visual profiler * More cleaning updates * Set camera system type on config profiles * Reverting preferences change * Improved hand sim poses (microsoft#4211) * Use full pos+rot poses for simulated hand joint interpolation. * Finish json data files for joint poses. * Revert unwanted changes. * Document ArticulatedHandPose class and remove TransitionTo function. * Include customized profiles for the hand recording demo scene to make speech commands work. * Expose property for default file name and add timestamps for hand recording. * Remove spurious meta file. See microsoft#4237 * Handle error case when reading unknown joint name from JSON. * Move gaze pointer state management into helper class * Revert "First version of 'DebugStatus' hacked into visual profiler" This reverts commit 7c6866d. * Remove DebugUtilities.log * Comment SpeechWakeWordRecognized method * Adding support for SRP directional light in player builds. * Fixing edge case. * Improving rotation handle alignment to always point out from the bounding box in addition to being aligned with the positive axis of the edge. * Cleaning up code for PR * New approach: FocusProvider listens for speech keyword "select" command via IMRSpeechHandler instead of adding new wake method. * Add select to default speech commands, remove "toggle diagnostics" and "toggle profiler" since they are not used anywhere. * Fix key binding for select command to be Alpha1 * Only count far pointers if interaction is enabled for the pointer * Adding styles file * Fixing structure * Fix check for value change when setting Vector2 values in interaction mappings (microsoft#4116) * Fix check for value change when setting Vector2 values in interaction mappings * Added unit test * Additional testing * Updated Vector2 unit tests * Disambiguate NuGetCommand task Need to disambiguate for setting up pipeline in internal project. * From Dot comparison to binary float comparison * Remove agent pool from yaml * solver updates for registry * Updating tooltip for rotation handle prefab to document orientation behavior. * Both and Any support in HandJointUtils' FindHand - Updates `FindHand` in `HandJointUtils` to properly return a hand if `Both` or `Any` flags are past in. * pointer scripts updated to use the service registry * Cleaning up code for merge * Fixing default profiles to be customprofile * Cleaning up code for PR * eye tracking demos updated to use service registry * Removed logic from Instance getter * Removing duplicate line We introduced this dupe in this commit. Fixing. microsoft@55557c2#diff-04c6e90faac2675aa89e2176d2eec7d8 * Expose number of near and far pointers, factor out state machine. * Comment NumNearPointersActive, NumFarPointersActive * Updates facade handler to destroy old facades when switching active instances, and when reloading scripts. * Fixing object collection to record changes to transform of children. Also making default orient type None since Face Origin is generally awkare for Plane grid default * Removed an editor ping that could result in a 'ping loop' on recompile * Adding documentation for the MRTK/Standard shader, LWRP upgrade path, and Texture Combiner tool. * Adding color variation to frame rate display. * Add FocusProviderTests * Fix incorrect DestroyImmediate condition * Move GenerateHandPose into utilities class * Updates to the optimization tool * pr feedback updates * Fixed a bug that causes NullReferenceExceptions every frame if the grabbed object is destroyed. * Adds playspace in ensure requirements * Add unit test for gaze pointer state machine * Removed meta file * Removed debug logs * Removed XRSettings asset * Final updates to optimize window * Removed unnecessary camera parenting * Add Unity editor macro for Undo * Removed unnecessary lock object from initializer * revert projectversion change * CR comments - Remove TestXX prefix - Fix stat machien bug - Use service registry * Adding warning window for upgrade. * Fix copyright comment * Rename to GazePointerVisibiiltyStateMachine. * Fix up tests after merge * Simulate voice command 'select' on WMR because it does not trigger otherwise. * Exposing the ability to get and set the scale limits for a bounding box. * Memory stats toggle. * Fixed broken link "loaded additively" Fixed a broken link. * Initial commit for IndeterminateLoader prefab and controller. New Unlit Standard shader * fixed missing material, few more optimizations * fixed missing reverse * changing values to match shell * fixing a few white space nits * Fixed issue where SetSelection was not triggering an update to the radial colleciton. * Fix broken .NET build The default keyword isn't available in the .NET language version that the .NET backend supports. * Delete the non-functional AttachToController Per issue: microsoft#4033 It looks like this solver was added but never fully completed (all of the interesting code was commented out). Rather than ship code that doesn't work, we'll delete it. If we're doing a controller or solvers pass, we can add this back at that time. * Take pointer rotation into account when manipulating bounding box (microsoft#4362) * Adding ability to enable/disable portions of the visual profiler. * Updated mixed reality toolkit tests * Adding options to enable/disable frame info and memory stats in the visual profiler. * Fix for the air-tap gesture on HL1 holographic remoting. microsoft#3909 This issue has been around for a while, and has some interesting history. The underlying cause is a bug in a Unity API (see the code for details), which was fixed at some point in the past, but regressed at some unknown time. With no ETA on when the underlying Unity bug, we're checking in a workaround to our code so that customers can get unblocked. Note that this scopes the fix to only situations where remoting is active. * Prevents multiple initializations on enter playmode * Combined fixes for PR. Clean uped buttons & enums. Fixed incorrect depthbuffersharing func for 2019. * Clean up a dead meta file. It looks like with this change, microsoft@3f99837#diff-0839b16787c3c63d682e567be8e56a92 this file was added unintentionally (probably some leftover file during profile testing) * Removed ExecuteAlways to prevent un-registration in OnDestroy when entering playmode. * Exception raised in call to InteractionManager.GetCurrentReading() There's an underlying Unity bug where calling InteractionManager:GetCurrentReading when there are no sources trips this asssertion. This change addresses a couple of things: 1) Works around this underlying bug by checking to see if numSourceStates == 0 (if so, don't bother calling that API) 2) As a related but not totally related change, making it so that we now support cases where the number of states is > the value of 20. Note that I didn't change the naming of it because the value is public (and thus, it would be a value-less breaking change). When we detect that our array size is not large enough, we will re-allocate (and in a way that should reduce the number of repeat allocations if the numbers are fluctuating a lot). * Adding an asset reference for the instanced color material. * Updating to filter items that aren't .mat files * Use default states in interactable if none are specified / when instantiated from code. * Add PlayMode test to check that Interactable can be added to GameObject at Runtime. * Changed dead zone in MRTK-created input axes to the Unity default (0.19) (microsoft#4367) * Move pipeline steps into templates and add internal Release build * Update Assets/MixedRealityToolkit.SDK/Features/UX/Interactable/Scripts/Interactable.cs Co-Authored-By: Bernadette Thalhammer <36998103+thalbern@users.noreply.github.com> * Mark the speech event data as "used" to prevent double invocation of the handler. (microsoft#4375) * Fixed editor mrtk tests * Cleaning up commented code, removing warnings of unused variables, fixing indents, renaming isProfileLock to be clearer * Simplify enabling/disabling eye tracking feature support with MRTK, Part 1 There are three steps that you need to take to enable eye tracking, and one of them is flipping a checkbox on the GazeProvider. The interesting thing is that even if this is set to true, the gaze provider will end up defaulting back to HL1 behavior anyway, so there's no harm in having the setting default to true - for folks who have HL1, they will still get HL1 behavior. For folks building on HL2, they will have one less step to take. Note that this does increase the work for folks who have HL2 and actively want HL1 behavior, however. * add config profile property to system interface, start adding typed in system interfaces * Update EyeTracking_BasicSetup.md * Revert "Fix NullRef when Interactable created at runtime" * Put interactable states in resources folder and load via Resources.Load * Remove UnityEditor using * Add PlayMode test to check that Interactable can be added to GameObject at Runtime. * Unity made some changes to the interactable states asset * Best-effort handle ReflectionTypeLoadExceptions in controller type assembly enumeration microsoft#4319 The controller type enumeration happens to look through every single assembly that is visible to the editor - it's possible that this certain classes of a module may not be loadable, so in that case we will end up using those best effort. * interfaces should name their typed profile appropriately * update core base classes input profile access * update windows mixed reality profiler activeprofile usage * update services input profile access * update interactable input system and profile access * update demo input system access and fix but in interactable inputsystem property * remove unneeded IsInitilialized check * remove IsInitialized from base cursor * Move the disposal of the dictation and speech providers off the main Unity thread. microsoft#4261 Starting and stopping play mode can be excruciatingly slow because of the time spent waiting for the dictation and speech providers to shut down. These are ultimately due to slowdowns in the Unity wrapped APIs themselves (and ultimately the UWP APIs) * Updated outdated description and axis constraint for 'Select' input action in default speech commands profile. (microsoft#4421) * remove unneeded input device manager constructor param * update per pr feedback * Turning off "Enable Hand Joint Visualization" turns off hand input in Editor microsoft#4268 It turns out that the hand pan interaction script was directly calling into the hand visualization code in order to get joint locations, instead of just calling the hand APIs themselves (which have the exact same data). As I understand, historically the only way of getting this information was through the visualization class. It's been a while that this hasn't been the case, so now it's good to just go straight to the joint data. * Remove Resources.Load and instead load states via new Method States.GetDefaultInteractableStates() * update camera system profile access * update typed profile accessor pattern * Update Assets/MixedRealityToolkit.SDK/Features/UX/Interactable/Scripts/Interactable.cs Co-Authored-By: Kurtis <kurtie@microsoft.com> * Update Assets/MixedRealityToolkit.Examples/Demos/HandTracking/Script/GestureTester.cs Co-Authored-By: Kurtis <kurtie@microsoft.com> * Update Assets/MixedRealityToolkit.Examples/Demos/HandTracking/Script/ToggleHandVisualisation.cs Co-Authored-By: Kurtis <kurtie@microsoft.com> * Fixing broken links * Formatting and typos * Update README_MRTKStandardShader.md * Make method to determine default interactable states virtual * Comment GetDefaultStates method. * Update the docs to mention that set of supported Unity 2018 backend/install options microsoft#3878 We had a question a little while ago about which components should be installed with Unity - this just updates our docs to explicitly state that we support both of the scripting backends on Unity 2018, so people install Unity 2018 can install either (or both!) Also updates the MRTK_UnitySetupPrompt.png image, because we got rid of the "forcing people to IL2CPP" dialog. * Remove Debug.Log * Add 'component governance' task for release signing * Fixed numbers in enumeration * Eye targeting is not working with near hand pointers microsoft#4431 In a previous change (microsoft#4270) a gaze provider state machine was added to bring the head gaze pointer behavior in-line with how the shell works. This had the side effect of also impacting eye-gaze state. This change brings the old eye-gaze behavior back by updating the state machine to be aware of the type of gaze being provided. Also added a bunch of tests for this new thing, along with tests that show that transitions between the two (for example, when the eye tracking system gains tracking or loses tracking) works well. * Remove unneeded configuration profile references * Visual fit & finish. * Make sure sim hands are initialized when starting in persistent mode. (microsoft#4495) * Extend the Files utility to support non-core modules and loading. (microsoft#4476) * Extend the Files utility to support non-core modules and loading. * Remove empty hash sets after removing folders. * Experimental commit to see how to enable .NET CI * Add a fix for issue microsoft#4405 in response to CR feedback.
|
A friend at Unity just informed me that they were able to get the GazeInput capability surfaced in Unity 2019.3.
It would be great to finally update the eye tracking service to automatically turn on the capability if running on 2019.3+ |
|
Let me check if we are also getting this fix into Unity 2018.4 |
Yoyozilla
added
External
|
@Yoyozilla any new info on ingestion into Unity 2018.4. If Unity doesn't surface the API, that will make it trickier. FYI @jbienzms and all, the MRTK build window can auto-add the Gaze Input capability since it builds the manifest itself: |
|
Thanks @Troy-Ferrell. The stated goal was that whenever any core MRTK service was "enabled" it should also enable any UWP capabilities that are dependencies for it to function. Eye Gaze was the only service that was not able to follow this rule because it was the only one that didn't have its related capability surfaced through the Unity editor. While I appreciate the check in the build window, I don't believe that helps enforce the goal of having each service ensure its related capabilities are enabled. |
This was referenced Oct 15, 2019
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the problem
Work is being done in the MRTK to try and automatically enable the app capabilities (permissions) necessary for services to work. On HoloLens 2, a special capability called 'Gaze Input' is needed, but that capability is not currently being set when eye tracking is enabled.
This is mainly due to #4193 which is currently out of our control, but it could be possible to work around this issue with a post-build step.
Describe the solution you'd like
Ideally we should work with Unity to address the missing capability from the interface and then turn that capability on as we do with all other services. If that is not possible, we should implement it as a post-build step.
The text was updated successfully, but these errors were encountered: