Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Automatically calibrate cursor position to match head tilt (Bonus: generalizes head tilt mode) #45

Open
1j01 opened this issue Apr 13, 2024 · 1 comment
Labels
affects:desktop-app Affects the Electron app affects:web-library Affects usage of the tracky-mouse.js library enhancement New feature or request

Comments

@1j01
Copy link
Owner

1j01 commented Apr 13, 2024

Currently, the user's face is detected in order to place tracking points, and optical flow is then used to track these points on your face, and only the optical flow influences the final cursor movement.

In this existing scheme:

  • Cursor movement is directly affected by the camera's perspective.
  • The head's tilt is not used directly.
  • Translating your head has as much of an affect as tilting it (or more).
  • When the cursor runs into the edge of the screen, it stops, whereas your head can continue of course, and this naturally leads to a way of "calibrating" on the fly to a more comfortable range of head movement.
  • However, you have to think about it a bit.

What if we auto-calibrated based on the head tracker's face orientation, perhaps making adjustments only during otherwise-detected movement?
Like a perpetual motion machine's secret magnetic "kick", or a magician's slight of hand, it would subtly adjust the mouse position so that it ends up at the edges of the screen when tilted a certain amount, and centered when facing forward.

Drawbacks:

  • It may not actually feel subtle in practice, when strong enough to work, or may feel subtly wrong and degrade the experience overall in a way that's hard to put your finger on, since it's like a magic trick.
  • Tilt is relative to camera position! So this auto-calibration would need its own calibration.
  • This may just be overcomplicating things.

Formula:
Assuming the tilt can be normalized, or assuming the camera and screen and face are directly in line with each other,
a formula for the other part might be fairly simple, something like
adjusted_x = x + (abs(delta_x) * auto_calibrate_strength * (x_implied_by_tilt - x)
where x_implied_by_tilt is the x position on the screen that would be mapped purely from the head tilt
where delta_x = x - x_previous
where x is the latest x position from optical flow tracking
It's probably a little more complicated, like maybe the delta_x factor should be raised to some power, or clamped, etc.

On the plus side, it could be made so that with auto_calibrate_strength = 1, it purely uses the head tilt, so a separate head tilt mode wouldn't be needed. (Again, I'm not sure about the delta_x part in regard to this.)

@1j01 1j01 added enhancement New feature or request affects:web-library Affects usage of the tracky-mouse.js library affects:desktop-app Affects the Electron app labels Apr 13, 2024
@1j01
Copy link
Owner Author

1j01 commented Apr 13, 2024

Maybe it should only make adjustments in the direction that you're already moving,
i.e. if sign(delta_x) = sign(x_implied_by_tilt - x)
I think that would help with the sleight of hand, although I'm still not sure if sleight of hand is desirable.

If nothing else, it would be an interesting experiment!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
affects:desktop-app Affects the Electron app affects:web-library Affects usage of the tracky-mouse.js library enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant