Mouse move to absolute coordinates

[xcarcelle]

Dear @csash7 thanks for your great project !
Have you thought about adding absolute coordinates to your library with specific HID Descriptor and Report ?
I would be willing to help and test on android/ios. Cheers.

[csash7]

Hi @xcarcelle,
yes, you can achieve that functionality by changing the report map.

In the file 'mouse.h',
Change the item of report map from 'INPUT(1), 0x06' -> 'INPUT(1), 0x02'
This changes the input from relative coordinates to absolute coordinates.
Remember, you can have either-or functionality as the report map is sent only once.
Please let me know your test results.

[xcarcelle]

Hi @csash7 thanks for your reply and your interest.

• changed the INPUT to be absolute (0x02) and it is not working on Android (the mouse pointer does not even point-out) and not working on ios14 (as w/ the 0x06 INPUT) - but in ios14 the BLE is not detected in the BLE scan result even with the 0x06 INPUT
• we might need to change the HID Report to be a pointer supporting 16bits for (x, y) to be sent to the host
• can we test/adapt to use the same HID Descriptor than this project (https://github.com/jonathanedgecombe/absmouse/blob/master/src/AbsMouse.cpp#L5) and have (x ,y) on uint16_t size ?
  Thanks a lot !

[xcarcelle]

Dear @csash7 I have been more tests today with Android 8.0 :

• changing the HID Report to have a logical minimum larger, still in relative move (if INPUT = 0x02 the mouse is not recognized)
• changing the size of the (x, y) coordinates sent to the host in the BLE HID payload
  It seems it allows the mouse to move more than 127 but like move(5000, 0) allows a move of half the screen with LOGICAL_MAXIMUM = 0x1027 (10000). Attached are the 2 files I am modifying if you want to make tests or feedbacks somehow. Cheers.
  mouse.cpp.txt
  mouse.h.txt

[csash7]

Dear @xcarcelle ,
I agree that if we change the LOGICAL_MAXIMUM value, we can move to the cursor to a higher coordinate value (The value accepted by the central device depends upon its screen resolution). And LOGICAL_MINIMUM can be set to 0, as absolute coordinates start from (0,0).
I have tested these changes on a windows PC and it is working as expected. But as you've mentioned, in android even after pairing I am unable to view the cursor, and in ios, the device is not recognized.

[xcarcelle]

Dear @csash7
thanks for your feedback on these tests and the confirmation on the current test with LOGICAL_{MIN, MAX} to reach wider zone of the screen. Now we need to understand :

• How to make Android/iOS understand the INPUT = 0x02 (absolute)
• How to move in the negative range in relative mode <-- Have you tried ?
  I wish I could have a tool on the host (Android/iOS) to debug the host side by seeing how the absolute INPUT is dealt in the host.
  Are you aware of external touchscreen for Android/iOS that we can mimick for HID Report ?

[xcarcelle]

@csash7 : I have made a fork of your repo to push some tests of HID Report for larger movements of mouse and then for absolute test. Will keep you updated.
xcarcelle@d39d37b

[Nesh108]

@csash7 Yeah, is it normal that 0x02 on mobile doesn't seem to get recognized? But using 0x06 (for relative) works without any issue. I wonder if there is something else needed 🤔

[street-grease-coder]

I agree this would be super useful, but only if you also get the screen resolution over the BLE interface somehow - I have not figured out how to that (yet), let me know in case this is obvious...

[xcarcelle]

@street-grease-coder : did you make any progress on your end ? For screen resolution feedback from the host maybe one can use hm-10 ble/spp uart dongle or ssp profile of esp32 board to connect to the host with a dedicated piece of app code ?

[street-grease-coder]

I noticed the mouse interface in micropython allows for easy access of absolute coordinates. I opted to go that route, away from C/C++/ino code, maybe I'm wrong there. It seems to be possible to replace the default HID file to make this work though

[xcarcelle]

@street-grease-coder : have you already made a test with a board and micropython abs mode ? I would like to use that route to if possible as well.

[street-grease-coder]

no, im my hands tedious to set up with external libraries compared to the Arduino environment. first make sure that the speed is fast enough

[street-grease-coder]

reporting back, micropython library examples are not available as of now for a mouse. Basic functionality is implemented on level of a bluetooth library and mouse library (but the combination is missing). In circuitpython I could also not find easily-accessible example code to achieve this. As it stands, this library seems to be the only feasible way for beginners to setup anything like a mouse or keyboard (at least on an ESP32, which is the platform I develop on atm)

[xcarcelle]

@csash7 : I have tested this code NicoHood/HID#123 (comment) with an ArduinoMicro and a Android (7.0) device and it looks great with the digitizer HID Report, I am trying to modify this library to use this digitizer with the identifier/touch/coordinates HID Report to have a clean absolute digitizer over BLE mouse. What do you think ?

[henrygab]

reporting back ... In circuitpython I could also not find easily-accessible example code to achieve this.

CircuitPython has bluetooth HID keyboard and mouse support. Here's a full guide. It's also not terribly difficult to use custom descriptors ... just replace the parameterless HIDService() call with one that provides the overrides.

Here's some snippets using CircuitPython 8 on a Circuit Playground Express:

Expand for code

import board
import keypad

import adafruit_ble
from adafruit_ble.advertising                   import Advertisement
from adafruit_ble.advertising.standard          import ProvideServicesAdvertisement
from adafruit_ble.services.standard.hid         import HIDService
from adafruit_ble.services.standard.device_info import DeviceInfoService

import adafruit_hid
from adafruit_hid.mouse                         import Mouse
# from adafruit_hid.keyboard                      import Keyboard

# This adds a service to the root BLE radio, advertising the device's existence
device_info = DeviceInfoService(
    manufacturer="SampleMan",
    software_revision=adafruit_ble.__version__,
    # model_number: Optional[str] = None,
    # serial_number: Optional[str] = None,
    # firmware_revision: Optional[str] = None,
    # hardware_revision: Optional[str] = None,
    # service: Optional[_bleio.Service] = None,
    )

# Default HID descriptors expose keyboard, mouse, and a consumer control device
hidService = HIDService()

# initialize Bluetooth (and BLE) radio
ble = adafruit_ble.BLERadio()

# Create the "friendly" wrapper objects for the mouse and keyboard
m = Mouse(hidService.devices)
# k = Keyboard(hidService.devices)


while True:
    if ble.connected:
        m.release_all()
        m.move(x=-8000, y=-8000)
        m._send_no_move() # hack to ensure each movement is seen as distinct by certain OS
        m.move(x=300, y=100)
        m._send_no_move() # hack to ensure each movement is seen as distinct by certain OS
        m.press(Mouse.LEFT_BUTTON)
        m.move(x=100, y=300)
        m._send_no_move() # hack to ensure each movement is seen as distinct by certain OS
        m.release(Mouse.LEFT_BUTTON)
        sleep(4)

---

From what I could find, Apple (iOS) does not support either USB nor bluetooth digitizer HID input. I wish it did, so please let me know if I'm mistaken. To have accurate absolute positioning without significant hacks would be great!