Feature Request - absolute mouse

[jrsmile]

Would it be possible to adapt the mouse hid class to send absolute coordinates like a touchscreen?

[benjaminaigner]

Dear @jrsmile ,
should be possible.
I cannot estimate if and when we will include this, joystick support and stable connections have a higher priority.
If you want to try it on your own:
This is the HID report map:

esp32_mouse_keyboard/main/ble_hid/hid_device_le_prf.c

Line 44 in f7ab88a

static const uint8_t hidReportMap[] = {

There we need to replace the relative mouse by an absolute positioned.
Here is an example for this report map: https://forums.obdev.at/viewtopic.php?t=2559

[jrsmile]

i have seen a map here: HERE unfortunately i know to little to modify the code the correct way.
would the link help with the implementation?

[ChrisVeigl]

hi,
we added absolute mouse position and joystick reports to our HID-actuator several years ago - it worked. mybe this is helpful:

https://github.com/asterics/AsTeRICS/blob/master/CIMs/HID_actuator/Mouse_Keyboard_Joystick/Descriptors.c
https://github.com/asterics/AsTeRICS/blob/master/CIMs/HID_actuator/Mouse_Keyboard_Joystick/Descriptors.h

[benjaminaigner]

@ChrisVeigl , @jrsmile THX for the suggestions, based on this I will implement an additional report with an absolute mouse.

[jrsmile]

thank you very much, looking forward to it.
i will use it to build a firmware test tool.
best possible outcome would be a keyboard,mouse,touchscreen device and a ble serial device.
being able to test with a mobile phone as gui via ble a machine that has a bluetooth dongle attached is my target :-) i wonder if pairing with two devices is possible. if not usb serial will be suficient too.

[benjaminaigner]

Ah, sounds nice!
I've started a new branch with the absolute mouse included.
If there is time this week, I will test it.

Regarding the serial device:
This is a feature we won't implement by ourselves, but we could need that :-)

Pairing with multiple devices:
Pairing is possible, connecting should be too.
I think it is necessary to store different HID connection IDs for multiple devices, but this is also something thats not on our agenda.

[benjaminaigner]

@jrsmile

In the branch "absoluteMouse", there is a basic implementation of an absolute mouse report.
I've some input on Android, but I think it is not completely correct.

Could you please test it and report back?

[zuozhehao]

Is there any progress change.

[benjaminaigner]

@zuozhehao

I have a branch with the absolute mouse descriptor for testing.
Please try it out and report back. If everything is working, I'll merge the changes and make it available via config files.

[xcarcelle]

@benjaminaigner @zuozhehao thanks for the absolute mouse branch/progress and any help needed on test or code i will try to support this issue.

• I have tested the last branch commit on an Android tablet running 9.0 and it acts weird (like keys from a keyboard pushed very fast) when trying the absolute command from the uart in esp-idf (running make monitor flash)
• The mouse click also does not seem to work
  Do you have some unit tests (devices, os, test) you want to run to validate this current state ?

[benjaminaigner]

Dear all,
I'm sorry this feature takes so long.
Absolute mouse is not our top priority & due to COVID my workload regarding lectures is rising dramatically.

Still, I try my best to support you and see what we can do about the absolute mouse.

Current status:
@xcarcelle I've seen the same behaviour on my Android device.

I wanted to try my "real" absolute mouse (touchscreen) on iOS for @zuozhehao , but iPads do not accept USB accessory with 100mA current. Unfortunately I have no USB-C OTG cable available to test it on Android.

I will keep you updated, today I can spare some time for further testing.

[benjaminaigner]

OK, I tested following absolute HID descriptors:

https://www.codeproject.com/Articles/1001891/A-USB-HID-Keyboard-Mouse-Touchscreen-emulator-with

Variant 0: works on Debian, but not on iOS / Android
Variant 1: works on Android, but not on Debian / iOS
Variant 2: didn't get it to work on all platforms

https://www.schoeldgen.de/avr/

(USB Interface for Tektronix 4957 tablet using V-USB)
Works on Debian, not on iOS/Android
(most promising for iOS, because the Assistive Touch indicator went from grey to black)

Next try: remove all other reports except the absolute mouse, maybe this helps for iOS.
Does anybody has an alternative absolute mouse descriptor to test?

[xcarcelle]

@benjaminaigner thanks for the update,
i will test android and ios on multiple devices tomorrow. How do you test variant0/1/2 from https://www.codeproject.com/Articles/1001891/A-USB-HID-Keyboard-Mouse-Touchscreen-emulator-with with your absolute branch ? Can you confirm the place to replace HID Descriptors. For Tektronix 4957, can I test something at this point with an iOS device ?
Keeping posted to further instructions.

[xcarcelle]

@benjaminaigner I have tried replacing the "touch_device=1" HID Descriptor from codeproject into this block : HID Descriptor Mouse Absolute but it did not work for abs move on Android (8.0 and 9.0) tablets and iOS. Could you tell me how did you replace the HID Descriptor in your code so I can follow-up tests ? Regards.

[benjaminaigner]

Dear @xcarcelle ,

for testing, I replaced following parts accordingly (in master branch):

esp32_mouse_keyboard/main/esp_hidd_prf_api.c

Line 135 in ffc6d08

void esp_hidd_send_mouse_value(uint16_t conn_id, uint8_t mouse_button, int8_t mickeys_x, int8_t mickeys_y, int8_t wheel)

change the bytestuffing for the mouse report here (some HID descriptors require more than 5 Bytes; change HID_MOUSE_IN_RPT_LEN accordingly)

Place the HID report map here:

esp32_mouse_keyboard/main/hid_device_le_prf.c

Line 145 in ffc6d08

0x05, 0x01, // Usage Page (Generic Desktop)

Please be careful to place the report ID in any report map to be tested:
0x85, 0x03, // Report Id (3)

For testing, I placed this source into the main file:

//at the beginning of uart_console_task:
static uint16_t absX = 0;
static uint16_t absY = 0;

//in the switch(character) block
case '0':
	ESP_LOGI(CONSOLE_UART_TAG,"abs mouse: l");
	esp_hidd_send_mouse_value(hid_conn_id,(1<<0),absX,absY,0);
	esp_hidd_send_mouse_value(hid_conn_id,0,absX,absY,0);
	break;
case '1':
	absX = 0;
	absY = 0x2710;
	ESP_LOGI(CONSOLE_UART_TAG,"abs mouse: %d / %d",absX,absY);
	esp_hidd_send_mouse_value(hid_conn_id,0,absX,absY,0);
	break;
case '2':
	absX = 0x2710/2;
	absY = 0x2710;
	ESP_LOGI(CONSOLE_UART_TAG,"abs mouse: %d / %d",absX,absY);
	esp_hidd_send_mouse_value(hid_conn_id,0,absX,absY,0);
	break;
case '3':
	absX = absY = 0x2710;
	ESP_LOGI(CONSOLE_UART_TAG,"abs mouse: %d / %d",absX,absY);
	esp_hidd_send_mouse_value(hid_conn_id,0,absX,absY,0);
	break;
case '4':
	absX = 0;
	absY = 0x2710/2;
	ESP_LOGI(CONSOLE_UART_TAG,"abs mouse: %d / %d",absX,absY);
	esp_hidd_send_mouse_value(hid_conn_id,0,absX,absY,0);
	break;
case '5':
	absX = absY = 0x2710 / 2;
	ESP_LOGI(CONSOLE_UART_TAG,"abs mouse: %d / %d",absX,absY);
	esp_hidd_send_mouse_value(hid_conn_id,0,absX,absY,0);
	break;
case '6':
	absX = 0x2710;
	absY = 0x2710/2;
	ESP_LOGI(CONSOLE_UART_TAG,"abs mouse: %d / %d",absX,absY);
	esp_hidd_send_mouse_value(hid_conn_id,0,absX,absY,0);
	break;
case '7':
	absX = absY = 0;
	ESP_LOGI(CONSOLE_UART_TAG,"abs mouse: %d / %d",absX,absY);
	esp_hidd_send_mouse_value(hid_conn_id,0,absX,absY,0);
	break;
case '8':
	absX = 0x2710/2;
	absY = 0;
	ESP_LOGI(CONSOLE_UART_TAG,"abs mouse: %d / %d",absX,absY);
	esp_hidd_send_mouse_value(hid_conn_id,0,absX,absY,0);
	break;
case '9':
	absX = 0x2710;
	absY = 0;
	ESP_LOGI(CONSOLE_UART_TAG,"abs mouse: %d / %d",absX,absY);
	esp_hidd_send_mouse_value(hid_conn_id,0,absX,absY,0);
	break;

Yesterday night, I tried my 13" touchscreen with Android, connected via USB.
It works flawlessly for touch input, but there is no cursor available.
Nevertheless, I cannot try this device with iOS, because the iPad refuses to use this USB device ("accessory requires too much power"). It would be very interesting to see if this device works with iOS (would be a good start for our own device).

[xcarcelle]

Dear @benjaminaigner ,
thks a lot for your update, first the step i used to test your proposal :

• using the master branch
• replacing the HID Descriptor with the one from codeproject (TOUCHDEVICE=1) as is
• not changing the HID_MOUSE_IN_RPT_LEN at this point
• adding the console command for abs move
  Then the test using (esp32 board + Huawey Mediapad T5 Android 8.0) is that the mouse moves but only in a small up-left portion of the tablet screen, the move(0, 0) seems to work but the other values does not move on the way to half/full-screen position.
  I did sent you a video of the current test. Will try with iOS tablet over BLE.
  Do I miss something for the half/full screen moveto values in the code to enable the full screen ?
  Regards.

[xcarcelle]

@benjaminaigner : I was wondering if you had more tests with master or absolute branch to test the different HID Descriptors. I want to test different descriptors with different devices but I want to be sure to replicate your methodology to push significant results. Cheers.

[benjaminaigner]

Dear @xcarcelle
I'm sorry that my reply time is so high, currently we have a high workload of preparing online classes for Corona-safe lectures :-).

Regarding your question with your test:

• absY = 0x2710/2; should be half the screen size, BUT: this is only valid, if you use 0x2710 as value in the HID descriptor (Logical Maximum (10000))
• In absolute mouse settings, the host device maps the logical max value to the current screen size. So if you send the logical maximum value as coordinate, the mouse should be in the other corner (X/Y: max-> bot /right corner; X max, Y 0 -> top/right corner; and so on).

I had still no time until now to extract the HID descriptor of my USB-Touchscreen, which works on Android & Linux (can't be tested on iOS because the Lightning to USB adapter does not provide sufficient current).

[xcarcelle]

@zuozhehao : have you been able to do some tests on your side with android/ios ?

[xcarcelle]

Dear @benjaminaigner I guess the workload is crazy w/ the current Covid situation and remote session of classes, good luck !
I will continue more tests tomorrow and I have a set-up with ios and Lightning-to-USB_and_power cable. Which model of USB-Touchscreen are you trying ?
I wish I could test on Android first, if you have some ready-to-test code I am willing to build/test it. Cheers and good luck !

[benjaminaigner]

Dear @xcarcelle ,
I finally got time to dump the HID descriptor from the touchscreen. It is a
Iltek ICI2511 controller, USB: 222a:0001
I was quite surprised, this HID map is really big.
touchscreen_HIDMap.txt

I've uploaded the raw dump from USBlyzer, which needs re-formating into a C-array.
If i can spend time, I will put this report into our project (please be patient, could take a few days).

[benjaminaigner]

At least I found the time to put into a C-array...
If anybody can test it, I would be really thankful.
But be careful, this report descriptor needs heavy reformating to put the data in the necessary format (report IDs, mouse clicks & X/Y data).
reportMapDigitizer.txt

[benjaminaigner]

This is the corresponding dump of HID data:

• Press 1 finger
• Release 1 finger
• Press 3 fingers
• Release 3 fingers

As this seems to be a multitouch display (1B for button + 4B for X/Y repeat 10x times), the HID report map
is a little bit messed up. USBlyzer cannot assign the single touch values correspondingly (everything is mapped many times to the first touch input).
Contact count is used to set the number of input points.
I don't know what the scan time means, but it seems to be an incrementing counter...

Sry for the format of this file, quick & dirty copy'n'paste...

touchscreen_dumps.txt

[benjaminaigner]

Following problems with this HID report map:
BLE MTU limits the packet sizes for HID payload to 20B. The original frame size was 64B.

I've tried to strip down this report map to following result, but currently not functioning on Android (if anybody can spot the error, I would be really happy):

0x05, 0x0D, // Usage Page (Digitizer)
0x09, 0x04, // Usage (Touch Screen)
0xA1, 0x01, // Collection (Application)
	0x85, 0x03, // Report ID (3)
	0x09, 0x22, // Usage (Finger)
	0xA1, 0x02, // Collection (Logical)
		0x05, 0x0D, // Usage Page (Digitizer)
		0x95, 0x01, // Report Count (1)
		0x75, 0x06, // Report Size (6)
		0x09, 0x51, // Usage (Contact Identifier)
		0x15, 0x00, // Logical Minimum (0)
		0x25, 0x3F, // Logical Maximum (63)
		0x81, 0x02, // Input (Data,Var,Abs,NWrp,Lin,Pref,NNul,Bit)
		0x09, 0x42, // Usage (Tip Switch)
		0x25, 0x01, // Logical Maximum (1)
		0x75, 0x01, // Report Size (1)
		0x95, 0x01, // Report Count (1)
		0x81, 0x02, // Input (Data,Var,Abs,NWrp,Lin,Pref,NNul,Bit)
		0x75, 0x01, // Report Size (1)
		0x95, 0x01, // Report Count (1)
		0x81, 0x03, // Input (Cnst,Var,Abs,NWrp,Lin,Pref,NNul,Bit)
		0x05, 0x01, // Usage Page (Generic Desktop)
		0x75, 0x10, // Report Size (16)
		0x55, 0x0E, // Unit Exponent (-2)
		0x65, 0x11, // Unit (SI Lin: Length (cm))
		0x09, 0x30, // Usage (X)
		0x26, 0x00, 0x40, // Logical Maximum (16384)
		0x35, 0x00, // Physical Minimum (0)
		0x46, 0x15, 0x0C, // Physical Maximum (3093)
		0x81, 0x42, // Input (Data,Var,Abs,NWrp,Lin,Pref,Null,Bit)
		0x09, 0x31, // Usage (Y)
		0x26, 0x00, 0x40, // Logical Maximum (16384)
		0x46, 0xCB, 0x06, // Physical Maximum (1739)
		0x81, 0x42, // Input (Data,Var,Abs,NWrp,Lin,Pref,Null,Bit)
	0xC0,  // End Collection
	0x05, 0x0D, // Usage Page (Digitizer)
	0x09, 0x56, // Usage (Scan Time)
	0x55, 0x00, // Unit Exponent (0)
	0x65, 0x00, // Unit (None)
	0x27, 0xFF, 0xFF, 0xFF, 0x7F, // Logical Maximum (2147483647)
	0x95, 0x01, // Report Count (1)
	0x75, 0x20, // Report Size (32)
	0x81, 0x02, // Input (Data,Var,Abs,NWrp,Lin,Pref,NNul,Bit)
	0x09, 0x54, // Usage (Contact Count)
	0x25, 0x7F, // Logical Maximum (127)
	0x95, 0x01, // Report Count (1)
	0x75, 0x08, // Report Size (8)
	0x81, 0x02, // Input (Data,Var,Abs,NWrp,Lin,Pref,NNul,Bit)
0xC0,  // End Collection

I've used this output function, now HID_MOUSE_IN_RPT_LEN is 10B.

		static uint32_t scantime = 0;
		uint8_t buffer[HID_MOUSE_IN_RPT_LEN];
		
		buffer[0] = 0x40 | (mouse_button-1);   // Buttons
		buffer[1] = mickeys_x & 0xFF;           // X low byte
		buffer[2] = (mickeys_x>>8) & 0xFF;       // X high byte
		buffer[3] = mickeys_y & 0xFF;           // Y low byte
		buffer[4] = (mickeys_y>>8) & 0xFF;       // Y high byte*/
		
		//scantime
		buffer[5] = scantime & 0xFF;
		buffer[6] = (scantime>>8) & 0xFF;
		buffer[7] = (scantime>>16) & 0xFF;
		buffer[8] = (scantime>>24) & 0xFF;
		 
		buffer[9] = 0x01; //contact count, increase for multiple touch input
		
		scantime++;

[xcarcelle]

@benjaminaigner thanks a lot for all these informations ! I will give a try on iOS later today and feedback you on this HID Descriptor if I can implement this Descriptor/Report.