Capacitive Touch Screen Emulation

[mbt28]

Hi,

Is it possible to emulate multi touch capacitive touch screen with this library?

Thanks

[NicoHood]

I dont think so. You can only emulate an absolute mouse.

[mbt28]

Hi Nico,

What is the thing which makes it not possible?

Thanks

[NicoHood]

Because I dont know how to have multiple inputs? Do you have any sample of a working HID device?

[mbt28]

Actually yes, and there are device descriptors which can be found in those links. I am working on a simple library to make it with pro micro. Still it doesnt function, when it functions I will put on github.
(I am not windows fun but the explanation is quite good)
https://docs.microsoft.com/en-us/windows-hardware/design/component-guidelines/supporting-usages-in-digitizer-report-descriptors

With teensy:
https://www.codeproject.com/Articles/1001891/A-USB-HID-Keyboard-Mouse-Touchscreen-emulator-with

[mbt28]

Here is my first trial code if you are interested, but windows says that "A request for the USB device descriptor failed." I couldnt try it with linux yet because now I only have my work pc.

https://gist.github.com/mbt28/406bdf15a248029c774085832c7c0c0c

[amol-india]

Hi mbt28 ,
I tried your code on my arduino-leonardo +windows 7 os.
Windows recognized as single touch digitizer but it not worked further.
I checked under Usblyzer .I found following problem in your sketch.
You are sending 8 byte instead 7 byte in sendreport() function.
(first byte is Report Id and remaining 7byte.
I modified your code .Now windows recognizes as single touch digitizer.
I emulated simple click,double-click and moving cursor to selected x,y coordinate.

[mbt28]

Hi

Actually I wrote usb multi touch hid to Arduino. But my code is such a mess now, working properly. Nowadays I have heavy workload once I have free time I will put it to github.

[MahendraSondagar]

Hello mbt28

              I'm also working on the same project, but with raspberry pi Zero-W. I have emulator keyboard via Bluetooth  and it's working great

I also need to emulate HID mutitouch screen with raspberry pi Zero w to send the data to host via bluetooth
Any one have reference with multi-touch screen please ?
Thanks

[AiueoABC]

@mbt28
I'm playing with your code. I confirmed windows can recognize this as a touch device, but not sure how to check it's working correctly.

If there is any progress, I want to know about it.
What I want to do is emulating double touching to do pinch-in and pinch-out with pushing button.

[mbt28]

Hi, I am not maintaining this code anymore however the logic was simple. If you used a valid descriptor (please check microsoft webpage) just report your touch locations in parallel or serial. Serial worked me very well. Regards

…

On Thu, Aug 20, 2020, 07:30 Ueo ABC, Ai ***@***.***> wrote: @mbt28 <https://github.com/mbt28> I'm playing with your code. I confirmed windows can recognize this as a touch device, but not sure how to check it's working correctly. If there is any progress, I want to know about it. What I want to do is emulating double touching to do pinch-in and pinch-out with pushing button. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#123 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADM2W5UZ73BMHICNJ26RDUTSBSYGHANCNFSM4EBNKLXQ> .

[mbt28]

Hi,

I added here the source that I used for creating valid descriptor. I used the serial reporting and you just reporting to OS and it somehow understand the multiple fingers. It worked for the Windows, Linux and Android.

https://docs.microsoft.com/en-us/windows-hardware/design/component-guidelines/sample-report-descriptor-for-a-touch-digitizer-device

Hope somebody will have enough time and put a working code online.

[NicoHood]

I am inlining the content here, in case the link will break in the future. Thanks for sharing!

0x05, 0x0d,                         // USAGE_PAGE (Digitizers)
    0x09, 0x04,                         // USAGE (Touch Screen)
    0xa1, 0x01,                         // COLLECTION (Application)
    0x85, REPORTID_TOUCH,               //   REPORT_ID (Touch)
    0x09, 0x20,                         //   USAGE (Stylus)
    0xa1, 0x00,                         //   COLLECTION (Physical)
    0x09, 0x42,                         //     USAGE (Tip Switch)
    0x15, 0x00,                         //     LOGICAL_MINIMUM (0)
    0x25, 0x01,                         //     LOGICAL_MAXIMUM (1)
    0x75, 0x01,                         //     REPORT_SIZE (1)
    0x95, 0x01,                         //     REPORT_COUNT (1)
    0x81, 0x02,                         //     INPUT (Data,Var,Abs)
    0x95, 0x03,                         //     REPORT_COUNT (3)
    0x81, 0x03,                         //     INPUT (Cnst,Ary,Abs)
    0x09, 0x32,                         //     USAGE (In Range)
    0x09, 0x47,                         //     USAGE (Confidence)
    0x95, 0x02,                         //     REPORT_COUNT (2)
    0x81, 0x02,                         //     INPUT (Data,Var,Abs)
    0x95, 0x0a,                         //     REPORT_COUNT (10)
    0x81, 0x03,                         //     INPUT (Cnst,Ary,Abs)
    0x05, 0x01,                         //     USAGE_PAGE (Generic Desktop)
    0x26, 0xff, 0x7f,                   //     LOGICAL_MAXIMUM (32767)
    0x75, 0x10,                         //     REPORT_SIZE (16)
    0x95, 0x01,                         //     REPORT_COUNT (1)
    0xa4,                               //     PUSH
    0x55, 0x0d,                         //     UNIT_EXPONENT (-3)
    0x65, 0x00,                         //     UNIT (None)
    0x09, 0x30,                         //     USAGE (X)
    0x35, 0x00,                         //     PHYSICAL_MINIMUM (0)
    0x46, 0x00, 0x00,                   //     PHYSICAL_MAXIMUM (0)
    0x81, 0x02,                         //     INPUT (Data,Var,Abs)
    0x09, 0x31,                         //     USAGE (Y)
    0x46, 0x00, 0x00,                   //     PHYSICAL_MAXIMUM (0)
    0x81, 0x02,                         //     INPUT (Data,Var,Abs)
    0xb4,                               //     POP
    0x05, 0x0d,                         //     USAGE PAGE (Digitizers)
    0x09, 0x48,                         //     USAGE (Width)
    0x09, 0x49,                         //     USAGE (Height)
    0x95, 0x02,                         //     REPORT_COUNT (2)
    0x81, 0x02,                         //     INPUT (Data,Var,Abs)
    0x95, 0x01,                         //     REPORT_COUNT (1)
    0x81, 0x03,                         //     INPUT (Cnst,Ary,Abs)
    0xc0,                               //   END_COLLECTION
    0xc0,                               // END_COLLECTION

[xcarcelle]

@NicoHood : thanks for this great project allowing different options on HID functions.
@mbt28 : I have tested the absoluteMouse example along with Arduino Micro board and the master branch HID Report then with the microsoft HID Report you proposed (I assumed REPORTID_TOUCH=0x01).
It did work w/ Ubuntu 20 but not with Android 8.0/9.0 nor ios14. Have you been able to absoluteMouse moveTo with Android ? Have you tested with ios ? Cheers.

[mbt28]

Hi @xcarcelle ,

Yes it was working with android but I dont really remember if I test it with an IOS device. But it was working with Linux, Windows, and Android 8 or 7 at that time. (I had still nexus 4 these days).

Look at the code on touch.cpp, after the correct descriptor used you should have a different procedure for reporting not the absolute mouse. You should write it with getting help from Nico's code, he has actually everything. Use the move function everytime you have a touch event, thats just it. I can post it with the descriptor included too if you want.

//	Multi Touch Reporting

Touch_::Touch_(void)
{
    static HIDSubDescriptor node(_hidReportDescriptor, sizeof(_hidReportDescriptor));
    HID().AppendDescriptor(&node);
}

void Touch_::begin(void) 
{
}

void Touch_::end(void) 
{
}

void Touch_::move(int16_t x, int16_t y)
{
	uint8_t m[3];
	m[0] = contactCount;
	m[1] = contactID;
	m[2] = tipSW;
	m[3] = wheel;
	HID().SendReport(m, x, y);
	// With this declaration a data packet must be sent as:
	// byte 1   -> "contact count"        (always == 1)		
	// byte 2   -> "contact identifier"   (any value)
	// byte 3   -> "Tip Switch" state     (bit 0 = Tip Switch up/down, bit 1 = In Range)
	// byte 4,5 -> absolute X coordinate  (0...10000)
	// byte 6,7 -> absolute Y coordinate  (0...10000)
}

Touch_ Touch;

#endif

[xcarcelle]

Hi @mbt28
thanks a lot for your reply. If possible can you paste your working code/HID used for Android/Linux ? I will test it out with iOS and Android and feedback you ? (paste so I confirm I got it right as in "use the latest master branch", "call the Touch::move(2500, 3200)" for example and the mouse should move to 25%, 32% of the screen size ?
In the code you pasted "uint8_t m[3]" should not be "uint8_t m[4]" to have the correct size ? Best.

[erictroebs]

@xcarcelle
I'm currently developing a game controller using an Arduino Micro (ATmega32U4) for touchscreen controlled games on Android using information from this issue and linked web pages. Some basic functions are now working with Linux Mint 20 (Kernel 5.4.0-52) and Android 10 (Huawei Honor 10 / Kernel 4.14.116).

Would you like to try my example code on whatever platform you're using?
https://gist.github.com/erictroebs/3e6ca8aa2b9ed06e0b5527fd38dd2c2f

Touchy.ino emulates ten fingers to draw ten rectangles concurrently on your screen. On my Android device with touch position overlay enabled it looks like this:

I'm looking forward to your answer. If time permits I'm going to create a pull request.

Edit: I edited my gist to support a report id. This means HID().SendReport can be used according to its actual definition. I don't know if it behaves differently or more compatible though. Compact and easy to understand information about HID is pretty rare.

[xcarcelle]

@erictroebs : thanks for your message.
I have tested the ino (using an Arduino Micro) with some devices :

• Android 7.0 (Huawey MediaPad T3) : woks well
• IOS14+ (ipad 7th generation) : does not work (normal as your HID is a digitizer and not supported since ios14)
  Have you tried your code with ESP32-BLE-Mouse using this HID ? As it looks smooth it could be great to have a BLE support.
  Let's iterate on this topic, thanks.

[xcarcelle]

@erictroebs : I made more tests with you ino touchscreen code with a Android tablet and it looks like the (0, 0) origin of your code is down/left corner and from there absolute move are able. Do you know why this position ? Can it be possible to have it in the upper/left corner ? Cheers.

[ShiverZm]

@xcarcelle screen touch cannot use in ios 14? right? below 14 is ok? have you tried it?