Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ESP32 BT-HID Device #230

Open
gitmh opened this issue Nov 26, 2017 · 216 comments
Open

ESP32 BT-HID Device #230

gitmh opened this issue Nov 26, 2017 · 216 comments

Comments

@gitmh
Copy link

@gitmh gitmh commented Nov 26, 2017

Is it possible to use the ESP32_BLE_Arduino library to build a HID-Keyboard to send keystrokes to another device? My goal: I would like to connect a ESP32 board to a BT device like an iPad and send a space character. I found https://github.com/asterics/esp32_mouse_keyboard but can't figure out how to this running in Arduino IDE. Any help would be appreciated - I found a lot of users asking for a similar feature for the ESP32.

@nkolban
Copy link
Owner

@nkolban nkolban commented Nov 26, 2017

I'm afraid there is nothing yet in the BLE C++ classes that would make your request possible. Maybe a question posted to one of the forums here might give some positive results:

https://esp32.com/index.php

It is my loose understanding that HID support is based on Bluetooth classic (BR/EDR) as opposed to BLE and, so far, these classes only cover BLE functions.

@gitmh
Copy link
Author

@gitmh gitmh commented Nov 26, 2017

Thank you very much @nkolban. I already found a project for esp-idf for HID and BT but I was hoping for an example for the arduino IDE. I have no experience with IDF and the tool chain and how to integrate those stuff with arduino. Maybe this will come in the future.

@nkolban
Copy link
Owner

@nkolban nkolban commented Nov 26, 2017

Do you have a link to the HID and BT project that uses ESP-IDF? I'd like to file that and have a look to see what it would take to eventually encapsulate that in the C++ libraries. That would be the first step towards an Arduino encapsulation.

@gitmh
Copy link
Author

@gitmh gitmh commented Nov 27, 2017

The most complete and supported solution seems to be the BTStack port for the esp32 for IDF here
https://github.com/bluekitchen/btstack/tree/master/port/esp32 within this library there are various examples for HID devices and other BT applications.

@nkolban
Copy link
Owner

@nkolban nkolban commented Nov 27, 2017

OOOh thats cool. But scary as well. If I get a sense of that story, it is a replacement for the Bluetooth stack supplied by Espressif (which I believe is based on Bluedroid stack implementation).

@gitmh
Copy link
Author

@gitmh gitmh commented Nov 28, 2017

Yes so this might be a problem to have both stacks available at the same time in an Arduino library. The example under https://github.com/asterics/esp32_mouse_keyboard uses the bt.h from the regular Espressif library so (see ble_hidd_demo_main.c).

@chegewara
Copy link
Collaborator

@chegewara chegewara commented Dec 14, 2017

New day, new lessons:
hid

My esp32 just become HID keyboard. ANd now its HID mouse:
mouse

Of course its not real keyboard and mouse, but my windows 10 is stupid and thinks it is ;)

@gitmh
Copy link
Author

@gitmh gitmh commented Dec 14, 2017

@chegewara Have you build an Arduino project for that already? I would be interested :-)
Sending a keyboard key to any BT device like an iPAD would be a great use case for the ESP32.

@chegewara
Copy link
Collaborator

@chegewara chegewara commented Dec 14, 2017

Oh no, just fooled my laptop by setting appearance(). The rest of code has nothing to do with hid keyboard. But i may do some research and see if this is hard to do.

@chegewara
Copy link
Collaborator

@chegewara chegewara commented Dec 15, 2017

I have some good news and some bad news. Good news that work is in progress. Bad news is driver crash under windows with error code 0x0a(device cant start). I can say for sure what is the reason, but its a chance that hid keybord requires secure connection and we dont have implemented it yet.

My next step will be to rewrite code in esp-idf and see what happen and maybe add some security there.

EDIT more good news, my samsung s4 connects to esp32 hid keyboard, so now i have to find way to send some text

@chegewara
Copy link
Collaborator

@chegewara chegewara commented Dec 15, 2017

I have good news. Esp32 hid keyboard is connecting with my android phone and i can send text. This mean hid keyboard works, at least software part. Still is some work to do but we know it can be done.

@gitmh
Copy link
Author

@gitmh gitmh commented Dec 15, 2017

@chegewara great work, would you share your current project to play around with it? I just need a running starting point for the Arduino IDE :-)

@chegewara
Copy link
Collaborator

@chegewara chegewara commented Dec 15, 2017

Im not sure yet, i will provide code or share links ive been using during research. Im thinking about writing blog article about this. For now maybe this will help you a bit:
https://github.com/I0x0I/DIY-A-BLE-Keyboard

https://docs.mbed.com/docs/ble-hid/en/latest/api/md_doc_HIDService.html

@chegewara
Copy link
Collaborator

@chegewara chegewara commented Dec 15, 2017

This is very important documentation if you want to play with HID devices. Appendix E gives some good look at what descriptors are required to setup hid device:
http://www.usb.org/developers/hidpage/HID1_11.pdf

@chegewara
Copy link
Collaborator

@chegewara chegewara commented Dec 15, 2017

Services and characteristics:

This XML file does not appear to have any style information associated with it. The document tree is shown below.
<configuration>
<service uuid="1800">
<description>Generic Access Profile</description>
<characteristic uuid="2a00">
<properties read="true" const="true"/>
<value>BGT Keyboard Demo</value>
</characteristic>
<characteristic uuid="2a01">
<properties read="true" const="true"/>
<value type="hex">c103</value>
</characteristic>
</service>
<service type="primary" uuid="180A" id="manufacturer">
<characteristic uuid="2A29">
<properties read="true" const="true"/>
<value>Bluegiga</value>
</characteristic>
<characteristic uuid="2A50">
<!--  PnP ID required by HID Profile  -->
<properties read="true" const="true"/>
<value type="hex">014700ffffffff</value>
</characteristic>
</service>
<service uuid="180f">
<description>Battery</description>
<characteristic uuid="2a19" id="xgatt_battery">
<properties read="true"/>
<value type="hex">32</value>
</characteristic>
</service>
<service uuid="1812" advertise="true">
<characteristic uuid="2a4d" id="hid_keyboard_in">
<!--  Keyboard input report  -->
<properties notify="true" read="true"/>
<value length="20" variable_length="true"/>
<descriptor uuid="2908">
<!--  Report reference id=0x00 type=0x01 (input)  -->
<properties read="true" const="true"/>
<value type="hex">0001</value>
</descriptor>
</characteristic>
<characteristic uuid="2a4d" id="hid_keyboard_out">
<!--  Keyboard output report  -->
<properties write="true" write_no_response="true" read="true"/>
<value length="20" variable_length="true"/>
<descriptor uuid="2908">
<!--  Report reference id=0x00 type=0x02 (output)  -->
<properties read="true" const="true"/>
<value type="hex">0002</value>
</descriptor>
</characteristic>
<characteristic uuid="2a4d" id="hid_keyboard_feature">
<!--  Keyboard feature report  -->
<properties write="true" read="true"/>
<value length="20" variable_length="true"/>
<descriptor uuid="2908">
<!--  Report reference id=0x00 type=0x03 (feature)  -->
<properties read="true" const="true"/>
<value type="hex">0003</value>
</descriptor>
</characteristic>
<characteristic uuid="2a4b">
<!--  Report map example from USB HID Specification  -->
<properties read="true" authenticated_read="true" const="true"/>
<value type="hex">
05010906A101050719E029E71500250175019508810295017508810195057501050819012905910295017503910195067508150025650507190029658100C0
</value>
</characteristic>
<characteristic uuid="2a4a">
<!--
 HID Information version=0x0100 countrycode=0x00 flags=0x02 (normally connectable) 
-->
<properties read="true" const="true"/>
<value type="hex">00010002</value>
</characteristic>
<characteristic uuid="2a4c" id="hid_control">
<!--  HID Control point, used to control suspending  -->
<properties write_no_response="true"/>
<value length="1"/>
</characteristic>
<characteristic uuid="2a22" id="hid_boot_keyboard_in">
<!--  Boot Keyboard input report  -->
<properties notify="true" read="true"/>
<value length="20" variable_length="true"/>
</characteristic>
<characteristic uuid="2a32" id="hid_boot_keyboard_out">
<!--  Boot Keyboard output report  -->
<properties write_no_response="true" read="true" write="true"/>
<value length="20" variable_length="true"/>
</characteristic>
<characteristic uuid="2a4e" id="hid_protocol_mode">
<!--  Protocol mode select  -->
<properties write_no_response="true" read="true"/>
<value length="1"/>
</characteristic>
</service>
</configuration>
@gitmh
Copy link
Author

@gitmh gitmh commented Dec 15, 2017

@chegewara I managed to run the BlueKitchen ESP32 Port for ESP-IDF (https://github.com/bluekitchen/btstack/blob/master/example/hid_keyboard_demo.c) it runs on my ESP32 board and sends continuously text messages to may mac book and to my android phone as soon as you pair them. The whole setup for the ESP-IDF and libraries is quite complicated so I cannot use this process for teaching. This is why I was looking for a nice wrapper for the Arduino IDE so that you can fire up a example project, hit flash and project runs :-)

@chegewara
Copy link
Collaborator

@chegewara chegewara commented Dec 16, 2017

Can you wait few days, im still working on arduino code. At the moment im just sending random letter but i still missing one thing. My code requires that notifications to be turned on on client side (my android phone). At the moment only way to achieve this is to open nRF connect, connect to esp32 hid and turn on notifications.

@chegewara
Copy link
Collaborator

@chegewara chegewara commented Dec 16, 2017

This is what i did so far. There is still a lot to do but it should send random character in loop:

#include <BLEDevice.h>
#include <BLEUtils.h>
#include <BLEServer.h>
#include <BLE2902.h>
#include <string>

// See the following for generating UUIDs:
// https://www.uuidgenerator.net/

bool connected = false;
  BLEService *pService;
  BLEService *pService1;
  BLECharacteristic *reportChar1;
  BLECharacteristic *reportChar2;
  BLECharacteristic *reportChar3;
  class MyCallbacks : public BLEServerCallbacks {
    void onConnect(BLEServer* pServer){
      connected=true;
    }
    
    void onDisconnect(BLEServer* pServer){
      connected=false;
    }
  };
  
void setup() {
  Serial.begin(115200);
  Serial.println("Starting BLE work!");

  BLEDevice::init("ESP32");
  BLEServer *pServer = BLEDevice::createServer();
  pServer->setCallbacks(new MyCallbacks());
  pService = pServer->createService((uint16_t)0x180a);
  pService1 = pServer->createService((uint16_t)0x1812, 30);
  setupCharacteristics();

  pService->start();
  pService1->start();
  BLEAdvertising *pAdvertising = pServer->getAdvertising();
  pAdvertising->setAppearance(961);
  pAdvertising->addServiceUUID((uint16_t)0x1812);
  pAdvertising->start();

  Serial.println("Characteristic defined! Now you can read it in your phone!");
}
uint8_t v[] = {0x0, 0x0, 0x0, 0x0,0x0,0x0,0x0,0x0};

void loop() {
  // put your main code here, to run repeatedly:
  delay(5000);
  if(connected){
    uint8_t a[] = {0x0, 0x0, random(0x04,0x26), 0x0,0x0,0x0,0x0,0x0};
    reportChar1->setValue(a,sizeof(a));
    reportChar1->notify();    

    reportChar1->setValue(v, sizeof(v));
    reportChar1->notify();
  }
}

void setupCharacteristics() {

  BLECharacteristic *manufacturer = pService->createCharacteristic(
                                       (uint16_t)0x2a29,
                                       BLECharacteristic::PROPERTY_READ 
                                     );
  std::string name = "espressif";
  manufacturer->setValue(name);
                                     
  BLECharacteristic *pnpIDChar = pService->createCharacteristic(
                                       (uint16_t)0x2a50,
                                       BLECharacteristic::PROPERTY_READ 
                                     );
  const uint8_t pnp[] = {0x01,0xe5,0x02,0xcd,0xab,0x01,0x00};
  pnpIDChar->setValue((uint8_t*)pnp, sizeof(pnp));
  
  BLECharacteristic *hidInfoChar = pService1->createCharacteristic(
                                       (uint16_t)0x2a4a,
                                       BLECharacteristic::PROPERTY_READ 
                                     );
  const uint8_t val1[] = {0x00,0x01,0x00,0x02};
  hidInfoChar->setValue((uint8_t*)val1, 4);

  BLECharacteristic *reportMapChar = pService1->createCharacteristic(
                                       (uint16_t)0x2a4b,
                                       BLECharacteristic::PROPERTY_READ 
                                     );
  const uint8_t val2[] = {
    0x05,0x01,0x09,0x06,0xA1,0x01,0x05,0x07,
    0x19,0xE0,0x29,0xE7,0x15,0x00,0x25,0x01,
    0x75,0x01,0x95,0x08,0x81,0x02,0x95,0x01,
    0x75,0x08,0x81,0x01,0x95,0x05,0x75,0x01,
    0x05,0x08,0x19,0x01,0x29,0x05,0x91,0x02,
    0x95,0x01,0x75,0x03,0x91,0x01,0x95,0x06,
    0x75,0x08,0x15,0x00,0x25,0x65,0x05,0x07,
    0x19,0x00,0x29,0x65,0x81,0x00,0xC0}; //TODO
  reportMapChar->setValue((uint8_t*)val2, sizeof(val2));

  BLECharacteristic *hidControlChar = pService1->createCharacteristic(
                                       (uint16_t)0x2a4c,
                                       BLECharacteristic::PROPERTY_WRITE_NR 
                                     );

  reportChar1 = pService1->createCharacteristic(
                                       (uint16_t)0x2a4d,
                                       BLECharacteristic::PROPERTY_READ   |
                                       BLECharacteristic::PROPERTY_NOTIFY                                       
                                     );
  BLEDescriptor *desc1 = new BLEDescriptor(BLEUUID((uint16_t)0x2908));
  const uint8_t desc1_val[] = {0x01, 0};
  desc1->setValue((uint8_t*)desc1_val, 1);
  reportChar1->addDescriptor(desc1);
  reportChar1->addDescriptor(new BLE2902());

  reportChar2 = pService1->createCharacteristic(
                                       (uint16_t)0x2a4d,
                                       BLECharacteristic::PROPERTY_READ   |
                                       BLECharacteristic::PROPERTY_WRITE                                       
                                     );
  BLEDescriptor *desc2 = new BLEDescriptor(BLEUUID((uint16_t)0x2908));
  const uint8_t desc2_val[] = {0x02, 0};
  desc2->setValue((uint8_t*)desc2_val, 1);
  reportChar2->addDescriptor(desc2);

  reportChar3 = pService1->createCharacteristic(
                                       (uint16_t)0x2a4d,
                                       BLECharacteristic::PROPERTY_READ     |
                                       BLECharacteristic::PROPERTY_WRITE    |
                                       BLECharacteristic::PROPERTY_WRITE_NR 
                                     );
  BLEDescriptor *desc3 = new BLEDescriptor(BLEUUID((uint16_t)0x2908));
  const uint8_t desc3_val[] = {0x03, 0};
  desc3->setValue((uint8_t*)desc3_val, 1);
  reportChar3->addDescriptor(desc3);

  BLECharacteristic *protocolModeChar = pService1->createCharacteristic(
                                       (uint16_t)0x2a4e,
                                       BLECharacteristic::PROPERTY_WRITE_NR 
                                     );

  BLECharacteristic *bootInputChar = pService1->createCharacteristic(
                                       (uint16_t)0x2a22,
                                       BLECharacteristic::PROPERTY_NOTIFY
                                     );
  bootInputChar->addDescriptor(new BLE2902());


  BLECharacteristic *bootOutputChar = pService1->createCharacteristic(
                                       (uint16_t)0x2a32,
                                       BLECharacteristic::PROPERTY_READ     |
                                       BLECharacteristic::PROPERTY_WRITE    |
                                       BLECharacteristic::PROPERTY_WRITE_NR 
                                     );



}
@gitmh
Copy link
Author

@gitmh gitmh commented Dec 17, 2017

@chegewara Great work!!! It works like a charm on my Android phone. I can also connect the ESP32 with iOS and Mac OS, but no characters appear there - guess this is the security issue problem you were talking about. For testing I used two ESP32 Hellcat LoRa boards. One board is sending a string over LoRa, to the other board which is connected as HID device on Android transferring all input received as keystrokes.

@chegewara
Copy link
Collaborator

@chegewara chegewara commented Dec 18, 2017

Sorry but my windows has Polish localization :(

hid

@ripper121
Copy link

@ripper121 ripper121 commented Feb 13, 2018

Any news about this?
Would be super cool to use it as a AT/PS2 to BLE HID Keyboad adapter.

@chegewara
Copy link
Collaborator

@chegewara chegewara commented Feb 13, 2018

@ripper121 What news do you need? Its working for few weeks now. There is one example added. If you have questions, issues or suggestions im open and ready to help.

@ripper121
Copy link

@ripper121 ripper121 commented Feb 14, 2018

Is there a Arduino Lib for this or is it only possible with the SDK?

@chegewara
Copy link
Collaborator

@chegewara chegewara commented Feb 14, 2018

It should work with arduino too. Last time i checked it did work with arduino.

@ripper121
Copy link

@ripper121 ripper121 commented Feb 16, 2018

Yesss thx its working :)

@hmatulis
Copy link

@hmatulis hmatulis commented Mar 22, 2018

Hi great work, I tried it on my windows 10, but couldn't manage to make it work.
First I got a "This device cannot start (Code 10)" and fixed it with https://www.kapilarya.com/fix-this-device-cannot-start-code-10-in-windows-10

After that I've got: "Driver error" - It connects and pairs ok, but doesn't work.

Monitor log:
Starting BLE work!
Characteristic defined! Now you can read it in your phone!
�[0;31mE (27800) BT: lmp_version_below LMP version 6 < 8�[0m
�[0;31mE (27860) BT: Call back not found for application conn_id=3�[0m
�[0;31mE (33978) BT: bta_gattc_conn_cback() - cif=3 connected=0 conn_id=3 reason=0x0013�[0m

any hint?
thanks

@chegewara
Copy link
Collaborator

@chegewara chegewara commented Mar 22, 2018

Hi,
most likely, and im only guessing right now, its problem with report map. Its this part of code (example code):

		const uint8_t reportMap[] = {
			USAGE_PAGE(1),      0x01,       // Generic Desktop Ctrls
			USAGE(1),           0x06,       // Keyboard
			COLLECTION(1),      0x01,       // Application
			REPORT_ID(1),		0x01,		// REPORTID
			USAGE_PAGE(1),      0x07,       //   Kbrd/Keypad
			USAGE_MINIMUM(1),   0xE0,
			USAGE_MAXIMUM(1),   0xE7,
			LOGICAL_MINIMUM(1), 0x00,
			LOGICAL_MAXIMUM(1), 0x01,
			REPORT_SIZE(1),     0x01,       //   1 byte (Modifier)
			REPORT_COUNT(1),    0x08,
			INPUT(1),           0x02,       //   Data,Var,Abs,No Wrap,Linear,Preferred State,No Null Position
			REPORT_COUNT(1),    0x01,       //   1 byte (Reserved)
			REPORT_SIZE(1),     0x08,
			INPUT(1),           0x01,       //   Const,Array,Abs,No Wrap,Linear,Preferred State,No Null Position
			REPORT_COUNT(1),    0x05,       //   5 bits (Num lock, Caps lock, Scroll lock, Compose, Kana)
			REPORT_SIZE(1),     0x01,
			USAGE_PAGE(1),      0x08,       //   LEDs
			USAGE_MINIMUM(1),   0x01,       //   Num Lock
			USAGE_MAXIMUM(1),   0x05,       //   Kana
			OUTPUT(1),          0x02,       //   Data,Var,Abs,No Wrap,Linear,Preferred State,No Null Position,Non-volatile
			REPORT_COUNT(1),    0x01,       //   3 bits (Padding)
			REPORT_SIZE(1),     0x03,
			OUTPUT(1),          0x01,       //   Const,Array,Abs,No Wrap,Linear,Preferred State,No Null Position,Non-volatile
			REPORT_COUNT(1),    0x06,       //   6 bytes (Keys)
			REPORT_SIZE(1),     0x08,
			LOGICAL_MINIMUM(1), 0x00,
			LOGICAL_MAXIMUM(1), 0x65,       //   101 keys
			USAGE_PAGE(1),      0x07,       //   Kbrd/Keypad
			USAGE_MINIMUM(1),   0x00,
			USAGE_MAXIMUM(1),   0x65,
			INPUT(1),           0x00,       //   Data,Array,Abs,No Wrap,Linear,Preferred State,No Null Position
			END_COLLECTION(0)
		};

btw, ive been testing hid examples with windows 10 and there is no need to edit registers to make it works

If you can post report map i will try to check it, also screenshot with error would be great

@ossvr
Copy link

@ossvr ossvr commented Dec 8, 2019

Just like in the example “SendKeyStrokes.ino” the command “bleKeyboard.releaseAll()” is sent to release all keys, so I need to send the commands “bleKeyboard.press(KEY_RIGHT_ARROW)” and “bleKeyboard.release(KEY_RIGHT_ARROW)” ? Oops, I think I was messing up. Should key "KEY_RIGHT_ARROW" be sent by "bleKeyboard.press()"? I was using "bleKeyboard.write()". Looking at "BleKeyboard.cpp" the "write" command sends "keydown/keyup" together. So that was the problem. I'm sorry guys, it was my fault. I will modify the code in ESP and do the tests.

@T-vK
Copy link

@T-vK T-vK commented Dec 8, 2019

Yes, if you want to control when a given key is released then you must not use bleKeyboard.write(...), but bleKeyboard.press(...) and either bleKeyboard.release(...) or bleKeyboard.releaseAll(...).

If you want to mimic a real keyboard then you don't implement the repeated keypresses on the ESP because as I already said in the previous post, this is a feature that is meant to be controlled by the target device.

@ossvr
Copy link

@ossvr ossvr commented Dec 8, 2019

Running perfectly now. Thank you all for the clarifications.

@maximeCote819
Copy link

@maximeCote819 maximeCote819 commented Feb 18, 2020

Hi, I'm really knew to Arduino and C++.

I think I'm at the right place, I'm trying to make a page turner for an Ipad app, and so far, this is the thread that got me the furthest to reach my goal.

I'm sorry in advance if it's supposed to be "so simple", but how and where in the code (thank you @chegewara ) am I supposed to declare the keyboard character ? And if I want to link the keyboard character to an event (pressing a button, indeed), how would I do so ?

Thank you guys !

@T-vK
Copy link

@T-vK T-vK commented Feb 18, 2020

Look at the examples in the IDE. There are examples for buttons and examples for this ble keyboard library. You really just need to merge them and use common sense.

@maximeCote819
Copy link

@maximeCote819 maximeCote819 commented Feb 19, 2020

Wow, thanks so much ! It's almost done, and I basically just replaced some lines on the example.

@lemmingDev
Copy link

@lemmingDev lemmingDev commented Mar 19, 2020

Am looking to modify @T-vK mouse library to be an absolute mouse so I can use it for a wireless IR lightgun, similar to the SAMCO project https://github.com/samuelballantyne/IR-Light-Gun/tree/master/Samco_2.0 where they use the AbsMouse library.

I've got it working, where I can pass the modified library the X,Y coordinates I'd like it to go to, and it goes really well.

One limitation is that you have to pass in the screen resolution when starting the library. This can be a hassle where you want to use your lightgun in many different emulator and the resolution may change from game to game,

Question: Is there a way to have the ESP query what resolution Windows 10 is running, then pass that information back to the ESP over BLE so the library can use the current resolution?

I guess I could make a calibration button to throw it into regular BT serial mode, make a C# app that passes the values to the ESP, save them to SPIFF, and then reboot to use the values.

Sending the values to the ESP without a calibration app would be heaps better though

@chegewara
Copy link
Collaborator

@chegewara chegewara commented Mar 19, 2020

I would have to dig some more in HID specification, i dont remember now if its possible.
I know i did help very log time ago with some mouse like device:
https://github.com/markingle/Assist_IoT_ESP32_LipSync
but dont remember how it was handled in there. You can try to read code.

@T-vK
Copy link

@T-vK T-vK commented Mar 19, 2020

I don't think that's possible without your own driver/script/app on Windows. Be aware that the screen resolution can change at any time and you get pretty interesting results for multimonitor setups, especially if they run different resolutions on each monitor or their virtual placement is customized.
It should be possible to create an hid descriptor that combines serial and mouse into one device. This way you could just send the data during runtime.
Another challenge I can see are applications that take over your entire display output, like directx fullscreen etc. I don't think the WinAPI will be able to tell you the resolution in those cases and reading it from the processes memory could get you banned if some sort of anticheat mechanisms are in place.

@chegewara
Copy link
Collaborator

@chegewara chegewara commented Mar 19, 2020

Worst case scenario you can have button/buttons to select resolution from most common.

@lemmingDev
Copy link

@lemmingDev lemmingDev commented Mar 20, 2020

Thanks for the well reasoned replies.

I would be very interested to see how to incorporate serial communication as a composite device.
I will probably be using this mouse/keyboard combo library
https://github.com/blackketter/ESP32-BLE-Combo
which is by @blackletter based on @T-vK 's work to facilitate the mouse moves (will change the mouse to absolute) and also a number of keyboard keys. I wonder how hard it would be to add. Are there any examples?

At the end of the day though, maybe a button to select resolutions on the device is the way to go.
Might also give me a reason to pop an SSD1306 OLED in there for other options too to see some type of feedback.

I do like problem solving though and I can't help thinking that adding serial on top of BLE mouse/keyboard would be super great and might come in useful for others.

@lemmingDev
Copy link

@lemmingDev lemmingDev commented Mar 23, 2020

So, any idea if it's possible to run serial (where a COM port number is allocated and data may be sent at say 115200 or 250000 baud) at the same time as BLE keyboard/mouse?

@chegewara
Copy link
Collaborator

@chegewara chegewara commented Mar 23, 2020

Sure it is possible.
Regular debug log is running on serial (UART 0). But you will need ftdi uart to usb to connect with PC.

@lemmingDev
Copy link

@lemmingDev lemmingDev commented Mar 23, 2020

Hi

Thanks for the reply
Sorry - I meant both over bluetooth

@chegewara
Copy link
Collaborator

@chegewara chegewara commented Mar 23, 2020

Hi

Thanks for the reply
Sorry - I meant both over bluetooth

In theory it should be possible, but i didnt work with classic bluetooth, so i cant help with that.

@lemmingDev
Copy link

@lemmingDev lemmingDev commented Mar 23, 2020

I didn't even think to try running Classic BT at the same time as BLE. I just assumed you'd need to add some type of BLE COM port wizardry.

I threw together a sketch with both of them and it compiles.
I'll test it later on and reply with how it goes.

Thanks!

@lemmingDev
Copy link

@lemmingDev lemmingDev commented Mar 23, 2020

@cezarymalek
Copy link

@cezarymalek cezarymalek commented Apr 3, 2020

Hi!
Does anyone have problems with the mouse cursor not moving smoothly?
I'm trying to run code from https://github.com/T-vK/ESP32-BLE-Mouse in the simplest way possible, that means repeating move(1,1,0) every 20 ms(tried with various numbers).
Mouse doesn't move nicely one pixel down and one left(it shall have 50 fps) instead of this it jumps few at once and it's very clear. Is esp not robust enough? shall I do it some other way? BLE frames are being quequed or what? Can anyone tell me? Maybe i shall write something more about what and how i do? I use WROOM32U without devboard(tho i guess it doesn't matter). Maybe you can tell me about some other uC that would perform in this task better?
This time i did it simplest possible, but i had the same problem in my previous project with some code provided by @chegewara (btw thank you dude for your awesome work in this field. You and NKolban are awesome! :D Pozdrawiam!).
I've already worked on the subject a year ago (or so) but dumped it due to the lack of time and this issue...

Greetings again!

PS. It's important for me because I'll use this device for gaming... when mice works this way you may get a headache during rotating view :D It's not so visible while using it in perfect conditions but very visible when i compare my bluetooth mice(real) and this soltion

PSS. Ive made some wireshark sniffing and every move() frame sent by esp is 16 bytes long.
with 50Hz it gives 800 B/s.
Here (https://esp32.com/viewtopic.php?t=2262) some guy writes about 12kB(after some fight to squeeze as much as possible) so i assume that the bandwidth is not a problem.
ESP isn't also too slow because it sends 50 frames like this is 11 ms(measured by looping it in for 50 times and printing difference between millis() results)
Ive also measured how long sending move() then waiting 20ms(delay) will take while doing it that 50 times(it shall be 1011 but surprisingly i got equal 1000)
Can it be improved by using normal Bluetooth instead of BLE? Maybe the connection and data loss is the problem?
So whats wrong then....

@chegewara
Copy link
Collaborator

@chegewara chegewara commented Apr 4, 2020

Hi,
my advice is to send data faster, esp32 should handle about 100Hz with BLE (with good conditions maybe up to 200Hz).

Everything depends how you build mouse from HID point of view. Usually mouse is using relative data, which means you are sending how far move pointer from current position. Then, after sending that you probably want to send value 0x0 or pointer will continue moving.

If you are asking about other uC to make this project i would rather suggest to try nrf52840, 53811, 52833. NRF chips are specialized BLE chips with very good power consumption ratio. esp32 is too powerful to only build mouse. Problem with nrf is that you have to learn new RTOS, because of lack good arduino support.

@cezarymalek
Copy link

@cezarymalek cezarymalek commented Apr 6, 2020

Thank you for your answer! :-) Latest tests ive made by arduino IDE and the oscillator was set on 240Mhz. Unless you mean the internal BLE transmission rate (dunno how i can set it up tho tomorrow I'll check all the documentation. Maybe it can help really.)

And about sending the stop value... well... you really made me think where's the mistake in the code. I remember that at first try ive took your keyboard example and modified it to be usable as a mouse, tho it didn't require a stop val. It was just moving cursor the given pixels in y and x once. If it's really supposed to work as you say then it's much better and the mistake is just a matter of a bug in my code. It casts new light on the case :D.

Thanks for the chip proposition also. Now i have plan B if esp wont work.
By the way project is a bit more complicated than a mouse. It's supposed to integrate both mouse and a keyboard in a single one-handed device, and it does require some computation power indeed. Maybe after some time i'll share somewhere the results :-)

Thanks again!

@chegewara
Copy link
Collaborator

@chegewara chegewara commented Apr 6, 2020

To send stop moving you just have to send X = 0x0 and Y = 0x0, just like in keyboard you have to send key = 0x0 or it will be repeated by system. I think its the same way in case of mouse, but its a small chance and you dont have to send 0x0 for mouse. To make sure you can try to send X = 100 just 1 time and see if mouse pointer will move only 1 time or more.

nrf chips are enough powerful to handle keyboard + mouse + even more, especially nrf52840.

aerialist added a commit to aerialist/ESP32-BLE-Keyboard that referenced this issue May 31, 2020
Implement change suggested by ossvr at
nkolban/esp32-snippets#230 (comment)

This solved reconnection problem I faced with Macbook Pro with MacOS Catalina
@xcarcelle
Copy link

@xcarcelle xcarcelle commented Oct 19, 2020

@lemmingDev :

I've got it working, where I can pass the modified library the X,Y coordinates I'd like it to go to, and it goes really well.
This is great ! Did you get it to work with Android on absolute mode on landscape orientation ?

Question: Is there a way to have the ESP query what resolution Windows 10 is running, then pass that information back to the ESP over BLE so the library can use the current resolution?
Yes use the SPP chanel for that purpose (even if it only once).

Sending the values to the ESP without a calibration app would be heaps better though
The HID ble channel at handshake transaction with the host sending back the resolution to the peripheral could be great.

@lemmingDev
Copy link

@lemmingDev lemmingDev commented Nov 28, 2020

Hi All

Any idea how to get Windows Game Controller panel to list the actual device name?

In my library https://github.com/lemmingDev/ESP32-BLE-Gamepad the name is set and comes up correctly in device manager and Bluetooth settings
image
image

In the game controller panel however, it just lists the number of axis and buttons as the name.
What's worse, is that it lists them for the first time the BLE gamepad was connected. For example, during development of the library, I started with 11 buttons, and now I'm at 32 buttons, but it still lists it as 11.

image

Hoping someone may be able to point me in the right direction

@lemmingDev
Copy link

@lemmingDev lemmingDev commented Nov 28, 2020

On a perhaps related note:

It should be noted that I edited the the https://github.com/T-vK/ESP32-BLE-Mouse library in order to make this library, so I have a feeling some values could be set wrong too.

I know my usage is set correctly in the HID report descriptor

USAGE(1),            0x05, // USAGE (Gamepad)

however I'm not sure about the following which were blindly copied

  BleGamepadInstance->hid->pnp(0x01,0x02e5,0xabcd,0x0110);
  BleGamepadInstance->hid->hidInfo(0x00,0x01);

If I change the pnp hex values above slightly: eg 0xabcd to 0xabcc so it thinks it's a new device, then Windows correctly changes the amount of buttons to:
image
however I'd still like it so say the actual name

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet