Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Common BLE API #5051

Open
wants to merge 14 commits into
base: master
from

Conversation

@jimmo
Copy link
Contributor

commented Aug 28, 2019

This PR is based on #4589 by @aykevl and #4893 by @dpgeorge

It implements a Python API for working with BLE devices in both central and peripheral role (including scanning and advertising), and is implemented with Nimble for STM32. I have WIP ESP32 support (based on #4859, but including central role) almost ready to go, although it might be an option to use Nimble on ESP32 instead. NRF is also in progress using the SD (but could potentially use Nimble also!).

The API is lower-level than #4589 -- it more closely matches the underlying BLE spec. The idea is that specific use cases (beacon, beacon detector, UART, etc) can be implemented with higher-level Python wrappers.

Some outstanding questions:

  • Memory management for Nimble. At the moment a soft reset will crash because Nimble allocates on the uPy heap.
  • Naming. Unsure about the module (bluetooth), class (Bluetooth), and method names (e.g. 'characteristics' vs 'chrs'). I hope that a future BT Classic implementation can share some of this so some thought required about whether to rename anything to be BLE specific.
  • Address types (private/public/static/random) and configuration.

Registering a service (e.g. NUS):

import bluetooth
bt = bluetooth.Bluetooth()
bt.active(True)

UART_SERVICE = bluetooth.UUID('6E400001-B5A3-F393-E0A9-E50E24DCCA9E')
UART_TX = (bluetooth.UUID('6E400003-B5A3-F393-E0A9-E50E24DCCA9E'), bluetooth.FLAG_READ|bluetooth.FLAG_NOTIFY,)
UART_RX = (bluetooth.UUID('6E400002-B5A3-F393-E0A9-E50E24DCCA9E'), bluetooth.FLAG_WRITE,)

tx, rx = bt.gatts_add_service(UART_SERVICE, (UART_TX, UART_RX,))

Scanning:

def bt_irq(event, data):
    if event == IRQ_SCAN_RESULT:
        addr_type, addr, connectable, rssi, adv_data = data
        print('Scan result', adv_decode_name(adv_data))
    elif event == IRQ_SCAN_COMPLETE:
        print('Scan complete')

bt.irq(bt_irq, IRQ_ALL)
bt.scan(2000)  # Scan for 2000 ms

Advertising:

# The encoding will be available as Python helpers.
bt.advertise(100, adv_encode_ble_only() + adv_encode_name('pybd-jimmo'))

The methods available on the Bluetooth object are:

# General
active([enable])
config(name) # e.g. config('mac')
irq(handler, trigger)

# Peripheral
gatts_add_service(uuid, ( (chr_uuid, flags), ... ) )
advertise(interval_ms, data)
gatts_notify(conn_handle, value_handle [, data])  # Optional data
gatts_read(conn_handle, value_handle)
gatts_write(conn_handle, value_handle, data)

# Central
scan(timeout_ms)  # None to stop scanning.
connect(addr_type, addr)
disconnect(conn_handle)
gattc_discover_services(conn_handle)
gattc_discover_characteristics(conn_handle, start_handle, end_handle)
gattc_discover_descriptors(conn_handle, start_handle, end_handle)
gattc_read(conn_handle, value_handle)
gattc_write(conn_handle, value_handle, data)

All events are raised via the irq handler. On all ports this is a "soft" handler (i.e. you can allocate memory). The events (and their corresponding data tuple) are:

# Peripheral role
IRQ_CENTRAL_CONNECT  (conn_handle, addr_type, addr,)
IRQ_CENTRAL_DISCONNECT  (conn_handle,)
IRQ_CHARACTERISTIC_WRITE  (conn_handle, attr_handle,)

# Central role
IRQ_SCAN_RESULT  (addr_type, addr, connectable, rssi, adv_data,)
IRQ_SCAN_COMPLETE  
IRQ_PERIPHERAL_CONNECT  (conn_handle, addr_type, addr,)
IRQ_PERIPHERAL_DISCONNECT  (conn_handle,)
IRQ_PERIPHERAL_SERVICE_RESULT  (conn_handle, start_handle, end_handle, uuid,)
IRQ_PERIPHERAL_CHARACTERISTIC_RESULT  (conn_handle, def_handle, value_handle, properties, uuid,)
IRQ_PERIPHERAL_DESCRIPTOR_RESULT  (conn_handle, descriptor_handle, uuid,)
IRQ_PERIPHERAL_READ_RESULT  (conn_handle, value_handle, char_data,)
IRQ_PERIPHERAL_WRITE_STATUS  (conn_handle, value_handle, status,)
IRQ_PERIPHERAL_NOTIFY  (conn_handle, value_handle, notify_data,)
IRQ_PERIPHERAL_INDICATE  (conn_handle, value_handle, notify_data,)

As a peripheral (i.e. GATT server), you can use the gatts_* methods to read/write/notify/indicate a characteristic (using the characteristic handle returned from gatts_add_service, and optionally a conn_handle from IRQ_CENTRAL_CONNECT)

As a central, you can obtain a connection handle and characteristic handle using the GATT client methods (gattc_*) and use gattc_read/write, or receive notifications by IRQ_PERIPHERAL_NOTIFY.

@jimmo jimmo force-pushed the jimmo:common-ble-api branch from 2df9bec to 260bf77 Aug 28, 2019

@andrewleech

This comment has been minimized.

Copy link
Contributor

commented Aug 28, 2019

If you want a reference to look at re BT classic api compatibility, zephyr supports some classic in their stack, so perhaps their application api would be useful:
https://docs.zephyrproject.org/1.12.0/subsystems/bluetooth/bluetooth.html
https://docs.zephyrproject.org/latest/guides/bluetooth/overview.html

https://docs.zephyrproject.org/latest/reference/bluetooth/index.html

While that doesn't list a2dp there is code for it in the repo: https://docs.zephyrproject.org/apidoc/latest/a2dp_8h.html

As first skim though, it looks like there's a separate C interface/functions for each classic profile, which sits alongside the ble gattc etc top level functions. There may not be much compatibility/interacting between them at all.

@dpgeorge

This comment has been minimized.

Copy link
Member

commented Aug 30, 2019

Thanks @jimmo for the hard work. I'm fully supportive of the approach to the API that is proposed here, building and improving on the work done by @aykevl .

The API is flat (a set of functions, no classes), minimal and powerful, reflecting closely the actual BLE spec. This means it can be implemented with minimal code, yet allows to do (almost) anything that the BLE spec allows.

Using the API here for the core functionality, it's then possible to write wrappers in Python to expose a more user-friendly API, eg to provide a simple way to make peripherals like heart-rate monitors. It's also possible to make wrappers to provide compatibility with other BLE APIs, eg to match how BLE would work on unix (compatibility with bluepy or pygatt), or on CircuitPython.

@andrewleech

This comment has been minimized.

Copy link
Contributor

commented Sep 1, 2019

Hi @jimmo, Have you already written some example helper functions you mentioned for advertising data?
adv_encode_ble_only() + adv_encode_name('pybd-jimmo')

STATIC mp_obj_t bluetooth_active(size_t n_args, const mp_obj_t *args) {
// TODO: Should active(False) clear the IRQ?
//self->irq_handler = mp_const_none;
//self->irq_trigger = 0;

This comment has been minimized.

Copy link
@andrewleech

andrewleech Sep 1, 2019

Contributor

I don't think irq should be cleared.
If it was bluetooth_reset then yes, but something to stop and start the bluetooth stack shouldn't necessarily reset configuration.


void mp_bluetooth_disable(void) {
ble_state = BLE_STATE_OFF;
mp_hal_pin_low(pyb_pin_BT_REG_ON);

This comment has been minimized.

Copy link
@andrewleech

andrewleech Sep 1, 2019

Contributor

This function should possibly also call ble_hs_shutdown()
https://mynewt.apache.org/latest/network/docs/ble_hs/ble_att.html#c.ble_hs_shutdown

This line could be surrounded by a #ifdef pyb_pin_BT_REG_ON for non-pybd boards that don't have such an enable pin.

}

bool mp_bluetooth_is_enabled(void) {
return ble_state == BLE_STATE_ACTIVE;

This comment has been minimized.

Copy link
@andrewleech

andrewleech Sep 1, 2019

Contributor

There's a nimble function ble_hs_is_enabled() that looks like it would fit here too: https://mynewt.apache.org/latest/network/docs/ble_hs/ble_att.html#c.ble_hs_is_enabled

}

void nimble_uart_process(void) {
int host_wake = mp_hal_pin_read(pyb_pin_BT_HOST_WAKE);

This comment has been minimized.

Copy link
@andrewleech

andrewleech Sep 1, 2019

Contributor

I suspect the host_wake pin functionality would possibly be better in the cywbt file, I think this kind of hardware interrupt pin might be unique to the cyw chip.
Zephyr HCI controller at least doesn't appear to have anything like this, nor have I been able to find any other uart hci radio's with this. They all seem to rely on just the uart port.

@andrewleech

This comment has been minimized.

Copy link
Contributor

commented Sep 1, 2019

@jimmo The changes I made to also support generic HCI radio (nrf52 running zephyr) are on my branch here, on top of your changes: https://github.com/andrewleech/micropython/tree/common-ble-api-anl

entry = MP_OBJ_TO_PTR(elem->value);
os_mbuf_append(ctxt->om, entry->data, entry->data_len);

return 0;

This comment has been minimized.

Copy link
@andrewleech

andrewleech Sep 2, 2019

Contributor

It's common on other bluetooth stacks to allow for a callback here when a read request comes in. On NRF devices this is exposed as a callback event BLE_GATTS_EVT_RW_AUTHORIZE_REQUEST.

It's officially used to provide the application with a chance to authorise or deny access to the characteristic based on whether it's paired etc.

It's also commonly used to allow the application to change the value of the characteristic each read. This can be then used to stream data out the characteristic reliably, instead of notifications which are far more unreliable as they don't have the same handshaking to ensure they get through.

The problem in this case is your other events are soft handlers, whereas this would need to be hard / inline to allow the callback to change the return value of this function.

@andrewleech

This comment has been minimized.

Copy link
Contributor

commented Sep 2, 2019

It appears to only support one service currently?

If I run bt.gatts_add_service() multiple times for each service, I get printouts for for each svc & chr

gatts_register_cb: svc uuid=d0710dac handle=1
gatts_register_cb: chr uuid=d0710d58 def_handle=2 val_handle=3
gatts_register_cb: svc uuid=d071155c handle=5
gatts_register_cb: chr uuid=d0711448 def_handle=6 val_handle=7
gatts_register_cb: chr uuid=d0711460 def_handle=8 val_handle=9
gatts_register_cb: chr uuid=d0711478 def_handle=10 val_handle=11
gatts_register_cb: chr uuid=d0711490 def_handle=12 val_handle=13
gatts_register_cb: chr uuid=d07114a8 def_handle=14 val_handle=15
gatts_register_cb: chr uuid=d07114c0 def_handle=16 val_handle=17

But only the last service registered shows on mv nrf ble scanner app once connected.
If I reverse the order they're configured, I get the other one shown.

@andrewleech

This comment has been minimized.

Copy link
Contributor

commented Sep 2, 2019

FYI I've made a quick python util function to create appropriate advertising data to include device name. I'll eventually extend it to handle services and scan response as well, but for now I've chucked a snapshot here: https://gist.github.com/andrewleech/49c2a54313f57e544dc1b3a3360d6dc3

@andrewleech

This comment has been minimized.

Copy link
Contributor

commented Sep 4, 2019

I've added support for multiple services to the head of my extension branch, see in andrewleech@72b2d17
I do have quite a few other smaller changes on that branch too: https://github.com/andrewleech/micropython/commits/common-ble-api-anl

It required a fair bit of turning the code around, not sure it's the best option.
Now, Bluetooth().active(True) must only be called after all your services have been registered, this is what triggers the stack startup.
Bluetooth().handles() can then be called to get a dict of all the service uuid's keyed to a tuple of char uuids with a buffer of the matching handles.

It's quite possibly better to make the one add service function take a larger structure of all the services and their matching chars in the one call, rather than having the extra handles tracking dict I've got in my attempt.

@rf-ir-learner

This comment has been minimized.

Copy link

commented Sep 7, 2019

Hello, I am new to BLE on ESP32 and I'm wondering if I could get some clarification on this. I am quite excited by the prospect of being able to use BLE with Micropython, since so many of the sensors these days seem to have really long battery life with BLE (like a year or more on a button battery).

It seems like we can use some basic BLE but I'm a bit confused as to how it is implemented. I need to set up an ESP32 board as a client to monitor a few BLE sensors, nothing terribly complicated.

So is BLE contained within a version of Micropython that I can just burn and install on my board? If so, is there a particular version (link?) I need to download?

Or is the BLE component a separate library module that I would install in the LIB sub-folder on my ESP32? If so, is there a link for that?

Or is there some other thing I would have to do (compile something)?

Any help around this would be greatly appreciated; these are probably really stupid questions, but I really am confused.

Regards, AB

@andrewleech

This comment has been minimized.

Copy link
Contributor

commented Sep 8, 2019

Hi @rf-ir-learner, this PR doesn't yet have support for esp32, the stm32 support has been proofed out first.

I know Esp32 and others are being worked on, but for a feature as important as Bluetooth @jimmo and others are putting in a lot of effort behind the scenes planning it all to try to ensure its done right the first time.

@dpgeorge

This comment has been minimized.

Copy link
Member

commented Sep 9, 2019

It's quite possibly better to make the one add service function take a larger structure of all the services and their matching chars in the one call

It's probably not wise to rely on the underlying BLE stack (nimble, nordic, bluedroid) being able to add services "later on". In other words, the common denominator here would be to assume that all services must be registered at the same time.

In that case gatts_add_service() should be renamed to gatts_register_services() (or gatss_set_services) and take a list of what gatts_add_services previously accepted. You could call gatts_register_services more than once (and after bt.active(1)) but each call would completely reset the available service list. There could even be an extension in the future which didn't reset the list, eg gatts_register_services(..., reset=False).

Alternatively, it might be possible to keep gatts_add_service() and automatically detect when all services were added, eg:

gatts_add_service(...) # first call to this func, so reset gatts before adding
gatts_add_service(...) # accumulate service
advertise(...) # service list must be done so register them, then advertise

But that seems a bit flaky.

@jimmo jimmo force-pushed the jimmo:common-ble-api branch from 260bf77 to 00f3e37 Sep 9, 2019

@dhalbert

This comment has been minimized.

Copy link

commented Sep 9, 2019

the common denominator here would be to assume that all services must be registered at the same time.

I recently went through some API churn on this with bleio (now _bleio) for CircuitPython. My original API had all services passed at once to a peripheral, as @dpgeorge suggests. The tree of services, their characteristics, and the characteristic's descriptors was built up all at once and then passed in. As background, note that we are supporting peripheral and central mode, peripherals as clients, not just servers, and pairing/bonding.

One problem I encountered with this approach was that Service, Characteristic, and Descriptor had conventional constructors. So there would loose, unconnected objects during service tree construction. A user of the API might inadvertently reuse one of those attribute objects when constructing multiple services. To prevent that I ended up dropping the all-at-once addition and creating factory methods like Service.add_to_peripheral(), so that there were no unattached objects. I'm not completely happy with this either, but it's safer, and it removed the need for some bad-state checking internally. Here's the PR for that, with discussion: adafruit#2092, and here is a snapshot of a complicated use of the API, to build up a BLE HID service.

@tannewt has suggested that the service/characteristic/descriptor tree be described in a more declarative way, usable for both local and remote attributes. These service trees would then be passed to a peripheral or central object. In the case of a peripheral, the tree components would be instantiated and registered with the BLE stack. In the case of a central, the user would, for example, use a property in the declarative tree to set or get the value of a remote characteristic, and the internal logic would check that a matching characteristic had been found during discovery.

Because a BLE stack is so complicated, we decided to name the low-level module _bleio to indicate it should not be used by ordinary users, and so we can change its API between major versions of CircuitPython. We have a library layer (https://github.com/adafruit/Adafruit_CircuitPython_BLE) that will hide the details and provide common services and functionality.

(I could critique the BLE protocol, which has a multitude of role-changing chameleon objects: it is more of an n-dimensional basket-weave of orthogonal concepts than a "stack". But it is what it is and we have to figure out how to present it in a usable way.)

@jimmo

This comment has been minimized.

Copy link
Contributor Author

commented Sep 9, 2019

Thank you for the feedback @andrewleech & @dpgeorge.

I have updated this branch, incorporating all of the improvements from @andrewleech (with some modifications). I have also re-implemented adding multiple services based on @dpgeorge's suggestion. An example of adding multiple services now looks like:

bt = bluetooth.Bluetooth()
bt.active(True)

UART_SERVICE = bluetooth.UUID('6E400001-B5A3-F393-E0A9-E50E24DCCA9E')
UART_TX = (bluetooth.UUID('6E400003-B5A3-F393-E0A9-E50E24DCCA9E'), bluetooth.FLAG_READ|bluetooth.FLAG_NOTIFY,)
UART_RX = (bluetooth.UUID('6E400002-B5A3-F393-E0A9-E50E24DCCA9E'), bluetooth.FLAG_WRITE,)
HR_SERVICE = bluetooth.UUID(0x180D)
HR_CHAR = (bluetooth.UUID(0x2A37), bluetooth.FLAG_READ|bluetooth.FLAG_NOTIFY,)

((hr,), (uart_tx, uart_rx,),) = bt.gatts_register_services(
                                          (
                                           (HR_SERVICE, (HR_CHAR,),),
                                           (UART_SERVICE, (UART_TX, UART_RX,),),
                                          )
                                         )

This allows you to modify the service list multiple times.

@dhalbert

This comment has been minimized.

Copy link

commented Sep 9, 2019

@jimmo I see our comments just crossed, and that you create the tree in the call. If you add descriptors at some point, then you'll need further nesting in the tuple of tuples.

@dhalbert

This comment has been minimized.

Copy link

commented Sep 9, 2019

For something like BLE HID, I have also had to add specification of security modes, and values for an attributes for an attribute's max length and whether it is fixed length. I also added the ability to specify an initial value for an attribute, which is common for read-only descriptor values.

@jimmo

This comment has been minimized.

Copy link
Contributor Author

commented Sep 9, 2019

@jimmo I see our comments just crossed, and that you create the tree in the call. If you add descriptors at some point, then you'll need further nesting in the tuple of tuples.

Thanks for the feedback also @dhalbert -- sorry I just missed your first comment. Yes, descriptors would be an optional third entry in the characteristic tuple (and in the returned value handles tree), but this does start to get a bit messy. Not to mention other properties (default values, security, etc). (Edit: just saw your next comment :) Some thought required here - I will follow up with @dpgeorge )

We have some additional constraints due to using Nimble as the BLE stack, which requires all the services to be defined at once. See apache/mynewt-nimble#556 I need to spend some more time looking into your implementation, but I'm not sure that it would be possible currently to provide the _bleio API on top of Nimble.

Obviously we'd prefer not to be forced into a particular API design by the underlying stack...

Here's the PR for that, with discussion: adafruit#2092, and here is a snapshot of a complicated use of the API, to build up a BLE HID service.

Thank you -- this is really useful!

Because a BLE stack is so complicated, we decided to name the low-level module _bleio to indicate it should not be used by ordinary users, and so we can change its API between major versions of CircuitPython. We have a library layer (https://github.com/adafruit/Adafruit_CircuitPython_BLE) that will hide the details and provide common services and functionality.

This was our intention too (this is our low-level API). Our goal initially was to avoid any classes at all (hence the tuples) and then match the BLE gatts/gattc operation directly. I hope it should be still possible to make this match at the high-level API.

(I could critique the BLE protocol, which has a multitude of role-changing chameleon objects: it is more of an n-dimensional basket-weave of orthogonal concepts than a "stack". But it is what it is and we have to figure out how to present it in a usable way.)

:) Helps me feel better about how long this is taking!

@dhalbert

This comment has been minimized.

Copy link

commented Sep 9, 2019

@jimmo Thanks for your reply and for your work on this API. Your comments and approach have also given me some new ideas.

Our goal initially was to avoid any classes at all (hence the tuples) and then match the BLE gatts/gattc operation directly.

It's not so far from positional tuples to namedtuples to declarative classes: I think you may end up moving in that direction for convenience and readability as you need to add yet more properties for the attributes.

We have some additional constraints due to using Nimble as the BLE stack, which requires all the services to be defined at once.

I only found out about the existence of Nimble midway through working on bleio. That's an important potential constraint.

I think a key thing may be to separate the declarative and operational attribute classes. I have been assuming they would be the same, but it causes more state to need to be kept internally to track the object life cycle, and whether it is local or remote. If you look at something like the iOS BLE API, there's almost a Cartesian product of classes for objects and their roles, and maybe we could avoid that.

@jimmo jimmo force-pushed the jimmo:common-ble-api branch from 00f3e37 to f3b4e4b Sep 9, 2019

@jimmo

This comment has been minimized.

Copy link
Contributor Author

commented Sep 9, 2019

I've added another commit to address Nimble memory management (root pointers are tracked for nimble malloc, and it works across soft reset).

@jimmo jimmo force-pushed the jimmo:common-ble-api branch 4 times, most recently from af7ccc1 to d6e3675 Sep 10, 2019

@jimmo

This comment has been minimized.

Copy link
Contributor Author

commented Sep 11, 2019

I've implemented descriptor support and basic support for (gatts) writing of longer values. Fixed a few other crashes and cleaned up some debugging/logging code.

Now able to make the PYBD act as a HID keyboard for my Android phone.

@andrewleech

This comment has been minimized.

Copy link
Contributor

commented Sep 11, 2019

The other thing I haven't looked at yet is pairing & bonding.

"Just works" Pairing will likely work already, though I'm not sure about nimble's "LE Secure" support for using it safely. For other pairing (eg. Pin) another immediate/interrupt style event will likely be needed.

Bonding support will presumably need since functionality exposed/wired up to write the bonding info to flash.

If anyone else if a little lost in the wild range of different ble security options, this blog is a good primer: https://medium.com/rtone-iot-security/deep-dive-into-bluetooth-le-security-d2301d640bfc

extmod/modbluetooth.c Outdated Show resolved Hide resolved
extmod/modbluetooth.c Outdated Show resolved Hide resolved
extmod/modbluetooth.c Outdated Show resolved Hide resolved
extmod/modbluetooth.c Outdated Show resolved Hide resolved
extmod/modbluetooth.c Outdated Show resolved Hide resolved
extmod/modbluetooth.c Outdated Show resolved Hide resolved
extmod/modbluetooth.c Outdated Show resolved Hide resolved
extmod/modbluetooth.c Outdated Show resolved Hide resolved
extmod/modbluetooth.c Outdated Show resolved Hide resolved
extmod/modbluetooth.h Outdated Show resolved Hide resolved

@jimmo jimmo force-pushed the jimmo:common-ble-api branch from d6e3675 to 84ee7a3 Sep 16, 2019

@jimmo

This comment has been minimized.

Copy link
Contributor Author

commented Sep 16, 2019

OK thanks for the review feedback - I've addressed all those points exactly as suggested and updated the branch.

Note that the event numbers are now shifted right by one so they correctly start at 1 (oops), so Python code will need to be updated.

@jimmo jimmo force-pushed the jimmo:common-ble-api branch from 84ee7a3 to 18c27b2 Sep 16, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
6 participants
You can’t perform that action at this time.