Create/Refactor BLE Subsystem In CoreTools

[timburke]

Overview

This is a metaepic describing a high-level goal to modernize the way CoreTools interacts with IOTile devices of Bluetooth Low Energy. It has the following large-scale goals:

1. To modernize the BLED112 device adapter (iotile-transport-bled112) to use asyncio and have well tested, cancelable operations based on coroutines.
2. To permit us to remove legacy DeviceAdapter support from CoreTools since the BLED112DeviceAdapter is the last DeviceAdapter that uses it.
3. To decouple the TileBus protocol implementation over BLE generically from the specific implementation on top of a BLED112 dongle so that we are positioned to support other BLE hardware such as the native BLE inside Mac OS as well as BlueZ on linux.
4. To properly support at least one alternative BLE implementation beyond the BLED112 dongle, which is now an old, legacy product.

Current Implementation

As it stands, we have a tightly coupled implementation where all BLE concepts are linked with their specific implementation inside the BLED112 dongle. There is no generic "BLE abstraction layer". Furthermore, the BLED112 device adapter was written before we moved to asyncio for concurrency management so it based on threading. In particular, it has 3 threads involved in all process:

• The calling thread. This thread makes nonblocking calls to the legacy x_async() DeviceAdapter API to take bluetooth actions. These actions are queued for work on the background BLED112CMDProcessor thread.
• The BLED112CMDProcessor thread synchronizes all activity on the BLED112 dongle in the form of a work queue that it works through synchronously, one item at a time. There is very limited concurrency support. During a connect operation that may take 2-3 seconds, no other operations can be performed on other connections, even though the underlying dongle supports them.
• A background reader thread that blocked on read from the underlying bled112 dongle and emits packets read into a queue that is used to advance the state machine inside BLED112CMDProcessor based on what operation it is trying to perform.

New Implementation

The high level new implementation concept is to have:

• Everything be asyncio based using coroutines to allow for cancelable operations and remove the multiple threads that we currently use.
• A generic interface for what a BLE Central (AbstractBLECentral) dongle provides (scan, connect, read handle, write hadnle, etc) that permits the creation of a generic BLE Device Adapter.
• A specific implementation of this AbstractBLECentral for the BLED112 dongle
• A specific emulated implementation, EmulatedBLECentral, that allows for testing any bluetooth related functionality at the bluetooth level, i.e. what if a connect times out, what if GATT handle write fails, etc).
• The ability to easily produce device adapters for other BLE stacks such as the native ble stack on Mac OS by writing another AbstractBLECentral subclass.

Implementation Phases

Since this will be a fairly large refactor, we need to be clear about intermediate milestones that can add value on their own and how we can structure the code changes to be continuously mergeable into master without breaking existing BLED112 functionality until we're ready to flip off the old threaded code and move over to the new asyncio code.

So, I propose the following 3 step plan:

1. Collect all general bluetooth functionality into a new package iotile-transport-blelib, similar to iotile-transport-socketlib. We can commit freely into this package since it hasn't been released and has no compatibility guarantees.
2. As we rewrite the BLED112 device adapter on top of asyncio and iotile-transport-blelib, put it adjacent to the current device adapter and provide access to it on a separate entry_point such as asyncbled112. This allows us to release it for testing without breaking the bled112 existing bled112 device adapter.
3. Once asyncbled112 is fully tested and ready to go, we swap the entry_point for bled112 to point to the new implementation and delete the old one as well as the legacy device adapter support code that we left in iotile-core to support it.

The goal of the 3 step plan is to avoid needing to maintain any long-running branches during this process and keeping iotile-transport-bled112 always in a releasable state so that we are able to provide incremental asyncbled112 releases for testing.

[timothyfroehlich]

I think we need to implement filters as early as possible and as efficiently in the code. Filters can include filtering for non-iotile packets, deduplication (with a configurable rate for allowing duplicate packets through), a "capture broadcasts only" mode to filter out non-broadcasts, and perhaps black or whitelisting.

Early deduplication is probably the most important of those.