BLE stack change proposal to scale for many devices

[reallew]

At the moment, the BLE stack supports only 4 devices max. becauses of the nature of the P2P connections.

Proposal:

• Let's send all broadcast packages as 'Extended Advertising' (BLE 5.0+, AUX_ADV_IND) packages. That is broadcasting, too, the packages are big enough for LoRa, and this works with infinite devices in the same room. Drawback: works only with devices with BLE 5 and newer.
  (I have ideas for a fallback for older devices, but I would support only newer devices first).
• P2P BLE connections are created only when there are actually packages for that destination to deliver.
• Link-awareness: if there is a link, a BLE P2P connection stays open until the link is closed.
• Idle-timeout for single packages. For a pure single package connection, the BLE P2P connection is closed after 2 minutes of silence.

[torlando-tech]

I am open to designing a new, separate BLE protocol. I have a few questions regarding your wording, I want to be sure I understand your proposals.

By packages, do you mean packets?
By P2P BLE, you are referring to the existing BLEInterafce as it is today?

It seems like you are suggesting something similar to how the RNode interface works today, by broadcasting packets instead of establishing connections. This alone would be much simpler than the BLE protocol as it exists today.

I think combining this new BLE Broadcast protocol with the existing P2P protocol in the ways you suggest is interesting but very complex, probably more complex than necessary. Most of all I think this would break the existing Transport/Interface abstraction and separation.

P2P BLE connections are created only when there are actually packages for that destination to deliver.
Wouldn't this require Transport to send commands to the Interface that it doesn't currently have? The Interface would have to know 1) that it could open a peer connection with another device, so it must keep track of the devices that are in range, and 2) instruct the interface to open that connection? Wouldn't this require keeping a sort of separate path table of paths that could be established but currently aren't?

Link-awareness: if there is a link, a BLE P2P connection stays open until the link is closed.
The Interface layer manages connection state, and is simply sent packets to transmit or returns packets that are received. I don't think the Interface knows (or needs to care) whether a given byte array is for a link or not.

The extended advertising is something I looked into early on, before I started Columba I was only working on a linux version of ble-interface, and at that time my only test hardware was a pair of raspberry pi zero Ws, which don't support BLE 5.0. It seemed like, at the time, BLE 5.0 support was inconsistent at best, so I elected not to rely on it. If the situation has changed since then, I do think it might be worthwhile to create a separate BLE Broadcast interface though.

[reallew]

By packages, do you mean packets?

Yes. Sorry for the wrong wording.

By P2P BLE, you are referring to the existing BLEInterafce as it is today?

Yes. Everything is a P2P connection, no broadcasting, that's why I called it P2P BLE.

It seems like you are suggesting something similar to how the RNode interface works today

I propose a hybrid approach. Broadcast packets (mostly announces) like RNode traffic (broadcast radio in the air for all). But on-demand-P2P for packets addressed to a single destination.

but very complex, probably more complex than necessary.

The hybrid splits into two independent planes. Broadcast plane is essentially RNode-style: fire-and-forget advertising, scan, dispatch announces. P2P plane is essentially today's BLEPeerInterface, just created lazily on first packet to a peer and torn down on idle. They share only the BLE-peer identity table. Each plane in isolation is simpler than today's protocol. Each remote BLE peer is still represented as a BLEPeerInterface to Transport, just as today. Extended Advertising sits purely below the interface boundary. The Transport/Interface separation stays exactly where it is.

Wouldn't this require Transport to send commands to the Interface that it doesn't currently have?

No. Transport just sends packets to the interface like before. 'Do I need to reopen the GATT connection or is it still open?' is something the interface decides for itself. And yes, the BLE interface would need a table of 'which Reticulum destination goes to which BLE device (MAC)'. But that table already exists. KotlinBLEBridge has addressToIdentity, identityToAddress, pendingConnections. The Python BLEInterface mirrors them. We don't add a table, we just change when entries get created: today on GATT connect, in the new design also on advertising-beacon reception.

The Interface layer manages connection state, and is simply sent packets to transmit or returns packets that are received. I don't think the Interface knows (or needs to care) whether a given byte array is for a link or not.

That's correct. The transport layer knows about links and could expose this info easily to an interface. But you are right to mean we shouldn't do that. Just because we don't need. KISS. A simple timeout would have the same effect. Let's just drop the BLE connection after n seconds of radio silence.

It seemed like, at the time, BLE 5.0 support was inconsistent at best.

My Reticulum devices running transport daemons: Notebook from 2019, Android phone from 2022, Heltec V4, Wismesh Pocket V2, Heltec T114. They all talk BLE 5.0. I think it would be a difficult quest to find a device younger than 2017 that is not BLE 5-able.

Btw, I hope that Leviculum will make the use of Pis obsolete, when my Firmware is stable enough for all the embedded LoRa devices out there.

[torlando-tech]

Thanks for the extra details. In case the current issue count isn't enough of an indicator, it will probably be a while before I'm able to really think about this deeply enough for long enough to build something meaningful. It is an interesting idea though, and is likely to occupy space in my brain now until I've thought it through.

[jrl290]

Ok, just to add my two cents because just like you two, BLE mesh seemed deliciously appealing to me as well

The biggest problem I see with each the hybrid "advertisement=announce, GATT=data" is that it still requires active phone radios in the background in order to carry the mesh traffic. While you can do that on Android, it is just too battery draining to consider viable

What I am considering however, is that RTNode would make very good use of the idea for completely configuration-less access since it is generally powered and stationary anyway

[reallew]

I thought about it more. My current idea is to implement ble-leviculum, a BLE5 interface that uses BLE5 broacasts only for everything, like the Lora interface broadcasts everything. This can live in addition to Columba's ble-reticulum interface.
I want to implement both as part of Leviculum. If ble-leviculum works well in my test setups, I'll propose to add it to Columba.
My (ai-written, sorry) notes of brainstorming details can be found here: https://codeberg.org/Lew_Palm/leviculum/src/branch/master/docs/src/concepts/bluetooth-interfaces.md