[Feature] Enable btstack in the virtualhub

[laurensvalk]

BTstack can be configured for Linux or OS X with a USB Bluetooth dongle. This should let us use it with the Virtual Hub. This would make local development much easier, and also help get our abstractions right from the get-go, both for classic and BLE.

The way I read it, an instance directly controls the HCI of the USB dongle rather than going through the host OS, so we'd get to test actual "brick-to-brick" communication too, on a single machine. I think this would even let us test and debug talking to Pybricks Code locally, which is quite a feat.

---

The Virtual Hub already has a "bluetooth driver", which currently just writes stdout to local stdout. But we can change it to use the existing bluetooth_btstack with posix hooks instead of uart platform hooks.

This would be a bit of work upfront, but if it can work like above it could be a big time saver overall. What do you think @dlech, @jaguilar?

[jaguilar]

It's an interesting concept. It would likely be less work than what we had to do for ev3 or stm32 because btstack has a platform for libusb, which is less troublesome than having to set up our own transport as we do in Pybricks.

Hub-to-hub on a PC -- is the idea that we'd have two bluetooth dongles and configure each instance of virtual hub to use a separate one? It would be a pretty niche setup but, yeah, I think something like that could work. I'm not positive exactly how much better/easier that would be than having a true HIL setup for testing (i.e. flash two bricks with firmware then send over test programs), in terms of the fidelity/cost tradeoff.

[dlech]

Yeah, I expect it would be just a matter of swapping out the HCI hook function. Would be interesting to do eventually.

[laurensvalk]

Hub-to-hub on a PC -- is the idea that we'd have two bluetooth dongles and configure each instance of virtual hub to use a separate one?

That, or PC to EV3. Or gamepads when we get around to them, etc. Also new BLE stuff for the other hubs.

Really anything that involves the high level stuff. E.g. the EV3 mailbox API we'd be doing next. Things like the crash in the MicroPython async code you ran into earlier would trivially give you a full stacktrace.

I used to believe it wasn't worth the time, but have already found a few (non-bluetooth) issues using the Virtual Hub with far fewer headaches 😄.

[jaguilar]

I investigated this somewhat further. I think the most straightforward thing to do would be to copy a chunk of the initialization code in port/libusb/main.c. This code serves a similar purpose to the cc256x init code I added in pybricks/pybricks-micropython#405 -- it tries to detect the manufacturer and version of the device, and load the appropriate init script. I think this is all relatively straightforward but it would require y'all being okay with the license at the top of that file (non-commercial, attribution).

If y'all aren't okay with that license, we could rewrite the code. It's a little annoying but not impossible.

[laurensvalk]

Thanks for looking into this. We're already using BTstack, so I don't think using (the contents of) one more file for this purpose is an issue, especially if it is mostly contained in one file similar to the other init code.

It would be interesting to see if we can spin this up with our existing bluetooth_btstack code to start with. If this all works out I think you have convinced me that bluetooth_classic does indeed not need to go in a separate driver 😄.

[laurensvalk]

FWIW: if you use Visual Studio Code you can launch the Virtual Hub by selecting virtualhub on the debug tab and just press F5. You can review tasks.json and launch.json to find the respective commands if you prefer other tools.

By default it runs:

            "args": [
                "${workspaceFolder}/tests/virtualhub/basics/hello.py",
            ],

You can modify that or delete it to access the REPL instead.

---

Optionally, you can also run (requires activating the virtual environment in the repo):

./lib/pbio/platform/virtual_hub/animation.py

to get a live visual with motors that can move. You can leave this on even as you restart the virtualhub/debugger.

[jaguilar]

I have done some more investigation. Takeaways:

1. Getting the C code into okay shape won't be that hard. It's just a matter of adding a packet handler that listens for the local version information and manufacturer information, and then sets the correct chipset from that.
2. Getting the thing to reliably support the constellation of bluetooth dongles out there will be less fun.

In a similar way as we need the cc256x init scripts for the Lego bricks, Bluetooth dongles also require initialization with init scripts. There is no one universal source of truth for these scripts. The broadcom firmwares are only shipped via Windows Update. Some hackers collect these init scripts.. The realtek and intel firmware are generally located in the linux-firmware git repo and can be apt-get installed to Debian systems. Alternatively, they can be fetched directly from the corresponding git repo.

Note that we can't redistribute at least the broadcom firmwares because we do not have a license to do so.

Okay, so, practically, what to do? I think we could write a script to download these firmware into the user's .cache directory. This means that we are not redistributing these firmwares. We could use something like .cache/pybricks/virtualhub/btfw/... We would add a script to sparse checkout these repos into the cache. It's worth noting that the names of the broadcom firmwares also have to be massaged to match what btstack expects, so there's slightly more to do than just checking things out, but the principle is clear enough.

With the cache directory populated by the relevant firmware, we should be able to interface with a decently broad spectrum of bluetooth dongles, though by no means all of them.

Does this sound okay to y'all? Also do we care about getting this to work on Windows, because that is a whole 'nother kettle of fish.

[dlech]

I think we should start reselling a BLE dongle that doesn't require firmware to be loaded every time you use it. 😄

I wouldn't put too much effort into making this work with arbitrary BLE dongles. Personally, I would use it with something like an nRF dongle that is meant to be used for something like this and just shows up as a serial port on the computer.

[laurensvalk]

Right, the goal here is to make development easier; not release it as its own thing. So one particular dongle/device would be just fine. It sounds like @dlech has a good recommendation? We can sponsor one for @jaguilar.

Does this sound okay to y'all? Also do we care about getting this to work on Windows, because that is a whole 'nother kettle of fish.

So as above, just something that us three can use will be fine. I'm mostly on Ubuntu myself. What do you use?