BTHome does not include Illuminance Measurements

[jhbruhn]

No description provided.

[rbaron]

That's nice that you used BTHome. How was the experience so far?

Yes, unfortunately there wasn't enough space in the advertising packet for all sensors in this first implementation. We have a few bytes left in the BTHome encoding, but I'm reserving those to add a counter/packet ID for deduplication if needed.

For now, if you really need illuminance, you can either update the encoding code and trade it off for some other sensor or use the default b-parasite BLE encoding + b-parasite component on ESPHome. There might be a workaround for BTHome by using less bytes per sensor, although the fixed factors make it more challenging.

The reason I'm reluctant to use extended advertising or scan response for now are increased power consumption and for Coded PHY/Long range we don't have that option as far as I understand.

[jhbruhn]

Ah sorry, I was going to make the description more descriptive, but the message is clear anyways.

I really like the proposition of BTHome, that's why I flashed all my devices with BTHome enabled. Currently there is a bug (home-assistant/core#78702) which is marking the sensors as unavailable to quickly, but in general it is working.

I modified the firmware to also transmit the illuminance value by extending the advertisement-data as needed. Interestingly that is working flawlessly. Did the softdevice enable extended advertising automatically?

[jhbruhn]

Had another look at my code, and I am using exactly 20 bytes with the illuminance data added. To add a packet_id, we need another 3 bytes.

We could gain two bytes by reducing the precision of the humidity and moisture to 8 bit which IMO is enough for the percentage values, especially the moisture. Perhaps the battery voltage could also be reduced to 8 bit? Might be too imprecise though. Or we calculate a percentage from the battery voltage, but that is a whole other problem.

[rbaron]

The whole packet must have at most 31 bytes. I've noticed the SDK encodes the adv elements like flags + service data + name ("prst"). If the service data is too big, the name starts to get chopped off. Setting PRST_BLE_DEBUG to 1 lets us see the final packet and this behavior.

For BTHome, reducing the precision was my first idea, but I'm not sure we can, unless I'm misreading bthome.io. Take humidity for example:

Let's say we use 1 byte for the value, instead of the preferred 2 bytes. It means we can send [0, 255], but the factor is fixed at 0.01, which means HA would interpret these as [0, 2.55%]. Unless I'm missing something, the factor would also have to be conditioned to the length, but it isn't. If the factor was adjusted implicitly to 1, we'd be set, but I don't think that happens.

Precision-wise, I think it would be totally fine to have 1 byte for humidity and 1 byte for voltage in BTHome.

[jhbruhn]

You're right, BTHome requires b-parasite to use at 16-bit integers to transmit those values. That's an unfortunate decision, especially for measurements which are always a percentage value.

But on the other hand, not including a packet_id does seem to work for the HomeAssistant BTHome parser. Either the duplicate measurements are actually filtered out, or they are not visible. And for this application we do not get any problems should there actually be duplicate measurements be made, as they can easily be filtered on the receiving end.

[Ernst79]

BTHome is open to all suggestions, of course 😄 . Problem with making the factor also "variable" is that I need an extra byte or bit to be able to tell BTHome what it is. If we need an extra byte to tell the factor, there is no gain in number of used bytes.

Are you sure the name is included in these 31 bytes? I though the local name (either 0x08 or 0x09) are different parts of the message, not sure if the 31 bytes is for the service data or for the entire payload.

Edit: looks like you are right, when checking this page https://stackoverflow.com/questions/33535404/whats-the-maximum-length-of-a-ble-manufacturer-specific-data-ad#:~:text=A%20BLE%20advertisement%20can%20send%20out%2031%20bytes%20at%20max.

You can try to remove the "flags (typically 020106) to save three bytes, if I read one of the comments. BTHome isn't doing anything with this flags bytes. I thought it was mandatory, but according to one of the comments it isn't.

[jhbruhn]

A more complicated but IMO very efficient solution would be to, only for values which are a percentage like humidity, use a different factor for every integer-size, to normalize the full scale value to the resulting value.

E.g., for a size of uint8_t, the factor is 1/255, for uint16_t the factor is 1/65535 and so on. Thus we can encode percentages with the highest precision available for the values.

Edit: BTW I don't mean to sound too harsh here! I think BTHome is a lovely idea and a step in a very good direction.

[rbaron]

Hey @Ernst79 thanks for joining the discussion and for your work on BTHome.

To clarify, I mean the whole payload can be 31 bytes. With the current implementation, we're at 29 bytes. Here's an annotated sample payload. You're also right, many BLE devices encode much more data in one of the two ways that I know of:

1. Using scan request/response. The scanning device asks for extra data. The sensor device must remain active while this dance happens. Right now we just send out a few packets and go to deep sleep with the radio off to save battery
2. Using extended advertising (BLE 5). Here we broadcast simultaneously in multiple channels and can send up to 255 bytes, but we increase the power consumption by a factor of 3 in my tests (while the radio is on). If we use this, we cannot use the Coded PHY/Long range feature, which also broadcasts on multiple channels

Both of these are valid solutions, but come with tradeoffs. For me, it seems like a high price to pay for the illuminance values, since I use all the sensors inside cases anyway. I would still like to make it easy for users who do want to use them.

About the flags, I'm taking a look at the specs (Supplement 10, section 1.3.1):

The Flags data type shall be included when any of the Flag bits are non-zero and the advertising packet is connectable, otherwise the Flags data type may be omitted.
...
Note: If the Flags AD type is not present in a non-connectable advertisement, the Flags should be considered as unknown and no assumptions should be made by the scanner.

I think we have non-zero flags, and the current value 0x04 signals that this is a LE-only device (no support for Bluetooth classic). I have a feeling it might just work, but we would be technically out of spec and we need to test.

As for the factor, @jhbruhn's suggestion is what I mean by implicitly changing the factor. Right now the factor is implicit, but fixed. A dynamic, implicit scheme would be for example:

• If using 2 bytes, factor is 0.001, allowing for values in [0, 100.00], in 0.01 increments (this is the current scheme for humidity for example)
• If using 1 byte, factor is 1, allowing for values in [0, 100] in increments of 1

[Ernst79]

Regarding your last point, I understand what you are saying. but is would also limit values to 0 till 100. So you can't have 110%, which makes sense for humidity and battery, but for most other sensor types, it doesn't.

What we also can do, is just adding another data type - factor combination. So, we have one for two byte values with factor 0.01, and one with 1 byte value with factor 1. As it is most likely only needed for humidity, moisture and battery, it wouldn't take up too much of sensor types - factor combinations (we are limited to 255).

I'll think of it, what is best to do.

[jhbruhn]

That would also be the most backwards-compatible approach - I like it.

[jhbruhn]

I just had another look at my local changes: In its current state, the advertisement data can not be 20 bytes long, as the resulting advertisement packet is longer than 31 bytes then, and thus truncating the device name to "pr". Not terrible because the name is only there for beauty-reasons, but perhaps we can save enough bytes with the upcoming BT-Home changes to also support that?

That would mean that the packet-id probably can't be integrated due to space-reasons though.

[Ernst79]

I’m working on a v2 version of BTHome now, which doesn’t need the name anymore, but will use a adv_info (1 byte) + optional device_id (2 bytes), which is linked to all device info, like name, manufacturer, model, etc. Also, the packet_id isn’t needed (as HA already can decide on its own if the info has changed or not. Moreover, illuminance moisture and humidity with uint8 and factor 1 have been added. I hope that we can implement this in HA2022.11

[rbaron]

@jhbruhn that was my observation as well, the SDK chops off part of the name if we exceed 31 bytes.

@Ernst79 super happy to see progress here. Here's a positive feedback from a new b-parasite user. My two cents about the 1-byte values.

For humidity (and any %-representable value), 1 byte + factor 1 works well. For illuminance, if measured in lux, the factor 1 makes it tricky. Usual values for lux are (from wiki):

• 0 for complete dark
• 0.002 for full moon-lit night
• 300 for in indoor lighting
• up to 100.000 for direct sunlight

If we use factor 1, we can only represent [0, 255], which is not ideal. I think we would have to find a better way to encode this wide range of values. One idea is to encode the log(lux). It has some issues: -inf for 0 & negative values for lux in [0, 1), but in practice maybe we can solve it with some offsets and factors? Something like:

raw lux	log base 10	log base 10 + 10	10 * (log base 10 + 10)
0.000000001	-9	1	10
0.00002	-4.698970004	5.301029996	53.01029996
300	2.477121255	12.47712125	124.7712125
100000	5	15	150

The last column ranges from [0, 150], which is encodable in 1 byte and is more or less able to represent the full range (of course without much precision).

[Ernst79]

Ah, sorry, I meant moisture (in %), not illuminance. My fault. Illuminance doesn't make sense, I agree

[Ernst79]

@rbaron We have just released BTHome V2. some important changes that could be helpful for b-parasite are

• the length and format byte in each measurement is removed, length and format are now fixed. This saves 1 byte each for each measurement. Drawback is that you have to stick to the format and number of bytes, as given in the table on the website. In case it does not meet your needs, we can add a different format, of course (with a different object_id). Just make an issue on github
• We now have a paid UUID (0xFCD2) that is only for BTHome. This means that we are all legal now, not "misusing" a general UUID.
• We have added one mandatory byte in the beginning to indicate version (V2) and encryption (True/False).

In the end the advertisement should become shorter with multiple measurements.

Planning is that V2 will be supported in HA 2022.12, next month.

[rbaron]

@Ernst79 thanks so much for the heads up, these decisions sounds like a really good step forward. I will update the encoding on our side. Thank you!

[Ernst79]

The service load should be a lot shorter now, so perhaps enough for humidity and battery (or something else) with more precision. In case you miss a certain format, let me know.

I'm working on the HA update now, should be ready soon.

[pvvx]

All the necessary information can be conveyed by several advertisements. Sequence.
This is how many Xiaomi-Mijia-BLE and custom firmware for thermometers work.
A Xiaomi-Mijia typical sequencing of data transmission in turn over 4 advertisements events.

[Ernst79]

Just a heads up. BTHome V2 is merged into Home Assistant dev, so you can expect it to be included in the 2022.12 release.

[rbaron]

Tks for the heads up @Ernst79 👌

[rbaron]

Quick update: I'm working on rewriting the code to use the nRF Connect SDK in #76, and I added support for both BTHome v1 (tested) and BTHome v2 (untested yet).

I also implemented the custom b-parasite BLE protocol for backward compatibility, but I'm finding BTHome + bluetooth_proxy so convenient that I'm migrating my b-parasites to it 👌

[drspangle]

@rbaron Are you planning to put together some documentation (e.g., wiki articles) for how to set up b-parasites using BTHome/ESPHome? I'm having a hard time wrapping my head around how this is meant to work... I'm totally new to ESPHome and BTHome so I'm a bit baffled. Does this change something fundamental about the use of a BTLE-WiFi (MQTT) bridge device? Or is this really just changing the design of the firmware in a way that fundamentally doesn't change how you flash and deploy the device? Am I just confused about BTHome and ESPHome and what each is meant to do in this context?

[rbaron]

@drspangle from this project's perspective BTHome is a spec that describes how to encode sensor data inside a BLE message (this is described in https://bthome.io/). Home Assistant supports it directly (e.g.: BTHome devices BLE <-> Home Assistant) or you can also use some ESPHomes as proxies (e.g.: BTHome devices BLE <-> ESPHomes <-> Home Assistant). If you have questions that are specific to HA/ESPHome, I would suggest raising an issue on their repos to help them improve the docs, or open an issue here so other members of the community can also help.

[drspangle]

@rbaron Thanks, this makes sense. I was puzzling through whether perhaps the ESPHome piece was going to change the deployment mode for the b-parasite, where perhaps you'd first flash it with a configuration firmware that then gets replaced OTA with something else. I see now that this doesn't make sense for a b-parasite, and you're sticking with the "BTLE beacon" design style. This makes a lot more sense. Thank you!

[rbaron]

BTHome V2 now includes illuminance.