bluetooth_tracker doesn't work in 2022.7.x (not compatible with python 3.10)

[bahamut657]

The problem

No devices are located by bluetooth_tracker.
In registry I found that stacktrace:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/bluetooth_tracker/device_tracker.py", line 197, in update_bluetooth
    await perform_bluetooth_update()
  File "/usr/src/homeassistant/homeassistant/components/bluetooth_tracker/device_tracker.py", line 175, in perform_bluetooth_update
    rssi = await hass.async_add_executor_job(client.request_rssi)
  File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.10/site-packages/bt_proximity/bt_rssi.py", line 55, in request_rssi
    self.prep_cmd_pkt()
  File "/usr/local/lib/python3.10/site-packages/bt_proximity/bt_rssi.py", line 24, in prep_cmd_pkt
    b'6sB17s', bt.str2ba(self.addr), bt.ACL_LINK, b'\0' * 17)
SystemError: PY_SSIZE_T_CLEAN macro must be defined for '#' formats 

Possibily, it is related to Python3.10

What version of Home Assistant Core has the issue?

2022.7.3

What was the last working version of Home Assistant Core?

2022.6.x

What type of installation are you running?

Home Assistant Supervised

Integration causing the issue

bluetooth_tracker, device_tracker

Link to integration documentation on our website

https://www.home-assistant.io/integrations/bluetooth_tracker/

Diagnostics information

Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/components/bluetooth_tracker/device_tracker.py", line 197, in update_bluetooth
await perform_bluetooth_update()
File "/usr/src/homeassistant/homeassistant/components/bluetooth_tracker/device_tracker.py", line 175, in perform_bluetooth_update
rssi = await hass.async_add_executor_job(client.request_rssi)
File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/local/lib/python3.10/site-packages/bt_proximity/bt_rssi.py", line 55, in request_rssi
self.prep_cmd_pkt()
File "/usr/local/lib/python3.10/site-packages/bt_proximity/bt_rssi.py", line 24, in prep_cmd_pkt
b'6sB17s', bt.str2ba(self.addr), bt.ACL_LINK, b'\0' * 17)
SystemError: PY_SSIZE_T_CLEAN macro must be defined for '#' formats

Example YAML snippet

device_tracker:
    - platform: bluetooth_tracker
      request_rssi: true

Anything in the logs that might be useful for us?

No response

Additional information

No response

[probot-home-assistant]

bluetooth_tracker documentation
bluetooth_tracker source
_{^{(message by IssueLinks)}}

[frenck]

That is expected and was announced:

See: https://www.home-assistant.io/blog/2022/07/06/release-20227/#breaking-changes

[stefanuytterhoeven]

Why isn't this solved?

[frenck]

@stefanuytterhoeven see above? 🤷
Things don't solve itself 😉

[stefanuytterhoeven]

???

[frenck]

I am not understand you response, it only contains question marks. Could you clarify those?

[bigbigblue]

I have seen there is a pybluez2 library which was last updated August 2021 and requires python >=3.6. Could this be used instead of pybluez? Would it solve any, some or all of the issues?
I am now currently stuck on HA 2022.7.1 as a lot of my automations reply on the bluetooth device tracker. As I cannot upgrade past 2022.7.1, I cannot upgrade to the latest TRACCAR add-on, which I also use extensively. I have to say I am getting a bit frustrated with all the breaking changes.
I would like to help to fix this if I can. I have some programming experience (mainly assembler and C), but not with Python or HA, so would probably need some hand-holding.
Cheers

[frenck]

I have seen there is a pybluez2 library which was last updated August 2021 and requires python >=3.6. Could this be used instead of pybluez? Would it solve any, some or all of the issues?

The solution is documented in the breaking change, which is listed above as well. Move towards bleak.

[kevdliu]

I have seen there is a pybluez2 library which was last updated August 2021 and requires python >=3.6. Could this be used instead of pybluez? Would it solve any, some or all of the issues?

The solution is documented in the breaking change, which is listed above as well. Move towards bleak.

As far as I know bleak only supports BLE. It does not detect classic Bluetooth devices like smartphones, which I assume is what most of us are trying to track with this integration. It's not really a suitable replacement.

[bigbigblue]

The new Bluetooth integration in HA 2022.8.x seems to have re-introduced the bluetooth tracker functionality and it detects my smartphones perfectly.

[Rubquila]

That's true? I can update my ha without problems with classic bluetooth tracker?

[bigbigblue]

It is true! HOWEVER - I find it less reliable than the old bluetooth tracker - I find that it toggles between home and not home for my bluetooth trackers more often than is acceptable (using the RPI4 built in bluetooth). I then tried using one of the recommended USB bluetooth dongles and had a lot of problems with it.
So yes, basic function works, but look out for some quirks. I am hoping the bluetooth integration will go from strength to strength as I rely HEAVILY on reliable bluetooth tracking for my automations.

[Mariusthvdb]

not so sure about that. I have it installed, and it finds exactly nothing at all...

100% certain my miniPC BT is functional, because when adding the blue tracker to yaml it finds many many devices (even the cards driving by)

however that doesnt find most of the phones in the household.

sofar, the new BT integration is no replacement at all for the former https://rc.home-assistant.io/integrations/bluetooth_tracker/

device_tracker:
  - platform: bluetooth_tracker

[bigbigblue]

Even with the updated bluetooth libraries in 2022.9.1 there isn't any improvement. Downloading the diagnostics from the bluetooth integration, I can see it picking up a myriad of BLE devices, but the mobile phones are not detected, or are only detected for a few seconds every couple of minutes.
I did notice that running 'hcitool lescan' shows lots of devices, however 'hcitool scan' returns nothing, which is probably why Home Assistant doesn't see anything either.
I am struggling to understand why this would be the case.

[Duke-Box]

I did notice that running 'hcitool lescan' shows lots of devices, however 'hcitool scan' returns nothing, which is probably why Home Assistant doesn't see anything either.
I am struggling to understand why this would be the case.

This is because Classic Bluetooth isn't supported anymore by HA as the title of this issue points out.

Further, as @Mariusthvdb and @kevdliu state, the new bluetooth in HA from 2022.7.x is based on Bleak which will probably never support classic bluetooth and therefore is no solution.
I suggest that what you are seeing are the BLE MACs of your phones that are randomised every few minutes therefore you see them for a few seconds then they change.

[kevdliu]

I ended up creating a custom component based off of the BLE Tracker and modified it to recognize BLE beacons by their service UUID instead of mac. On my Android phone I installed nRF Connect and have it advertise as a beacon with a preset service UUID that the custom component recognizes. Seems to be working well so far.

[bigbigblue]

Duke-box,
If HA won't ever support the proper tracking of mobile phones, then in one step HA has been rendered useless for 90% of the function of my automations. Hopefully someone will come along with a Bluetooth tracking integration for phones at some point. I don't understand why support for smartphone tracking via bluetooth would be dropped, I would imagine 99%+ of all HA users have bluetooth enabled smartphones and it makes perfect sense (to me anyway) to use it for presence tracking. The alternatives (wifi and gps) have issues which the old bluetooth tracker doesn't have. Wifi uses a lot of battery power and GPS has privacy issues.
Hey ho, perhaps I need to learn how to write an integration myself (I was a programmer many many moons ago)

[roybosch]

Duke-box, If HA won't ever support the proper tracking of mobile phones, then in one step HA has been rendered useless for 90% of the function of my automations. Hopefully someone will come along with a Bluetooth tracking integration for phones at some point. I don't understand why support for smartphone tracking via bluetooth would be dropped, I would imagine 99%+ of all HA users have bluetooth enabled smartphones and it makes perfect sense (to me anyway) to use it for presence tracking. The alternatives (wifi and gps) have issues which the old bluetooth tracker doesn't have. Wifi uses a lot of battery power and GPS has privacy issues. Hey ho, perhaps I need to learn how to write an integration myself (I was a programmer many many moons ago)

Have you considered using the BLE transmitter that's part of the Home Assistant mobile app? I think this has the same functionality as you're using now. You'll only have to change the entity-id in your automations. https://www.home-assistant.io/integrations/bluetooth_le_tracker/

[bigbigblue]

Roybosch,
Thanks for the suggestion, but I don't really think it is practical in my situation. With the bluetooth tracker, I tracked the presence (or not) of 5 mobile phones (and this number will only increase). It isn't really practical to ask all of the phone owners to install, setup and maintain the HA mobile app. With the old tracker, I only needed to know their bluetooth mac and they had to do nothing other than leave their bluetooth on.
The tracking was then used to control my heating, among other things. With the cost of energy spiralling, I want to maintain the tight control over when my property is being heated, which I no longer have through HA.

[roybosch]

I understand. I'm having the same issue right now: #76979

Although the BLE tracker might be suitable for me, as I'm tracking only the devices of my family who also use the mobile app. It still is strange that such an important integration doesn't have a good alternative within HA itself.

[bigbigblue]

It still is strange that such an important integration doesn't have a good alternative within HA itself.

Exactly my point, however this : https://github.com/robmarkoski/bt-mqtt-tracker might offer a glimmer of hope (at least for those of us who can install stuff on their HA installation).

[Duke-Box]

It still is strange that such an important integration doesn't have a good alternative within HA itself.

Exactly my point, however this : https://github.com/robmarkoski/bt-mqtt-tracker might offer a glimmer of hope (at least for those of us who can install stuff on their HA installation).

Room Assistant does the same thing

https://www.room-assistant.io/guide/#how-it-works

And it uses the built in MQTT server of HA so it works pretty much out of the box.
However it is quite slow at responding but it uses classic bluetooth as well as BLE.

I use it already to detect phone arriving in the kitchen and was considering building another instance with a greater range threshold just to replicate the bluetooth_tracker functionality.

Edit: And there is a Add On for HA so should be not too much pain to try it.

[Mariusthvdb]

Or, if you have a spare Pi, just install Ha in the last version that supported the native bluetooth tracker and use mqtt to publish those. It was my system before I switched to a single box. Have a main production system, a secondary with Mqtt add-on and bluetooth, and a ternary for Zwave add-on. Spread your risks ;-)

about these ble trackers in the app: I tried all of that, but not a single phone was detected, only hundreds of passing carkists... o, and 2 Hue Bt lights.

[roybosch]

about these ble trakers in the app: I tried all of that, but not a single phone was detected, only hundreds of passing carkists... o, and 2 Hue Bt lights.

You're right. I see that I already activated the BLE tracker in my config in the past. Not a single BLE_ prefix is present in the known_devices.yaml file.

I installed a few room-assistant nodes a few months ago but they don't work great yet. I find calibrating the things to show the correct room very difficult. At this moment, when I'm in the living room, it shows "office" often. That's a room upstairs...

[bigbigblue]

Or, if you have a spare Pi, just install Ha in the last version that supported the native bluetooth tracker and use mqtt to publish those. It was my system before I switched to a single box. Have a main production system, a secondary with Mqtt add-on and bluetooth, and a ternary for Zwave add-on. Spread your risks ;-)

Just prior to support for the mac based detection of bluetooth devices being deprecated, that is exactly the setup I had (4 x pi's running HA) JUST to detect the presence of my smartphones. Given the soaring cost of energy, I decided to go back to just 1 pi running HA, as generally it got the presence detection right on its own.
Ho hum, so much for trying to make a contribution to saving the planet.
Maybe I will have to go back to running another pi, just for presence tracking.

[Duke-Box]

Wow, so you guys are saying that the NEW ble_tracker doesn't work with 2022.9.x either!
I thought this was released back in 2022.8.0.

I can see from the linked issue in @Mariusthvdb comment above that it is being worked on though.
Yet another reason not to upgrade.
Also glad I didn't buy the quite expensive BLE tracker tags with fixed MACs that should have worked with this integration!

[roybosch]

That's about half right. In the past, I've been using ESP32's with ESPhome to track my Xiaomi BLE climate sensors. Now I'm using the Bluetooth integration combined with the Xiaomi BLE integration, which works great.

The BLE_tracker integration which you still have to configure via yaml doesn't seem to work. So tracking phones is still an issue.

[Mariusthvdb]

ble does work. as a matter of fact it finds huge amounts of devices...
just not my phones ;-)

[Duke-Box]

Thanks for the clarification on the ble integration.
I've learnt something today - when you enable the BLE sensor in the HA companion app for android it doesn't allow you to specify a MAC but allows you to specify a UUID.
But that is no good for the ble_tracker as it track MACs.
Therefore the phone does not show up in known devices using HA app.

In my known devices yaml I can see my phone appear and then re-appear with a different MAC 'cos it is randomising it. This is done deliberately so you can't track a phone.
What would be good (and I know this is off topic) is for the ble_tracker to capture both MAC and UUID, then we would have a fighting chance.

@kevdliu - Since you have made a Custom Component that tracks the UUID already, which would solve my immediate problem, could you share this? It would save me re-inventing the wheel, so to speak. :-)

Thanks

[kevdliu]

Thanks for the clarification on the ble integration. I've learnt something today - when you enable the BLE sensor in the HA companion app for android it doesn't allow you to specify a MAC but allows you to specify a UUID. But that is no good for the ble_tracker as it track MACs. Therefore the phone does not show up in known devices using HA app.

In my known devices yaml I can see my phone appear and then re-appear with a different MAC 'cos it is randomising it. This is done deliberately so you can't track a phone. What would be good (and I know this is off topic) is for the ble_tracker to capture both MAC and UUID, then we would have a fighting chance.

@kevdliu - Since you have made a Custom Component that tracks the UUID already, which would solve my immediate problem, could you share this? It would save me re-inventing the wheel, so to speak. :-)

Thanks

I uploaded my custom component to a github repo: https://github.com/kevdliu/bt_tracker. I added some examples and notes to the readme as well. Let me know if you run into any problems using it.

[Duke-Box]

@kevdliu - thanks for the link, I will look at your code and use it as is or as a base for any tweaks.
Again many thanks - appreciated

[andriilahuta]

It is also possible to use D-Bus API directly. Here's a proof of concept with a few caveats:

• Discovery is not implemented, so trackable devices should already be present in known_devices.yaml
• Bluez experimental features should be enabled (--experimental flag)
• Bluetooth integration should be configured

custom_bluetooth_tracker.zip

[bigbigblue]

It is also possible to use D-Bus API directly. Here's a proof of concept with a few caveats:

• Discovery is not implemented, so trackable devices should already be present in known_devices.yaml
• Bluez experimental features should be enabled (--experimental flag)
• Bluetooth integration should be configured

custom_bluetooth_tracker.zip

Probably a dumb question, but how do I enable --experimental flag in a supervised HA install?
Thanks

[andriilahuta]

I've never tried a supervised HA install, so I'm not entirely sure, sorry.

But if your OS uses systemd, you can try creating /etc/systemd/system/bluetooth.service.d/override.conf file with the following content:

[Service]
ExecStart=
ExecStart=/usr/lib/bluetooth/bluetoothd --experimental

And then running

systemctl daemon-reload
systemctl restart bluetooth

Hope that helps.

[issue-triage-workflows]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates.
Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍
This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

[bigbigblue]

Adding a comment to keep this thread alive. I will try the dbus api proof of concept above when I get a chance.

[llamafilm]

Have you considered using the BLE transmitter that's part of the Home Assistant mobile app? I think this has the same functionality as you're using now. You'll only have to change the entity-id in your automations. https://www.home-assistant.io/integrations/bluetooth_le_tracker/

@roybosch can you expand on what you mean by this? Is the iOS app supposed to turn an iPhone into a trackable BLE device? From what I can tell, the bluetooth_le_tracker finds iOS devices whether or not they have the Companion app installed, but the MAC is randomized for privacy so it's not useful.

[sniesler08]

I understand that there was a breaking change and all, but why is it that the bluetooth tracker integration page, at least unless I am blind does not show that it is broken and when used in home assistant, why is there no warning about this to let people know instead of people wasting time trying to figure out what is wrong?

[llamafilm]

@sniesler08 Although I disagree with the decision, the answer to your question is here: home-assistant/home-assistant.io#24115

[roybosch]

Have you considered using the BLE transmitter that's part of the Home Assistant mobile app? I think this has the same functionality as you're using now. You'll only have to change the entity-id in your automations. https://www.home-assistant.io/integrations/bluetooth_le_tracker/

@roybosch can you expand on what you mean by this? Is the iOS app supposed to turn an iPhone into a trackable BLE device? From what I can tell, the bluetooth_le_tracker finds iOS devices whether or not they have the Companion app installed, but the MAC is randomized for privacy so it's not useful.

That's right. In my android app, under settings > companion app > manage sensors > BLE transmitter I can choose to transmit a BLE signal that won't change. I'd be surprised if the iOS app doesn't have this feature.

[bigbigblue]

The issue I have with using the app is I use presence tracking for many family members by their mac and I do not want to ask everyone to install, maintain and run the HA app. The bluetooth tracker was the cornerstone of my automations, so after many hours invested in HA, I am looking for an alternative which does support bluetooth mac tracking. Such a shame! Kindest regards Peter Talbot

…

________________________________ From: Roy ***@***.***> Sent: Tuesday, February 14, 2023 2:48:27 PM To: home-assistant/core ***@***.***> Cc: bigbigblue ***@***.***>; Comment ***@***.***> Subject: Re: [home-assistant/core] bluetooth_tracker doesn't work in 2022.7.x (not compatible with python 3.10) (Issue #75049) Have you considered using the BLE transmitter that's part of the Home Assistant mobile app? I think this has the same functionality as you're using now. You'll only have to change the entity-id in your automations. https://www.home-assistant.io/integrations/bluetooth_le_tracker/ @roybosch<https://github.com/roybosch> can you expand on what you mean by this? Is the iOS app supposed to turn an iPhone into a trackable BLE device? From what I can tell, the bluetooth_le_tracker finds iOS devices whether or not they have the Companion app installed, but the MAC is randomized for privacy so it's not useful. That's right. In my android app, under settings > companion app > manage sensors > BLE transmitter you can transmit a BLE signal that won't change. I'd be surprised if the iOS app doesn't have this feature. — Reply to this email directly, view it on GitHub<#75049 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AEXQZPMOI7UVYWMF7ISEBU3WXOLLXANCNFSM53KLJVHA>. You are receiving this because you commented.Message ID: ***@***.***>

[llamafilm]

@roybosch thank you. That’s a cool feature. Unfortunately it looks like Android only in the docs .

[Duke-Box]

I'm surprised, this isn't a problem for me any longer. It's not perfect but it does work on my Pi4

Since having my "discussion" with Frenck last year as in the link @llamafilm posted above, I used an old laptop and installed HA on it (generic x64 version). I then restored my production configuration to it.
This enabled me to upgrade HA to (I think) 2022.11.x without disrupting my production system.
HA discovered the bluetooth adapter on the laptop and installed the integration. It also installed iBeacon Tracker so I was going to experiment with that as I had bought some iBeacons (Chippolo - because they were claimed to work with HA in the HA documentation - They DON'T, beware) but that's off topic.

To my surprise my bluetooth device trackers (phones) started to work and report home or away. I then upgraded my production system and they all work.
My testbed is running the latest HA (2023.2.4) and my production is running 2023.1.7

So I have assumed that in one of the upgrades this problem was quietly fixed. I can see my known_devices.yaml file getting populated by bluetooth (BT) devices as well as BLE devices (if I have the ble tracker enabled in my config yaml), so it all appears to work however the device tracker (phones) say home for a minute or so then away for a few seconds continually unless they are really away in which case this state is reported correctly. Hence me not relying on just Bluetooth device trackers, I use Ping and GPS as well to determine if someone is really home.

[llamafilm]

the device tracker (phones) say home for a minute or so then away for a few seconds continually unless they are really away in which case this state is reported correctly

This sounds like a symptom of iOS randomized MAC addresses. I'm not sure how often that happens... A few seconds sounds awfully fast; I would have guessed it happens maybe once per day, but in any case I've given up on the idea of BT tracking iPhones.

[Duke-Box]

"This sounds like a symptom of iOS randomized MAC addresses."

Nope - they are fixed BT MACs and show up in known_devices.yaml.

They are Android phones though, I can see the BT MAC in the phones status info. I don't have an iphone to hand to check if that's the same for iOS at the moment.

[kevindesousa]

Any news about this issue ?

[buhito81]

Any updates?

[bigbigblue]

There haven't been any updates and I doubt there ever will be. Nobody from the dev team is assigned to this issue and there are no pull requests for it.
Unless someone takes on the task of providing a new add on or re-adding the function to the HA core, then we will have to accept the function is lost. I would have a go myself (I was a programmer many moons ago), however I wouldn't even know where to start integrating a function into HA.
For me, the loss of this functionality has reduced the usefulness of HA by about 90%, as most of my automations relied on bluetooth tracker functionality.

[andriilahuta]

I've been successfully using D-Bus proof of concept for a while now (linked above) on a Raspberry Pi / Ubuntu server.
One thing I noticed is slightly increased battery drain. This can be mitigated by polling already seen devices less frequently.

[issue-triage-workflows]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates.
Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍
This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.