Cant find Homekit device in Home App

[MrJoki007]

Home Assistant release with the issue: 0.80.0

Last working Home Assistant release (if known): 0.77.?

Operating environment (Hass.io/Docker/Windows/etc.): Freenas IOCage

Component/platform: Homekit

Description of problem:
I upgraded my HomeAssistant Server from version 0.77.? to 0.79.3.
After the upgrade, the devices in the apple home app were no longer responding.
Then i deleted the bridge in the home app and wanted to add it again. But it doesnt find it anymore.

Problem-relevant configuration.yaml entries and (fill out even if it seems unimportant):

logger:
  default: warning
  logs:
    homeassistant.components.homekit: debug

homekit:
  filter:
    include_entities:
      - demo.demo

Traceback (if applicable):

2018-10-12 21:17:54 WARNING (MainThread) [homeassistant.components.http] legacy_api_password support has been enabled. If you don't require it, remove the 'api_password' from your http con$
2018-10-12 21:18:18 DEBUG (MainThread) [homeassistant.components.homekit] Begin setup HomeKit
2018-10-12 21:18:24 INFO (SyncWorker_3) [homeassistant.components.homekit.util] Pincode: 805-29-469
2018-10-12 21:18:24 DEBUG (SyncWorker_3) [homeassistant.components.homekit] Driver start

Additional information:

[cdce8p]

Try stopping HA, delete the .homekit.state file jn the config dir and start again

[MrJoki007]

Still same problem

[cdce8p]

Can you add pyhap: debug to your logger config and see what shows up. It's a bit strange, especially since it already work. There hasn't been any logic changes between the two versions you mentioned. Also try rebooting your iOS devices and make sure they are on the same local network.

[MrJoki007]

Ive tried it on 3 different ios devices but none found it.
Here's the log

2018-10-14 11:58:07 WARNING (MainThread) [homeassistant.components.http] legacy_api_password support has been enabled. If you don't require it, remove the 'api_password' from your http con$
2018-10-14 11:58:12 DEBUG (MainThread) [homeassistant.components.homekit] Begin setup HomeKit
2018-10-14 11:58:16 DEBUG (SyncWorker_4) [pyhap.characteristic] set_value: Name to Home Assistant Bridge
2018-10-14 11:58:16 DEBUG (SyncWorker_4) [pyhap.characteristic] set_value: SerialNumber to default
2018-10-14 11:58:16 DEBUG (SyncWorker_4) [pyhap.characteristic] set_value: FirmwareRevision to 0.80.0
2018-10-14 11:58:16 DEBUG (SyncWorker_4) [pyhap.characteristic] set_value: Manufacturer to Home Assistant
2018-10-14 11:58:16 DEBUG (SyncWorker_4) [pyhap.characteristic] set_value: Model to Bridge
2018-10-14 11:58:16 DEBUG (SyncWorker_4) [pyhap.characteristic] set_value: SerialNumber to homekit.bridge
2018-10-14 11:58:16 INFO (SyncWorker_12) [pyhap.accessory_driver] Loading Accessory state from `/root/.homeassistant/.homekit.state`
2018-10-14 11:58:16 INFO (SyncWorker_12) [homeassistant.components.homekit.util] Pincode: 016-05-588
2018-10-14 11:58:16 DEBUG (SyncWorker_12) [homeassistant.components.homekit] Driver start
2018-10-14 11:58:16 INFO (SyncWorker_16) [pyhap.accessory_driver] Starting the event loop
2018-10-14 11:58:16 INFO (SyncWorker_0) [pyhap.accessory_driver] Starting accessory Home Assistant Bridge on address 192.168.178.188, port 51827.
2018-10-14 11:58:17 DEBUG (SyncWorker_0) [pyhap.accessory_driver] AccessoryDriver started successfully

[MrJoki007]

One other thing i noticed is, that homeassistant always found my panasonic tv and denon avr, but they dont show up in the ui anymore.

[cdce8p]

Hm, since you mentioned an issue with discovery as well. What did change is the version of one dependency that discovery and HomeKit both use: zeroconf.

Could you try something?

1. Stop Home Assistant
2. Activate your virtual env where Home Assistant is running from
3. Run pip list | grep 'zeroconf' and post the version here.
4. If its: 0.21.x, run pip install zeroconf==0.20.0
5. After that's finished start HA with hass --skip-pip
6. Does it changed anything?

[MrJoki007]

after running pip list | grep 'zeroconf'
it does not return anything

[cdce8p]

Are you sure? That would explain, why you can't find the Home Assistant Bridge if it's never made accessible to the network..

I would suggest uninstalling and installing HAP-python manually. That should take care of zeroconf as well. After 2 run the following commands:

• pip uninstall HAP-python
• pip install HAP-python==2.2.2
• Restart HA and test

[MrJoki007]

pip uninstall HAP-python
Skipping HAP-python as it is not installed.

[cdce8p]

Then you've not activated the correct virtual env. Without HAP-python you woundn't have gotten the log output you posted above.

[MrJoki007]

Im not using a virtual env. Just running homeassistant in the freenas jail

[cdce8p]

Sorry, this could have been my mistake all along. I might have assumed something which isn't the case for you.

If you run just pip list it should output all installed python packages. Do you see zeroconf or HAP-python anywhere? It could have been that the suggested grep command doesn't work.

[MrJoki007]

root@homeassistant:~ # pip list
Package Version

---

aiohttp 3.4.4
asn1crypto 0.24.0
astral 1.6.1
async-timeout 3.0.0
attrs 18.2.0
bcrypt 3.1.4
certifi 2018.8.24
cffi 1.11.5
chardet 3.0.4
cryptography 2.3.1
homeassistant 0.80.1
idna 2.7
idna-ssl 1.1.0
Jinja2 2.10
lxml 4.2.4
MarkupSafe 1.0
multidict 4.4.0
pip 18.1
pycparser 2.18
PyJWT 1.6.4
pytz 2018.5
PyYAML 3.13
requests 2.19.1
setuptools 40.0.0
six 1.11.0
sqlite3 0.0.0
urllib3 1.23
virtualenv 16.0.0
voluptuous 0.11.5
voluptuous-serialize 2.0.0
yarl 1.2.6

[cdce8p]

pip install HAP-python==2.2.2

[MrJoki007]

Now theres HAP-python and zeroconf in the list, but it still doesnt work

[cdce8p]

Does the log show something new?
You mentioned that other devices haven't been picked up by HA, has that changed?

You could try downgrading the zeroconf version: pip install --upgrade zeroconf==0.20.0

[MrJoki007]

Its very weired. Today i got a Mi TV Box and it shows up in Homeassistant by it self.
But my tv does not show up and homekit still doesnt work

[MrJoki007]

Now ive restarted homeassistant and it does not show up anymore

[cdce8p]

I don't really know how to help you with this issue, to be honest.

Did you change something in your network configuration (especially with your router) between working and not working? I have seen the router settings being an issue: https://community.home-assistant.io/t/another-post-about-pairing-with-homekit/68255/27?u=cdce8p

My last attempt would be to see if downgrading Home Assistant works.
The command would be pip install --upgrade homeassistant==x.xx.x with x.xx.x being the version number that last worked.

[MrJoki007]

Downgraded to 0.77.3 but hass is now crashing with this error:
2018-10-19 11:11:03 WARNING (MainThread) [hbmqtt.mqtt.protocol.handler] BrokerProtocolHandler Unhandled exception in reader coro: IncompleteReadError('0 bytes read on a total of 1 expected bytes',)

[cdce8p]

This error is unrelated to HomeKit. Could you try temporarily disabling all components that have errors (as it seems MQTT ones)?

[cdce8p]

Any updates?

[MrJoki007]

Still not working

[cdce8p]

Try reading this section:
https://www.home-assistant.io/components/homekit/#troubleshooting

Maybe I've already sumed up a solution there but haven't mentioned it yet.

[MrJoki007]

@jigfox what device are you using to run homeassistant on?

[cdce8p]

If you delete the .homekit.state file without removing the Home Assistant Bridge from the Home App, you won't be able to pair again. Make sure to delete the Brigde manually in that case (accessory settings -> remove bridge from home).

[jigfox]

@jigfox what device are you using to run homeassistant on?

@MrJoki007 I use the same like you: Freenas IOCage

@cdce8p I deleted the bridge from my home, I also tried to create a complete new home with no devices, both ways I got the endless loading

[elsom25]

Seems like locking zeroconf to 0.20.0 conflicts with #16751. Any suggestions on how to have both components working together? (on Home Assistant v0.82.1)

[mathj30]

HA: v0.84.6
Almost the same setup here, Freenas on IOCage and unable to make homekit discoverable. Did some digging around using your recommendations in this thread and I downgraded zeroconf to 0.20.0 and netdisco to 2.0.0 and voilà it was there working flawless(homekit). Any ideas on how to make this work with zeroconf 0.21.x? What changed?

[postlund]

Same problem here. Would be nice with a real solution to this.

[postlund]

Downgrading python-zeroconf doesn't seem to work for me though.

[mathj30]

@postlund what version are you using? Because over the weekend I upgraded (along with all other requirements) to version 0.86.4 and manually downgraded zeroconf to 0.20.0 and it's still working. Did you add the argument --skip-pip otherwise it will re-upgrade automatically to 0.20.1.

[postlund]

@MATH609 I might have spoken to quickly, I think I have some issue with duplicate entity names. I'll have a look again when I'm home (I've verified that the versions are correct).

[postlund]

I have done some investigation and the problem seems to be that the HomeKit bridge never starts because a duplicate entry is found, but the exception is not propagated back to the log for some reason so it is not very clear that this happens.

Continuing the investigation, I have come to the conclusion that it is a hash collision: two different entities generate the same AID, so HomeKit never starts. Quite fascinating really:

>>> generate_aid('switch.outside')
729810366
>>> generate_aid('sensor.storage')
729810366

So switch.outside and sensor.storage generate the same AID, who could have known? @cdce8p, maybe something to look into how to handle this in a better way? 😄 If I remove one of the entities it works as it's supposed to.

[mathj30]

@postlund it works with zeroconf 0.20 or 0.21?

[postlund]

@MATH609 Latest, 0.21.3 (plain HASS 0.86.4).

[MrJoki007]

How do you change that to get it running?

[cdce8p]

@postlund We knew that there could be hash collisions, but we certainly didn't expected them for that combination 😄 At the moment I can't really do much about it, since fixing it is a longer endeavor. As you already know HomeKit doesn't like it if the same AID belongs to two different accessories.

[postlund]

@cdce8p Yeah, very unfortunate that a collision happened for such at simple case like this 😄 But the adler32 hashing algorithm is known to have a lot of collisions, any particular reasons why it was used instead of like CRC32 or so? The best case scenario as I see would be if the entity registry could manage the IDs, that would make most sense. Changing the hashing would of course be a breaking change, but it seems to be necessary to fix the problem.

[cdce8p]

Somebody suggested it to me when I started with the HomeKit component.
As you pointed out already changing the hashing would be a hugh breaking change. At some point we will have to do it, but I would perfer if there is permanent solution in place first. I don't want to just change the hashing algorithm. Maybe we can leverage the existing entity registry for that? As a side bonus people could start chaning there entity_ids again 🙂

However, since I'm pretty busy with university at the moment, there hasen't been much progress on that front. Maybe towards the end of march, but I can't promiss anything.

[postlund]

Right, I guessed that it was something like that 😄 Adler32 is pretty fast, but not very good when it comes to collisions. Anyway, using the entity registry should be the way to go, but since AIDs are 64 bits something will have to be added as I believe pretty much any identifier can be used in the core registry.

@balloob Sorry to tag you, but I believe you have the best knowledge of the entity registry. We have hash collision issues in the homekit component (switch.outside and sensor.storage has the same hash for instance), which might break the component. As each entity that is exposed to homekit needs a unique 64 bit identifier, we kinda feel that it would be appropriate that the entity registry keeps track of that. Currently this identifier is calculated based on the entity name (which also breaks support for renaming entities) using Adler-32. Changing the hash algorithm is a breaking change, so if we do that we want to do it the right way. Do you have any input regarding adding a unique identifier (assigned by home assistant core) to each entity for use cases such as this? Could just be a number that increments for each new entity I guess.

[cdce8p]

@postlund I haven't looked at the EntityRegistry yet, but I think one way forward could be as follows:

As backgrund, we would create a json file with the unique_id to aid mappings.

• Check if entiy is registered
  • If true, get unique Id; lookup aid in json file
    • If not existis: Generate random number and check that it's unique (against a set of all aids)
    • Else use aid
  • If false, lookup aid based on entity_id
    • Same rules apply

We would also need to make sure to update the entity_id mapping to the unique_id if an entiy starts supporting the entity_registry.

What do you think?

[balloob]

I think what @cdce8p proposes is a good idea, as it avoids most breaking changes. However, it is still going to be difficult to do this for all entities, as apparently it's very easy to collide.

FYI, only if an entity is in the registry entry can it be expected to be the same entity across reboots. So use entity registry entry ID as your primary key.

So when you generate an aid for an entity that is in the registry, persist it and don't allow changing it anymore. When hashing the entities without unique ID, keep track of existing AIDs from the registry and prevent collision (add a 1 etc). Also keep track of generated AIDs, but don't persist to disk.

You can do entity_registry.async_get(entity_id) to get the entity registry entry.

[cdce8p]

@balloob I would still prefer to write and later read the AIDs to a file though. As you said it's going to be pretty difficult to avoid any name collisions with hashing and preventive methods (adding a 1) don't solve that either. Say we needed to add 1 to one hash due to a collision and later the first entity is removed (through the filter), now the second entity won't work either since it would have the AID of the first one.

Since we only would need to do this on startup of the HomeKit component, it would be limited to one read and one write operation. I would also add a big warning not to touch/change anything in it 😉

[stale]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates.
Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍
This issue now has been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

[MrJoki007]

Anything new?

[vishnubob]

I have done some investigation and the problem seems to be that the HomeKit bridge never starts because a duplicate entry is found, but the exception is not propagated back to the log for some reason so it is not very clear that this happens.

Continuing the investigation, I have come to the conclusion that it is a hash collision: two different entities generate the same AID, so HomeKit never starts. Quite fascinating really:

>>> generate_aid('switch.outside')
729810366
>>> generate_aid('sensor.storage')
729810366

So switch.outside and sensor.storage generate the same AID, who could have known? @cdce8p, maybe something to look into how to handle this in a better way? 😄 If I remove one of the entities it works as it's supposed to.

I was also bitten by this problem as well (see issue #27954) and I think I have a solid fix to offer with PR #27955

[stale]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates.
Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍
This issue now has been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

[MrJoki007]

Just upgraded to python3.7 and now discovery and homekit works again