Unifi device tracking is difficult to use with no opt-in method for device trackers

[stbenjam]

Home Assistant release with the issue:

0.99.1

Last working Home Assistant release (if known):

0.94? Not entirely sure

Operating environment (Hass.io/Docker/Windows/etc.):

Docker

Component/platform:

https://www.home-assistant.io/components/unifi/

Description of problem:

The move to the UniFi platform for device tracking is the worst breaking change I've had in HA, and I'm not entirely sure how I'm supposed to be using it to get the same behavior I used to have.

I had a very long known_devices.yaml, with all my family and guests, their pictures and their names attached to the devices they have, all based on them connecting to my WiFi. It was a lot of work to get this exactly the way I wanted. I have automations based on certain people's arrival. It's all broken. The move to the UniFi didn't import any of this information, it just ignores it 😭

Now, with the Unifi integration every network device (well over 200) shows up in HA... do I really have to go through the UI and disable all the entities I don't want to use? Is there no easier way to do this?

Also, the devices names are ended called different names than the other platform for whatever reason, so I have to match things up based on mac addresses and rename the devices to what they used to be called so the automations work. This is all very painful. Since I'm not alone, and this is probably a couple hours of work, is there any chance to write something to import the data from known_devices.yaml?

Problem-relevant configuration.yaml entries and (fill out even if it seems unimportant):

I added the Unifi integration in the UI, but I also tried the below

I added the UniFi integration, and configured it like this:

unifi:
  controllers:
  - host: !secret unifi_controller
    site: Default
    dont_track_devices: True
    dont_track_wired_clients: True
    detection_time: 800
    ssid_filter:
      - "SSID 1"
      - "SSID 2"

Traceback (if applicable):

N/A

Additional information:

[probot-home-assistant]

Hey there @Kane610, mind taking a look at this issue as its been labeled with a integration (unifi) you are listed as a codeowner for? Thanks!

[Kane610]

Try using config entry options to change those settings if they don't apply properly

https://www.home-assistant.io/blog/#config-entry-options

[stbenjam]

Am I doing something wrong? Because it looks like this component is fundamentally flawed.

• There are two mechanisms to enable the component, right? Either configuration.yaml or on the integrations pane. I don't see how you would ever set it up in configuration.yaml, because you don't have fields anymore for username/password. I seem to always have to add it via integrations page so I can enter a username/password for my controller

• The workflow for adding the integration itself is flawed: there is no way to set any of the settings before creating the integration, so all the things I don't want (wired devices, unifi devices, etc) are always created the first time - and I have to go through the entity registry and disable the many dozens of them by hand

• Integrations page has no SSID filter, even if I do specify an SSID filter in configurations.yaml, it's ignored. I get all my IoT devices that are on a separate WLAN I don't want tracked.

• Also, it seems a bad precedent to set that my data in known_devices.yaml wasn't migrated and it was all just ignored, and I have to start over from scratch. Why couldn't you at least have looked at the MAC's it already knew about that were set to "track: False" and use that data in the component?

[Kane610]

Am I doing something wrong? Because it looks like this component is fundamentally flawed.

A common fault is that the wrong site name is given

• There are two mechanisms to enable the component, right? Either configuration.yaml or on the integrations pane. I don't see how you would ever set it up in configuration.yaml, because you don't have fields anymore for username/password. I seem to always have to add it via integrations page so I can enter a username/password for my controller

It is always configured through the integrations page. The configuration.yaml/config entry options are only to alter certain behaviours.

• The workflow for adding the integration itself is flawed: there is no way to set any of the settings before creating the integration, so all the things I don't want (wired devices, unifi devices, etc) are always created the first time - and I have to go through the entity registry and disable the many dozens of them by hand

This is a limitation of how the config entries work. Not everyone have the same use case as you. But I'm trying to follow up and improve as we go.

• Integrations page has no SSID filter, even if I do specify an SSID filter in configurations.yaml, it's ignored. I get all my IoT devices that are on a separate WLAN I don't want tracked.

That is not yet supported through config entry options. Haven't gotten to that yet.

• Also, it seems a bad precedent to set that my data in known_devices.yaml wasn't migrated and it was all just ignored, and I have to start over from scratch. Why couldn't you at least have looked at the MAC's it already knew about that were set to "track: False" and use that data in the component?

Apparently I missed this, I apologise.

[stbenjam]

Am I doing something wrong? Because it looks like this component is fundamentally flawed.

A common fault is that the wrong site name is given

• There are two mechanisms to enable the component, right? Either configuration.yaml or on the integrations pane. I don't see how you would ever set it up in configuration.yaml, because you don't have fields anymore for username/password. I seem to always have to add it via integrations page so I can enter a username/password for my controller

It is always configured through the integrations page. The configuration.yaml/config entry options are only to alter certain behaviours.

It doesn't seem to work though, I had it setup before I created the integration with dont_track_wired_clients: True and it still created entries for all of my wired devices

• The workflow for adding the integration itself is flawed: there is no way to set any of the settings before creating the integration, so all the things I don't want (wired devices, unifi devices, etc) are always created the first time - and I have to go through the entity registry and disable the many dozens of them by hand

This is a limitation of how the config entries work. Not everyone have the same use case as you. But I'm trying to follow up and improve as we go.

I'm not sure I understand the point of these settings if you can't set them before the integration creates the entities. FWIW, there's a ton of complaints about this on Discourse and reddit since the move to the new UniFi component. My use case isn't strange, I don't want the integration to create
hundreds of devices that I have to clean up with pointy clicky tools that's significantly slower than editing YAML.

• Integrations page has no SSID filter, even if I do specify an SSID filter in configurations.yaml, it's ignored. I get all my IoT devices that are on a separate WLAN I don't want tracked.

That is not yet supported through config entry options. Haven't gotten to that yet.

• Also, it seems a bad precedent to set that my data in known_devices.yaml wasn't migrated and it was all just ignored, and I have to start over from scratch. Why couldn't you at least have looked at the MAC's it already knew about that were set to "track: False" and use that data in the component?

Apparently I missed this, I apologise.

[Kane610]

Changing the options should disable the affected entities, isn't that the case?

[stbenjam]

Changing the options should disable the affected entities, isn't that the case?

I don't think so, since I had to use the entity registry to turn them all off. Is there a delay between changing the setting and the entities being disabled? I don't think I waited very long.

I can spin up an HA instance after work this evening to check.

[Kane610]

It should be near instant when updating options after unticking the include wired devices box

Just tried it on my dev instance and works as expected.

Please do try, it is better if I can focus on fixing the real problems :)

BTW I think your site name is wrong; I wouldn't name my site "Default"

[stbenjam]

Thanks, I tested it out and it does disable the relevant entities, however they are re-enabled after restarting HA, even though the relevant checkbox to not track them is still deselected.

To get it to not re-enable them, and have it persist across restarts, you have to disable both (1) Include wired network clients and (2) Enable newly added entities, otherwise wired clients are re-enabled on start-up.

Would you like me to file separate GitHub issues? one for SSID filtering, and one for the issue with wired clients being enabled after restart?

Thanks for taking the time to look at this.

BTW: What's wrong with Default? I only have one site, and I don't remember if I gave it that name or if UniFi did. It's not something I ever look at 🤷‍♂️

[Kane610]

Thanks, I tested it out and it does disable the relevant entities, however they are re-enabled after restarting HA, even though the relevant checkbox to not track them is still deselected.

Thats a step in the right direction but not home free yet! It seems I have the same issue as well. This is good! We're fixing what real issues there is!

To get it to not re-enable them, and have it persist across restarts, you have to disable both (1) Include wired network clients and (2) Enable newly added entities, otherwise wired clients are re-enabled on start-up.

That shouldn't be necessary. Some bug here...

Would you like me to file separate GitHub issues? one for SSID filtering, and one for the issue with wired clients being enabled after restart?

Sure. It's easier to track what is what.

Thanks for taking the time to look at this.

Hey! I want this integration to work as much as you do!

BTW: What's wrong with Default? I only have one site, and I don't remember if I gave it that name or if UniFi did. It's not something I ever look at 🤷‍♂

With the old implementation you used the site id to configure what you wanted, with the new it is the nice name that you give when creating the site

[Kane610]

Now I can't reproduce the change in wired devices. Did you restart too quickly perhaps?

[glassbase]

I would like to add that I have the unchecked "track wired clients" and "track network devices (Ubiquiti devices)" back when those options were added in 0.9x.x?? and manually deleted all the entities. I just noticed my entity registry filled with both wired and Ubiquiti devices again and all enabled. Wonder if there is something not working 100% with the code
edit - also all entities were enabled, including ones that I know I had previously disabled

[Kane610]

Hmm. They should be disabled if they get readded to the entity registry

[joshtbernstein]

I previously opened an issue on this and wanted to build on this discussion to give feedback to @Kane610 and @balloob.

I've honestly stopped using this component due to this change. It's nice that there is now some bit of control around what is tracked, but there's still the fact that this integration creates many entries in the entity registry without the ability to remove them or even just prevent them from being created in the first place. I have a lot of devices on my network and don't need nor want many of them in HA. But it really just clutters the entity registry and from my experience, disabled entities don't yet seem to be completed hidden or removed from all parts of HA.

I'm sure there was a lot of thought and discussion put into this change, but from a user perspective it seems that this change was made without a lot of forethought or testing. It's been pretty disruptive to me (and I think most users) and still isn't fully resolved. I realize that there's a reason to get away from known_devices.yaml, but this change should have only been implemented once it was confirmed that functionality and user experience would be the same for the users.

[Kane610]

@dummptyhummpty I've raised a question to front end developers if we can get some kind of toggle of showing enabled disabled entities in entity registry

[joshtbernstein]

@Kane610 That would be a good start. I think long term, it would be awesome to see the integration ask before adding entities and anything not selected by the user is never added to HA.

[balloob]

The entity registry should not be something one should aim on keeping clean. It's all entities that Home Assistant has ever seen. Devices on your network are part of this. If this is a page that you visit often, I would love to hear your use case for that page.

Yes, we can do a better job at management for devices/entities, including overviews, renaming etc. Constructive suggestions on how this can be done better is welcome in the Polymer repo.

[Kane610]

@dummptyhummpty try out the beta for 0.100, disabling entities is now supported on the entity registry view

[Kane610]

@stbenjam @glassbase upgrade to 0.100 beta and please verify if you still see issues with your disabling.

[joshtbernstein]

The entity registry should not be something one should aim on keeping clean. It's all entities that Home Assistant has ever seen. Devices on your network are part of this. If this is a page that you visit often, I would love to hear your use case for that page.

For me, physical world clutter is stressful and distracting, so the tendency to try and stay clean and organized caries over into my digital world. I do understand what you're saying about the entity registry, but for someone who likes to tweak and change things, it's nice to have the entity registry only show me things I care about. It looks like the change in 0.100beta may help with this.

I also do want to point out that the other part of my issue is that there are certain things that I just don't want in HA. So it's not only about keeping the entity registry clean, but also limiting what devices are actually imported in to HA. But I'm sure that's something that can be worked on.

Yes, we can do a better job at management for devices/entities, including overviews, renaming etc. Constructive suggestions on how this can be done better is welcome in the Polymer repo.

Awesome, I wasn't aware of that so I'll definitely try to hang out there and contribute.

[stbenjam]

I don't see the issue in 0.100b1 with the devices being recreated after restart like I saw before. I think that's working correctly. I'll close this. I filed #27243 to track SSID filtering.

I also noticed another oddity with the entity registry: #27242

Thanks all!

[charlietomo]

Whilst I appreciate this is closed I wanted to offer some constructive criticism to @Kane610 and any others.

I agree with @stbenjam that these breaking changes to Unifi were badly handled, resulting in a mess in the HA UI and no clear documentation or obvious way out. I understand that this is Open Source and some work is required on my side - but breaking a working system for no obvious reason seems like something was missed in this release process.

Specifically for me:

• everything on my network got added to my main HA page - despite track_new_devices: False being set and working for a long time
• disabling (commenting out) my unifi tracker config does not remove any of those items?!
• new unifi section in Integrations but I have not seen any documetation around this. It is a very messy page with lots of devices. I want to track 3 phone connections to the network; not every device that lives or connects on my network. I do not see how to enable or disable this?
• I now have a bunch of entities relating to all those network devices in Configuration > Entity Registry - which clutter HA interface and have no need to be known at all, in my use case, and they previous were excluded.

To add insult to injury, I manually went through the entity registry disabling all the entires I didn't need. Updating from .99 to .100 has turned them all back on! What a frustrating process.

If I have missed some crucial documentation notes then I apologise, but from the comments elsewhere on the internet it would appear I am not alone.