Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

2024.8.2 ZHA fails to start up with extra keys not allowed @ data['disable_new_entities'] #124160

Closed
Mark612 opened this issue Aug 18, 2024 · 34 comments · Fixed by #125710
Closed

Comments

@Mark612
Copy link

Mark612 commented Aug 18, 2024

The problem

Upgraded from 7.2 to 8.2, and ZHA fails all devices. Tried to reload, same error. Went back to 7.2 and all working fine.

What version of Home Assistant Core has the issue?

core-2024.8.2

What was the last working version of Home Assistant Core?

core-2024.7.2

What type of installation are you running?

Home Assistant Container

Integration causing the issue

ZHA

Link to integration documentation on our website

No response

Diagnostics information

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

2024-08-18 06:59:15.597 ERROR (DbWorker_0) [homeassistant.components.logbook.processor] Error with zha describe event for zha_event
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/logbook/processor.py", line 247, in _humanify
    data = describe_event(event_cache.get(row))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/zha/logbook.py", line 44, in async_describe_zha_event
    zha_device = async_get_zha_device_proxy(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/zha/helpers.py", line 1014, in async_get_zha_device_proxy
    zha_gateway_proxy = get_zha_gateway_proxy(hass)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/zha/helpers.py", line 993, in get_zha_gateway_proxy
    raise ValueError("No gateway object exists")
ValueError: No gateway object exists
2024-08-18 06:59:15.598 ERROR (DbWorker_0) [homeassistant.components.logbook.processor] Error with zha describe event for zha_event
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/logbook/processor.py", line 247, in _humanify
    data = describe_event(event_cache.get(row))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/zha/logbook.py", line 44, in async_describe_zha_event
    zha_device = async_get_zha_device_proxy(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/zha/helpers.py", line 1014, in async_get_zha_device_proxy
    zha_gateway_proxy = get_zha_gateway_proxy(hass)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/zha/helpers.py", line 993, in get_zha_gateway_proxy
    raise ValueError("No gateway object exists")
ValueError: No gateway object exists

Additional information

No response

@home-assistant
Copy link

Hey there @dmulcahey, @Adminiuga, @puddly, @TheJulianJES, mind taking a look at this issue as it has been labeled with an integration (zha) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of zha can trigger bot actions by commenting:

  • @home-assistant close Closes the issue.
  • @home-assistant rename Awesome new title Renames the issue.
  • @home-assistant reopen Reopen the issue.
  • @home-assistant unassign zha Removes the current integration label and assignees on the issue, add the integration domain after the command.
  • @home-assistant add-label needs-more-information Add a label (needs-more-information, problem in dependency, problem in custom component) to the issue.
  • @home-assistant remove-label needs-more-information Remove a label (needs-more-information, problem in dependency, problem in custom component) on the issue.

(message by CodeOwnersMention)


zha documentation
zha source
(message by IssueLinks)

@tbrausch
Copy link

I'm having the exact same problem. Once I upgade to 2024.8.x all of my Zigbee devices are unavailable. If I power off/on my Philips Hue lights I can get them to work again, but no luck with all of the Aqara devices. In addition, I did a ZHA backup prior to the installing the HA upgrade. I then tried to reload the ZHA backup and it reported an error. This same backup was verified to work in HA 2024.7.x

@dmulcahey
Copy link
Contributor

Folks we need debug logs to see what is up. Enable debug logging on ZHA, upgrade HA again and then after things fail disable debug logging and attach the downloaded log here. If the integration isn’t running after the upgrade and you can’t disable debug mode just attach the entire HA log here.

@tbrausch
Copy link

I've attached the log file here
ZHA backup 2024-08-18T17-56-20.676Z.json
Thanks for your help and let me know if you need anything else.

@dmulcahey
Copy link
Contributor

Nope, that is a diagnostics file. I need you to do what I outlined in the previous post...

@tbrausch
Copy link

Sorry, I clicked on the wrong file to upload. Is this better?
home-assistant_zha_2024-08-18T20-09-12.307Z.log

@Mark612
Copy link
Author

Mark612 commented Aug 18, 2024

Here is what happens on a reload:

2024-08-18 17:22:00.226 ERROR (MainThread) [homeassistant.config_entries] Error setting up entry HubZ Smart Home Controller, s/n: _xxxxx_DD - Silicon Labs for zha
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/config_entries.py", line 604, in async_setup
result = await component.async_setup_entry(hass, self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/components/zha/init.py", line 124, in async_setup_entry
zha_lib_data: ZHAData = create_zha_config(hass, ha_zha_data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/components/zha/helpers.py", line 1224, in create_zha_config
zha_options = CONF_ZHA_OPTIONS_SCHEMA(options.get(ZHA_OPTIONS, {}))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/voluptuous/schema_builder.py", line 205, in call
return self._compiled([], data)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/voluptuous/schema_builder.py", line 549, in validate_dict
return base_validate(path, data.items(), out)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/voluptuous/schema_builder.py", line 382, in validate_mapping
raise er.MultipleInvalid(errors)
voluptuous.error.MultipleInvalid: extra keys not allowed @ data['disable_new_entities']
2024-08-18 17:22:02.262 ERROR (MainThread) [aiohttp.server] Error handling request
Traceback (most recent call last):
File "/usr/local/lib/python3.12/site-packages/aiohttp/web_protocol.py", line 356, in data_received
messages, upgraded, tail = self._request_parser.feed_data(data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "aiohttp/_http_parser.pyx", line 563, in aiohttp._http_parser.HttpParser.feed_data
aiohttp.http_exceptions.BadStatusLine: 400, message:
Invalid method encountered:

b'\x16\x03\x01'
  ^

@puddly
Copy link
Contributor

puddly commented Aug 18, 2024

@Mark612 Can you post your Home Assistant config, specifically the zha section?

@Mark612
Copy link
Author

Mark612 commented Aug 19, 2024

@Mark612 Can you post your Home Assistant config, specifically the zha section?

config.txt

@puddly
Copy link
Contributor

puddly commented Aug 19, 2024

I wonder where this config is coming from?

    "config_entry": {
      "data": {
        "device": {
          "baudrate": 57600,
          "path": "/dev/ttyUSB1",
          "flow_control": null
        },
        "radio_type": "ezsp"
      },
      "disabled_by": null,
      "domain": "zha",
      "entry_id": "087ff3f5076411eba92f81ca128d68a2",
      "minor_version": 1,
      "options": {
        "custom_configuration": {
          "zha_options": {
+           "disable_new_entities": false,
            "enable_identify_on_join": true,
            "consider_unavailable_battery": 21600,
            "consider_unavailable_mains": 7400,
            "default_light_transition": 0,
            "enhanced_light_transition": false,
            "light_transitioning_flag": true,
            "always_prefer_xy_color_mode": true,
            "group_members_assume_state": true
          }
        }
      },
      "pref_disable_new_entities": false,
      "pref_disable_polling": false,
      "source": "user",
      "title": "HubZ Smart Home Controller, s/n: 314004DD - Silicon Labs",
      "unique_id": null,
      "version": 4

@Mark612
Copy link
Author

Mark612 commented Aug 19, 2024

It could be from the Z-Wave. The controller does both Z-Wave and Zigbee protocols.

@GregoryPye
Copy link

GregoryPye commented Aug 19, 2024

Same issue, though more intermittent, in case another source of data useful.

@sKalach
Copy link

sKalach commented Aug 20, 2024

I have exactly the same problem and rolled back. Tried two usb sticks - Conbee II & Zi-Stick. Hardware is Raspberry Pi 5.

@puddly
Copy link
Contributor

puddly commented Aug 20, 2024

Please attach debug logs and diagnostic information. Without this, there's not much that can be done to fix this problem.

@sKalach
Copy link

sKalach commented Aug 20, 2024 via email

@puddly
Copy link
Contributor

puddly commented Aug 20, 2024

@sKalach I think GitHub doesn't accept files over email, unfortunately. Could you attach it to a comment? Thanks!

@Sfre12
Copy link

Sfre12 commented Aug 22, 2024

Yesterday I startet Upgrade from 2024.7.4 to 2024.8.2. And now I have the same problem...
Will restore to old backup

@puddly
Copy link
Contributor

puddly commented Aug 22, 2024

@Sfre12 Please attach debug logs.

@martin3000
Copy link
Contributor

Can you see your zigbee USB stick? Maybe by issuing the command "lsusb" on the command line?
What installation method did you use for HA? HAOS? Or native core?

@waltailji
Copy link

waltailji commented Aug 23, 2024

i've been having the same issues with my zigbee network and had to roll back my home assistant version twice (i tried 2024.8.0 and 2024.8.2). the issue only occurs with my battery operated devices, mainly aqara branded buttons, temperature sensors, and leak detectors. all of my mains powered zigbee devices, like smart plugs and light bulbs, still show up fine. if i try to readd my aqara devices they get stuck in "configuring" mode. when i roll back, even after a few hours on 2024.8, my whole zigbee network eventually heals and shows back up. usb stick is a nortek-branded sigma designs HUSBZB-1. my home assistant installation is running on the "home assistant blue" o-droid n2+.

@puddly puddly changed the title 2024.8.2 ZHA Fails 2024.8.2 ZHA fails to start up with extra keys not allowed @ data['disable_new_entities'] error Aug 23, 2024
@stevied65
Copy link

Same problem

@dam57950
Copy link

Same problem, ha web interface sort of reload every few minutes with "disconnected" at the bottom left of the screen and ZHA constantly try to initialize every time it come back to life but can't do it most of the time and even if it can, it's only for little time before it goes down again

@puddly
Copy link
Contributor

puddly commented Aug 25, 2024

@stevied65 @dam57950 Please attach debug logs.

@dam57950
Copy link

Capture d'écran 25 08 2024 à 15 17 28 PM


Capture d'écran 25 08 2024 à 15 17 39 PM


Capture d'écran 25 08 2024 à 15 18 07 PM


Capture d'écran 25 08 2024 à 15 18 21 PM


Capture d'écran 25 08 2024 à 15 18 28 PM


Capture d'écran 25 08 2024 à 15 18 35 PM


Capture d'écran 25 08 2024 à 15 18 45 PM


Capture d'écran 25 08 2024 à 15 18 52 PM


Capture d'écran 25 08 2024 à 15 19 03 PM


Capture d'écran 25 08 2024 à 15 19 10 PM


@Sfre12
Copy link

Sfre12 commented Sep 1, 2024

@Sfre12 Please attach debug logs.

Hey.
I already switched back to the latest Backup from before and it is running again. That is the reason I have no Debug file

@Sfre12
Copy link

Sfre12 commented Sep 1, 2024

Can you see your zigbee USB stick? Maybe by issuing the command "lsusb" on the command line? What installation method did you use for HA? HAOS? Or native core?

I am Using a sonoff zigbee bridge, connected by wifi to my HA
I have HAOS
After restoring to the last backup all is running very well

@Mark612 Mark612 changed the title 2024.8.2 ZHA fails to start up with extra keys not allowed @ data['disable_new_entities'] error 2024.8.2 ZHA fails to start up Sep 4, 2024
@Mark612
Copy link
Author

Mark612 commented Sep 6, 2024

I tried 9.1, get the same errors. So what changed from version 7? Why am I one of the few that have this problem?

File "/usr/src/homeassistant/homeassistant/config_entries.py", line 1874, in async_setup
raise OperationNotAllowed(
homeassistant.config_entries.OperationNotAllowed: The config entry 'HubZ Smart Home Controller, s/n: 3xxx04DD - Silicon Labs' (zha) with entry_id '087xxxxxxd68a2' cannot be set up because it is in state ConfigEntryState.SETUP_ERROR, but needs to be in the ConfigEntryState.NOT_LOADED state

@martin3000
Copy link
Contributor

I see a lot of different errors here with different reasons.
Just because your zigbee does not run after an update does not mean that it is always the same error!

@puddly
Copy link
Contributor

puddly commented Sep 6, 2024

@Mark612 Please do not edit the issue title, I changed it because "fails to start up" is extremely vague. The integration can fail to start up for a multitude of reasons and yours fails due to a specific problem.

@puddly puddly changed the title 2024.8.2 ZHA fails to start up 2024.8.2 ZHA fails to start up with extra keys not allowed @ data['disable_new_entities'] Sep 6, 2024
@Mark612
Copy link
Author

Mark612 commented Sep 10, 2024

What options do I/we have to not get further behind on releases? I'm still running on 2024.7. Is there an ability to run the 2024.7 ZHA architecture on 2024.9?

@Mark612
Copy link
Author

Mark612 commented Sep 13, 2024

Update confirmed to fix the issue. Huge appreciation puddly and team. Thank you!

@hugojmj
Copy link

hugojmj commented Oct 3, 2024

Hi I have still have this issue, have been trying to upgrade from 2024.7 now to 2024.10. My config HAOS on Raspberry Pi4 with a Sonoff Dongle-E. Debug log attached shows [zigpy.quirks] Fail because endpoint list mismatch but I don't see what causes this. Appreciate your guidance to find the cause.
home-assistant_zha_2024-10-03T07-10-58.343Z.log

@puddly
Copy link
Contributor

puddly commented Oct 3, 2024

@hugojmj Your issue is unrelated. Can you create a new one?

@github-actions github-actions bot locked and limited conversation to collaborators Nov 2, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging a pull request may close this issue.