Merge info_object into state and emit state change diffs

[puddly]

This PR replaces the state dict with a strongly typed dataclass that absorbs info_object. We now also emit state change diffs in EntityStateChangedEvent's new state_diff attribute.

For stronger typing and to support the client/server architecture split in the future, we benefit from having more centralized state for entities. This will allow us to send batched state updates to the client instead of the client being notified of a general "update" and then reaching for the state dict of the ZHA entity.

The HA Core half is here: home-assistant/core@dev...puddly:core:puddly/zha-state-migration. It uses the state_diff values and also adds dynamic capability recomputation.

Note: I intentionally regenerated diagnostics and reverted the regeneration commit so that this diff is minimal. Otherwise, it'll be a hundred of thousand lines of code. This will be un-reverted before merge.

---

Part of OpenHomeFoundation/roadmap#93 and a continuation of #663.

[Copilot]

Pull request overview

This PR migrates ZHA entity and group “state” from untyped dicts (and info_object) to strongly typed dataclass-based state objects, and adds per-change diffs (state_diff) to EntityStateChangedEvent to support more efficient downstream consumers (e.g., Home Assistant Core).

Changes:

• Replace BaseEntityInfo / info_object usage with BaseEntityState and typed per-platform *State dataclasses across platforms and diagnostics.
• Emit EntityStateChangedEvent.state_diff computed from previous vs current state, and add event emission suppression for new-entity bootstrap.
• Update tests and mypy configuration to match the new typed-state API.

Reviewed changes

Copilot reviewed 39 out of 39 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
zha/zigbee/group.py	Switch group member/group “info” payloads from info_object to typed state.
zha/zigbee/device.py	Emit state-changed events after entity recomputation and adjust diagnostics output to state dataclasses.
zha/event.py	Add suppress_events() context manager and make EventBase.emit() suppression-aware.
zha/application/platforms/update.py	Replace dict state + info_object with UpdateState dataclass state.
zha/application/platforms/switch.py	Replace dict state + info_object with SwitchState/ConfigurableAttributeSwitchState.
zha/application/platforms/siren.py	Replace dict state + info_object with SirenState and adjust squawk typing.
zha/application/platforms/sensor/init.py	Replace dict state + info_object with typed sensor state dataclasses; adjust attribute handling.
zha/application/platforms/select.py	Replace dict state + info_object with SelectState/EnumSelectState.
zha/application/platforms/number/init.py	Replace dict state + info_object with NumberState.
zha/application/platforms/lock/init.py	Replace dict state with typed LockState dataclass and refine lock-state typing.
zha/application/platforms/light/init.py	Replace dict state + info_object with LightState dataclass state.
zha/application/platforms/helpers.py	Update aggregation helpers to work with BaseEntityState dataclasses.
zha/application/platforms/fan/init.py	Replace dict state + info_object with FanState dataclass; update group aggregation.
zha/application/platforms/device_tracker.py	Replace dict state with typed DeviceTrackerState.
zha/application/platforms/cover/init.py	Replace dict state with typed CoverEntityState.
zha/application/platforms/climate/init.py	Replace dict state + info_object with ClimateState/ThermostatState.
zha/application/platforms/button/init.py	Replace info_object with typed button state dataclasses.
zha/application/platforms/binary_sensor/init.py	Replace dict state + info_object with BinarySensorState.
zha/application/platforms/alarm_control_panel/init.py	Replace dict state + info_object with AlarmControlPanelState.
zha/application/platforms/init.py	Introduce BaseEntityState, add state_diff + compute_state_diff(), and switch base/platform/group state to dataclasses.
zha/application/gateway.py	Emit group gateway messages using zha_group.state.
tests/test_update.py	Update update-entity assertions to typed state accessors.
tests/test_switch.py	Update switch assertions to typed state accessors.
tests/test_siren.py	Update siren assertions to typed state accessors.
tests/test_sensor.py	Update sensor assertions to typed state accessors and extra-state handling.
tests/test_select.py	Update select assertions to typed state accessors.
tests/test_number.py	Update number assertions to typed state accessors.
tests/test_lock.py	Update lock assertions to typed state accessors.
tests/test_light.py	Update light assertions to typed state accessors.
tests/test_gateway.py	Update gateway assertions to typed state accessors.
tests/test_fan.py	Update fan assertions to typed state accessors.
tests/test_discover.py	Update discover assertions to typed state accessors.
tests/test_device.py	Update device assertions to typed state accessors.
tests/test_device_tracker.py	Update device tracker assertions to typed state accessors.
tests/test_cover.py	Update cover assertions to derive cover state from typed boolean fields.
tests/test_climate.py	Update climate assertions to typed state accessors.
tests/test_alarm_control_panel.py	Update alarm control panel assertions to typed state accessors.
tests/common.py	Update helper selection logic from info_object.unique_id to state.unique_id.
pyproject.toml	Update mypy overrides to reflect the refactor and reduced ignore needs.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[zigpy-review-bot]

Comment. I reviewed the substantive parts of this change (the info_object→state merge, compute_state_diff, suppress_events, and the _add_pending_entities emit changes), ran the non-snapshot tests, and checked types. The approach looks sound — there's one real test failure hiding in the otherwise-expected red CI, plus a verdict on the Copilot bot's three threads.

Must-fix: one leftover info_object fails CI on its own

tests/test_sensor.py:911 (in test_analog_input_bogus_resolution) still reads entity.info_object.suggested_display_precision, which this PR removes → AttributeError: 'AnalogInputSensor' object has no attribute 'info_object' for both the [0.0] and [inf] params. It came in via #838 after your rename pass over this file, so it slipped through — the sibling asserts at lines 811/886 already use entity.state.…. Same fix:

assert entity.state.suggested_display_precision is None

This is not one of the reverted-diagnostics failures — it fails independently of the snapshot regen and stays red after the un-revert. It also isn't in the current CI run (which predates the #838 merge), so current CI doesn't show it yet. (The Copilot review flagged this one as well.)

Expected red (no action)

The ~851 test_discover.py::test_devices_from_files[…] failures and test_device_override_entities are all the intentionally-reverted diagnostics — the old-format snapshots still carry the nested info_object/state blocks — matching your PR note. Nothing to do there beyond the planned un-revert.

On the Copilot review's other two threads

• compute_state_diff KeyError (platforms/__init__.py): not triggerable in practice — old and new are always the same entity's state, so they're the same dataclass type with identical keys, and the old is None first-emit case is already handled. A symmetric .get() would be defense-in-depth only, not required.
• siren.py # type:ignore missing a space: non-issue — mypy honors the no-space form. Verified locally (# type:ignore[assignment] does suppress the error), and CI's mypy hook is green.

Otherwise looks good

• suppress_events() — the ContextVar + sync-only with block in _add_pending_entities is task-safe: there's no await inside the suppressed region, so the flag can't leak into concurrently-scheduled tasks. The baseline-emit-under-suppression for new entities, plus the follow-up emit for existing entities whose primary/capabilities changed, reads correctly.
• mypy zha/ is clean on the changed source files. The # type: ignore[misc] on the Switch/SwitchGroup MRO overrides is legitimate — it's return-type variance only; dataclasses.replace(super().state, …) preserves the concrete SwitchState, so is_on survives at runtime (confirmed via the produced SwitchState(…, is_on=…)).

Not approving since it's pre-merge WIP with the diagnostics reverted, but the design is solid — just fold in the test fix.

[zigpy-review-bot]

Comment. Re-review at 1ccfbffc. The tests/test_sensor.py leftover from the last round is fixed. Locally: mypy zha/ is clean — 0 errors against the ~76-error dev baseline, ruff is clean, 467/467 non-snapshot tests pass, and the 849 test_discover failures are exactly the reverted regen you flagged in the description. Nothing fails independently of the snapshot revert.

I also walked every _zha_state.<field> your Core branch reads against the corresponding new dataclass. Two of them don't resolve.

Must-address

1. DeviceCounterSensor.state.available is always None. It's the only concrete entity subclassing BaseEntity directly, so it never passes through PlatformEntity.state/GroupEntity.state, which are what fill available. Inline below — measured values and impact there.

2. BitmapSensorState advertises extra_state_attribute_names that aren't fields on it. Raises AttributeError in the Core half for every Danfoss bitmap sensor. Inline below.

3. Bump DIAGNOSTICS_JSON_VERSION. zha/zigbee/device.py:107 is still 2, but get_diagnostics_json now emits a different zha_lib_entities shape. #657 bumped 1 → 2 for exactly this kind of restructure.

4. The new machinery has no tests. compute_state_diff, state_diff and suppress_events have zero direct coverage — grepping tests/ for any of the three returns nothing. Given the premise is that a consumer can reconstruct state from diffs alone, worth asserting a diff payload's contents, that a newly-added entity's baseline is not emitted, and that dataclasses.replace(cached, **diff) round-trips back to entity.state.

Design note, mostly for the Core half

The diff protocol assumes a consumer's baseline read and its listener attach are atomic. In the Core branch they aren't: ZHAEntity.__init__ captures self._zha_state = entity.state, but the STATE_CHANGED listener is only attached in async_added_to_hass. ZHA advances __previous_state on every emit whether or not anyone is listening, so anything emitted in that window is lost permanently rather than merely late — under the old re-read-everything model a missed event was self-healing. The re-sync at the end of async_added_to_hass doesn't cover it either, since it sits below if (state := await self.async_get_last_state()) is None: return. Moving that assignment above the early return closes the window.

The ZHA side of this reads correctly, for what it's worth: the suppress_events() baseline and the follow-up emit for existing entities whose primary/capabilities changed are both right, the suppressed block contains no await so the ContextVar can't leak into concurrent tasks, and every runtime recompute_capabilities() call site is either inside __init__ or immediately followed by maybe_emit_state_changed_event().

Nits

• EntityStateChangedEvent is now unhashable (frozen dataclass with a dict field). Nothing hashes it today, but it's a silent behaviour change on a public event type.
• BaseBinarySensor and BaseSelectEntity declare is_on / options but their state no longer carries them — those moved down to BinarySensor / EnumSelectState. Every concrete subclass in tree is fine, so this is only about a base class no longer describing itself; compare BaseSwitch/BaseLock/BaseSiren, which kept theirs.
• BitMapSensor.formatter recomputes the same per-bit dict that state now builds — could read self.state.bit_states.

Otherwise

The typing payoff is real: zha.application.platforms.{button,update,siren,lock} all dropped out of the pyproject.toml mypy override list, and the remaining zha/ tree type-checks clean. The # type: ignore[misc] on the Switch/SwitchGroup MRO overrides is legitimate (return-type variance only — I traced the cooperative super() chain and the concrete SwitchState/FanState/LightState survive with available correctly filled by PlatformEntity/GroupEntity).

I ran a second-opinion pass with GPT-5.6 Sol; it independently flagged the DeviceCounterSensor placeholders below. Its only other finding was the system_mode → sys_mode rename, which I checked and is fine — your Core branch maps it explicitly.

Not approving while the diagnostics regen is reverted, but the shape of this is good.

[zigpy-review-bot]

Nit: this guard is unrelated to the state migration and isn't mentioned in the description — it turns what used to raise into a silent None. Worth a short comment saying which case it covers (a quirks v2 entity naming an attribute the cluster doesn't define, presumably), so it doesn't read as unexplained defensive code later.

[zigpy-review-bot]

Nit: the pass is redundant — the docstring already forms the body.

[zigpy-review-bot]

Comment. Re-review at 6aa5076b. Everything from the 1ccfbffc round is addressed, and subscribe_state() is a better fix for the baseline/listener race than what I suggested — flushing the pending diff to existing subscribers before registering, then handing the new subscriber the full state, makes the baseline coherent with the diff stream by construction instead of by careful ordering on the consumer side. I checked the Core branch calls it below async_get_last_state() with no early return in the way, so the window I described is closed.

Locally at this head: mypy zha/ is 0 errors (vs the ~76-error dev baseline), ruff check and ruff format --check are clean, and 471/471 non-snapshot tests pass. The test_discover failures are the reverted regen per your PR note.

One new must-address, from re-walking the Core branch's state reads — that branch moved since my last round.

Must-address: the Core half now reads state.extra_state_attributes, which this PR doesn't provide

Your Core commit 5ff665f ("Clean up extra state attributes") replaced the per-name getattr(self._zha_state, name) resolution with a single mapping read:

extra_state_attributes = self._zha_state.extra_state_attributes

That's a cleaner answer to last round's BitmapSensorState finding than either option I offered — but the ZHA half of it isn't in this PR. At 6aa5076b the only related field is extra_state_attribute_names: frozenset[str]; no state dataclass declares or computes extra_state_attributes:

hasattr(BaseEntityState, "extra_state_attributes")  -> False
BitmapSensorState           fields tail: [..., 'bit_states']
ElectricalMeasurementState  fields tail: [..., 'max_value', 'max_attribute_name']

I extracted every distinct field name the Core branch reads off an entity-state object (self._zha_state.X, plus the meta = … / state = … aliases in entity.py, climate.py and light.py) and checked each against every BaseEntityState subclass. extra_state_attributes is the only one that doesn't resolve — the other non-matches (network_info, node_info, attributes, state) are gateway state and HA State objects, not entity state.

So Sensor.extra_state_attributes raises AttributeError for every sensor that sets extra_state_attribute_names: battery, electrical measurement, smart-energy summation, and the Danfoss bitmaps. It also leaves bit_states and max_attribute_name without a consumer, which reads like they're meant to be that mapping's inputs.

A computed property rather than a field is probably what you want here: it recomputes from the carried fields after dataclasses.replace, so it stays out of state_diff and doesn't duplicate data on the wire — BitmapSensorState returns bit_states, ElectricalMeasurementState maps max_attribute_name → max_value, and the rest resolve their extra_state_attribute_names off self. My earlier BitmapSensorState thread is the same issue in a different shape, so I'm leaving it open under this one.

(GPT-5.5 flagged this independently as well.)

Verified fixed — no action

• DeviceCounterSensor.state.available / .device_ieee — now filled via replace(super().state, …); the new asserts in test_device_counter_sensors pass.
• DIAGNOSTICS_JSON_VERSION → 3.
• Coverage for the new machinery — tests/test_platform_entity.py covers compute_state_diff, state_diff and suppress_events, and test_state_diff_round_trip pins exactly the property the Core half rests on (replay the diff stream from scratch, compare against entity.state). That's the one I most wanted; good.

Optional

Two small robustness points on subscribe_state, inline. Neither is a shipstopper.

[TheJulianJES]

Other than the tests / diagnostics regen, everything looks good to me. This is a really awesome improvement!

The HA Core side also looks good*. I'm not completely sure what to think about partly using _update_capability_attrs to update the "shorthand" _attr_something vs overriding the methods for those states but it all looks good for the next beta. (Maybe we do want to keep that approach and not cache everything in the library? I'm still a bit torn... 😅)

(*: test_diagnostics.ambr needs to be updated)

[puddly]

Regarding _update_capability_attrs: we effectively recompute the whole state on every event with the current approach in my branch. We could do it more granularly by only calling _update_capability_attrs on actual capability attribute updates? Or use properties, as you said, and let Core re-read them and compute changes?

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.26%. Comparing base (d7471c3) to head (ed1e8eb).

Additional details and impacted files

@@            Coverage Diff             @@
##              dev     #804      +/-   ##
==========================================
- Coverage   97.27%   97.26%   -0.01%     
==========================================
  Files          55       55              
  Lines       10941    10906      -35     
==========================================
- Hits        10643    10608      -35     
  Misses        298      298              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[TheJulianJES]

Regarding _update_capability_attrs: we effectively recompute the whole state on every event with the current approach in my branch.

Hmm, I'm not sure that's true? In HA, self.async_write_ha_state() does the heavy lifting* and shouldn't be called unnecessarily all the time. But just updating the short-hand _attrs from the ZHA state all the time, even if they're unchanged, shouldn't have any performance impact. "Basically nothing happens when you update a normal attribute".

(*: For changed capabilities that need to be updated in the entity registry, async_write_ha_state already checks if they really need to be updated: https://github.com/home-assistant/core/blob/b52e8646a819c47e8327a5b68d764b025d191d94/homeassistant/helpers/entity.py#L1217-L1226)

And async_write_ha_state() is just called on any "real events", which may have caused the other state attributes to change, so I think that's fine. That's the same behavior as before.

Only some things like _convert_supported_features are a bit unnecessary because they actually redo some calculation in HA, even though they're likely unchanged 99% of the time, but I also feel like the performance impact shouldn't be huge there to recalculate the enum.

(@zigpy-review-bot Verify/check everything and answer.)

Or did you mean something else?

[TheJulianJES]

Hmm, well. I guess the recalculations done in _update_capability_attrs for fan + light + siren + cover aren't ideal. I think others like select + number + climate are completely fine though – those just set HA attributes to ZHA ones.

I guess an easy way to "fix" this would be to make methods like _convert_supported_features (for cover) cached depending on the zha_features: ZHACoverEntityFeature, for example. That way, they'd only construct new HA enums from the ZHA ones if it's really necessary. But maybe a different approach might be nicer? (@zigpy-review-bot also consider this comment before answering)

[zigpy-review-bot]

Checked both of your comments. Short version: the first one is right — the _attr_* writes really are ~free and async_write_ha_state() isn't called more often than on dev — and the second one is right too, but the cost isn't spread over "fan + light + siren + cover": it's concentrated almost entirely in flag in zha_features membership tests on IntFlag, ~0.5 µs each. So caching the conversions is the correct lever, and it applies to climate as well, which you listed as fine.

The three claims, verified

1. Re-assigning an unchanged _attr_* costs nothing meaningful — confirmed, and for a more specific reason than "it's a normal attribute". On entity classes it isn't a normal attribute: CachedProperties replaces every _attr_<x> that backs a cached_property with a generated property whose setter early-returns (homeassistant/helpers/entity.py L353-356 on your branch):

if (old_val := getattr(o, private_attr_name, _SENTINEL)) == val and type(old_val) is type(val):
    return
setattr(o, private_attr_name, val)
o.__dict__.pop(name, None)   # invalidate the cached_property

So an unchanged write is one comparison (0.15 µs) and — importantly — does not invalidate the cached_property. The one trap is the type(old_val) is type(val) half: a value that compares equal but is rebuilt as a different type would invalidate on every event. I checked all eight _update_capability_attrs implementations (light, climate, cover, fan, siren, select, number, plus the no-op base) — each one always yields the same type, so no spurious invalidation.

2. async_write_ha_state() already guards the entity-registry update — confirmed. _async_write_ha_state compares capabilities / original_device_class / supported_features / original_name against registry_entry and only calls async_update_entity when one differs, exactly as you linked. And there's a backstop if capabilities ever genuinely flap: CAPABILITIES_UPDATE_LIMIT = 100 per hour, after which HA logs "is updating its capabilities too often".

3. "Called only on real events, same behavior as before" — confirmed, and the branch is actually a small behavior fix. ZHA still only emits when the state differs (zha/application/platforms/__init__.py:469 — if previous_state != state), so _handle_zha_entity_state_changed runs once per real change. dev's HA side subscribed via on_all_events rather than subscribe_state, but STATE_CHANGED is the only event a platform entity emits, so the write frequency is identical. The difference that isn't neutral: on dev, _attr_supported_color_modes and _attr_supported_features were computed once in Light.__init__ and never updated again — a light whose capabilities are learned later kept the stale set forever. _update_capability_attrs fixes that.

Where the time actually goes

Per ZHA event, HA side, light entity:

step	µs
dataclasses.replace(self._zha_state, **state_diff)	8.4
_update_capability_attrs() — climate	6.1
_update_capability_attrs() — cover	4.2
_update_capability_attrs() — light	3.7
_update_capability_attrs() — select / number	≈0.2–0.7 (assignments only)
capability_attributes + state_attributes (HA, per write)	2.5
hass.states.async_set — attrs changed / unchanged	6.7 / 4.7

Breaking _update_capability_attrs down is what makes the fix obvious:

component	µs
cover: the 7 flag in zha_features tests	3.84 (of 4.2)
climate: the 8 flag in zha_features tests	4.69 (of 6.1)
climate: [ZHA_TO_HA_HVAC_MODE[m] for m in hvac_modes]	0.16
light: the 3 flag in zha_features tests	≈1.9
light: color-mode set build	0.77
light: 2× color_temperature_mired_to_kelvin	0.23
one unchanged _attr_x = … write	0.15

Your instinct about _convert_supported_features was the right target; the surprise is that climate has the same problem (its 8 flag tests are the single most expensive capability recompute of any platform), while the parts that look expensive — building a set, a list comprehension, float conversions — are noise.

On the caching idea: yes, and it's dramatic

Wrapping the conversion in functools.cache (pure function of a hashable IntFlag): 3.83 µs → 0.08 µs, i.e. down to a dict lookup. Worth doing for cover, fan, siren, and the feature half of light and climate. Two notes: put it on a module-level function or a staticmethod (never a bound method — it would pin self in the cache), and the key space is a handful of flag combinations per platform, so an unbounded @cache is fine; the cached values are immutable IntFlags, so sharing them is safe.

The "different approach" I measured that is not nicer: iterating the set bits and mapping them (for bit in zha_features: features |= MAP[bit]) is 3.59 µs — barely better than the branches, because constructing the flag members during iteration is itself the slow part. Caching is what buys the 45×.

Not worth touching: light's color-mode set (0.77 → 0.24 µs, needs a frozenset key since supported_color_modes is a set, and you'd be sharing a mutable set), the mired conversions, and select/number (plain assignments).

On the original question — granular calls, or properties

Granular gating (if CAPABILITY_FIELDS & event.state_diff.keys()) costs 0.2 µs and would save the 4–6 µs, but it needs a per-platform list of capability field names kept in sync with the state dataclass by hand — a silent-staleness risk for a modest win. With the conversions cached there's ~0.3 µs left to gate, so it stops being worth it. Cache, don't gate.

Properties instead wouldn't reduce the work: capability_attributes is a plain @property (not cached) in light, climate and number, so HA rebuilds it — reading supported_features, hvac_modes, min_temp, … — on every state write regardless. Overriding with @property would also bypass the cached_property layer entirely, so every reader outside the write path recomputes too. The _attr_ + _update_capability_attrs shape fits HA's invalidation model better; I'd keep it.

For scale: even uncached, 6 µs at 100 entity state changes per second is ~0.06% of one core. The largest single item on this path is dataclasses.replace at 8.4 µs (copy.copy + __dict__.update measures 2.1 µs, but it bypasses __init__ on a frozen dataclass — only worth considering if a profile ever points here).

How this was measured

Python 3.14.5, timeit at 200k iterations per case, on your puddly/zha-state-migration core branch (37c6bd0) plus this PR's head (64218f0).

• HA-side numbers: real HA classes (LightEntity, CoverEntity, ClimateEntity) instantiated bare, with the branch's _update_capability_attrs bodies transcribed verbatim and the ZHA-side enums stood in by the HA ones (same dict-lookup / flag-OR shape, so the cost profile is preserved). color_util and the CachedProperties setters are the real ones.
• hass.states.async_set was measured on a real HomeAssistant object with no listeners attached — in a live install the recorder and websocket subscribers add substantially more, which is the "heavy lifting" your first comment refers to.
• ZHA-side numbers (dataclasses.replace, compute_state_diff, dataclass __eq__) used a real LightState (31 fields) built from this PR's code: __eq__ 0.06 µs when equal / 0.91 µs when one field differs, compute_state_diff 2.2 µs.

Absolute values are machine-specific; the ratios are the point.

[puddly]

I'm surprised the IntFlag bitwise operations are so expensive. I think this should work?

class ZHASupportedFeaturesEntity(ZHAEntity):
    """ZHA entity whose state carries `supported_features`."""

    @override
    def _update_capability_attrs(self) -> None:
        self._attr_supported_features = self._convert_supported_features(
            self._zha_state.supported_features
        )

    @staticmethod
    def _convert_supported_features(zha_features: IntFlag) -> IntFlag:
        raise NotImplementedError

And then:

# fan.py
class Fan(ZHASupportedFeaturesEntity, FanEntity):
    @staticmethod
    @functools.cache
    @override
    def _convert_supported_features(
        zha_features: ZHAFanEntityFeature,
    ) -> FanEntityFeature:
        features = FanEntityFeature(0)
        if ZHAFanEntityFeature.DIRECTION in zha_features:
            features |= FanEntityFeature.DIRECTION
        # ... unchanged
        return features

It speeds things up in my local tests from 4000ns to 73ns.