BIG rework of integration Xiaomi Gateway 3

[AlexxIT]

IMPORTANT

• The version is under development and will be called v4.0.0
• This post has been updated over time
• It is highly recommended to make a backup before upgrading!

Why? With the yet another Home Assistant update, the mechanism of creating entities was broken. They only worked when the Home Assistant was started. And became unavailable with any restart of the integration.

Creating entities is a very complicated part of integration. Some may be created on the first connection to the gateway. Some may appear when new devices are connected. If there are no converters for the device - entities can be created when the first data from the device appears.

But the most difficult part is that multiple integrations (gateways) may try to create the same entity. This works for BLE, Mesh, and sometimes even Zigbee devices. This behavior is not supported by the Home Assistant core. And needs to be coded very carefully.

So I decided to rework the entire component once again.

Breaking changes

A lot of things have changed in the integration and something may not work and something has to be manually updated.

Converters

All converters moved to /xiaomi_gateway3/core/devices.py.
All entities descriptions moved to /xiaomi_gateway3/hass/entity_description.py.

Basic converter class simplified

# before
@dataclass
class Converter:
    attr: str  # hass attribute
    domain: Optional[str] = None  # hass domain

    mi: Optional[str] = None
    parent: Optional[str] = None

    enabled: Optional[bool] = True  # support: True, False, None (lazy setup)
    poll: bool = False  # hass should_poll

    # don't init with dataclass because no type:
    childs = None  # set or dict? of children attributes
    zigbee = None  # str or set? with zigbee cluster

# after
@dataclass
class BaseConv:
    attr: str
    domain: str = None
    mi: str | int = None
    entity: dict = None

Deprecated enabled=False

• Moved to entity.enabled=False param

# before
BoolConv("led", "switch", mi="6.p.6", enabled=False),
# after
BoolConv("led", "switch", mi="6.p.6", entity={"enabled": False}),

Deprecated enabled=None

• Moved to entity.lazy=True param

# before
Converter("battery", "sensor", enabled=None),  # no in new firmwares
# after
Converter("battery", "sensor", entity={"lazy": True}),

Deprecated parent="..."

• Now the entities know what attributes they need to subscribe to

# before
{
    "lumi.light.aqcn02": ["Aqara", "Bulb CN", "ZNLDP12LM"],
    "spec": [
        BoolConv("light", "light", mi="4.1.85"),
        ZXiaomiBrightnessConv("brightness", mi="14.1.85", parent="light"),
        ZXiaomiColorTempConv("color_temp", mi="14.2.85", parent="light"),
        MapConv("power_on_state", "select", mi="8.0.2030", map=BULB_MEMORY, enabled=False),
    ],
}
# after
{
    "lumi.light.aqcn02": ["Aqara", "Bulb CN", "ZNLDP12LM"],
    "spec": [
        BoolConv("light", "light", mi="4.1.85"),
        ZLumiBrightness("brightness", mi="14.1.85"),
        ZLumiColorTemp("color_temp", mi="14.2.85"),
        ZTransitionConv("transition"),
        MapConv("power_on_state", "select", mi="8.0.2030", map={0: "on", 1: "previous"}),  # config
    ],
}

Setup entity description inside converter

MathConv("occupancy_duration", "sensor", mi="2.p.3", entity={"category": "diagnostic", "enabled": False, "units": UNIT_MINUTES}),

Multiple models

• Now is OK to add multiple models to device
• For all BLE and Mesh devices it is recommended to add MiHome model (hhcc.plantmonitor.v1)

# before
"lumi.gateway.mgl03": ["Xiaomi", "Multimode Gateway", "ZNDMWG03LM"]
"lumi.gateway.mgl001": ["Xiaomi", "Multimode Gateway 2 EU", "ZNDMWG04LM"],
152: ["Xiaomi", "Flower Care", "HHCCJCY01"],
1371: ["Xiaomi", "TH Sensor 2", "LYWSD03MMC"],

# after
"lumi.gateway.mgl03": ["Xiaomi", "Multimode Gateway", "ZNDMWG03LM", "ZNDMWG02LM", "YTC4044GL"],
"lumi.gateway.mgl001": ["Xiaomi", "Multimode Gateway 2 EU", "ZNDMWG04LM", "BHR6765GL"],
152: ["Xiaomi", "Flower Care", "HHCCJCY01", "hhcc.plantmonitor.v1"],
1371: ["Xiaomi", "TH Sensor 2", "LYWSD03MMC", "miaomiaoce.sensor_ht.t2"],

BLE old converters

• MiBeacon converter deprecated (MiBeacon)
• Prebuild converters deprecated (BLETemperature...)
• mi param now support eid from MiBeacon messages
• BLEByteConv, BLEMathConv, BLEFloatConv supports unpack MiBeacon data

# before
{
    152: ["Xiaomi", "Flower Care", "HHCCJCY01"],
    "spec": [
        MiBeacon, BLETemperature, BLEMoisture, BLEConductivity, BLEIlluminance,
        Converter("battery", "sensor", enabled=None),  # no in new firmwares
    ],
}
# after
{
    152: ["Xiaomi", "Flower Care", "HHCCJCY01", "hhcc.plantmonitor.v1"],  # 4100,4103,4104,4105
    "spec": [
        BLEMathConv("temperature", "sensor", mi=4100, multiply=0.1, round=1, signed=True),  # int16
        BLEMathConv("illuminance", "sensor", mi=4103),  # uint24
        BLEByteConv("moisture", "sensor", mi=4104),  # uint8
        BLEMathConv("conductivity", "sensor", mi=4105),  # uint16
        BLEByteConv("battery", "sensor", mi=4106, entity=ENTITY_LAZY),  # uint8
    ],
}

BLE new converters

# before
{
    4611: ["Xiaomi", "TH Sensor", "XMWSDJ04MMC"],
    "spec": [
        MiBeacon, BLETemperature, BLEHumidity,
        # https://github.com/AlexxIT/XiaomiGateway3/issues/929
        MathConv("temperature", mi="3.p.1001", round=1),
        MathConv("humidity", mi="3.p.1008", round=1),
        Converter("battery", mi="2.p.1003"),
        Converter("battery", "sensor", enabled=None),  # no in new firmwares
    ],
}
# after
{
    4611: ["Xiaomi", "TH Sensor", "XMWSDJ04MMC", "miaomiaoce.sensor_ht.t6"],
    "spec": [
        # mibeacon2 spec
        BLEFloatConv("temperature", "sensor", mi=19457, round=1),  # float
        BLEFloatConv("humidity", "sensor", mi=19464, round=1),  # float
        BLEByteConv("battery", "sensor", mi=18435),  # uint8
        # miot https://github.com/AlexxIT/XiaomiGateway3/issues/929
        MathConv("temperature", mi="3.p.1001", round=1),
        MathConv("humidity", mi="3.p.1008", round=1),
        BaseConv("battery", mi="2.p.1003"),
    ],
}

MiOT spec buttons

• ButtonMIConv("button") deprecated, ConstConv("action") should be used

# before
{
    11332: ["PTX", "Mesh Double Wall Switch", "090615.switch.aksk2"],
    "spec": [
        Converter("action", "sensor", enabled=False),
        ButtonMIConv("button_1", mi="8.e.1", value=1),
        ButtonMIConv("button_1", mi="8.e.2", value=2),
    ],
}
# after
{
    11332: ["PTX", "Mesh Double Wall Switch", "090615.switch.aksk2"],
    "spec": [
        BaseConv("action", "sensor", entity=ENTITY_DISABLED),
        ConstConv("action", mi="8.e.1", value=BUTTON_SINGLE),
        ConstConv("action", mi="8.e.2", value=BUTTON_DOUBLE),
    ],
}

MiOT events

• mi="3.e.1012" changed to "3.e.1012.p.1" if event has params

# before
{
    # https://github.com/AlexxIT/XiaomiGateway3/issues/826
    7184: ["Linptech", "Wireless Button", "K11"],
    "spec": [
        MiBeacon, BLEAction, Button, BLEBattery,
        Converter("battery", mi="2.p.1003"),
        BLEEvent("action", mi="3.e.1012", map={1: SINGLE, 8: HOLD, 15: DOUBLE}),
    ],
}
# after
{
    # https://github.com/AlexxIT/XiaomiGateway3/issues/826
    7184: ["Linptech", "Wireless Button", "K11", "linp.remote.k9b01"],
    "spec": [
        # mibeacon2 spec
        BLEMapConv("action", "sensor", mi=19980, map={"01": BUTTON_SINGLE, "08": BUTTON_HOLD, "0F": BUTTON_DOUBLE}),
        BLEByteConv("battery", "sensor", mi=18435),  # uint8
        # miot spec
        MapConv("action", mi="3.e.1012.p.1", map={1: BUTTON_SINGLE, 8: BUTTON_HOLD, 15: BUTTON_DOUBLE}),
        BaseConv("battery", mi="2.p.1003"),
    ],
}

Zigbee converters

• Now linked to ZHA contants to increase readability

# before
class ZOnOffConv(ZBoolConv):
    zigbee = "on_off"
    zattr = "on_off"

    def encode(self, device: "XDevice", payload: dict, value: bool):
        cmd = zcl_on_off(device.nwk, self.ep, value)
        payload.setdefault("commands", []).extend(cmd)

# after
from zigpy.zcl.clusters.general import OnOff

class ZOnOffConv(ZBoolConv):
    cluster_id = OnOff.cluster_id
    attr_id = OnOff.AttributeDefs.on_off.id

    def encode(self, device: "XDevice", payload: dict, value: bool):
        cmd = zcl_on_off(device.nwk, self.ep, value)
        payload.setdefault("commands", []).extend(cmd)

Custom entities for models

# before
def new_entity(gateway: XGateway, device: XDevice, conv: Converter) -> XEntity:
    if conv.mi == "4.21.85":
        return AqaraE1(gateway, device, conv)
    if device.model == 14050:
        return ScdvbHAVC(gateway, device, conv)
    else:
        return XiaomiClimate(gateway, device, conv)

# after
XEntity.NEW["climate.model.lumi.airrtc.tcpecn02"] = XAqaraS2  
XEntity.NEW["climate.model.lumi.airrtc.agl001"] = XAqaraE1  
XEntity.NEW["climate.model.14050"] = XScdvbHAVC

Customize deprecated

• Moved to xiaomi_gateway3.devices

# before
homeassistant:
  customize:
    binary_sensor.0x00158d00ccddeeff_motion:
      occupancy_timeout: 180

# after
xiaomi_gateway3:
  devices:
    "0x00158d00ccddeeff":
      occupancy_timeout: 180

Stats sensors

Now can be either sensor or binary_sensor. The sensor data has also changed:

• Many useful info about uid, did, mac, brand, model, market and cloud names
• List of connected gateways
• Time from last message and info about last gateway
• Total payload from all messages

extra:
  cloud_fw: 2.1.1_0037
  cloud_name: Home Lamp 1
  did: '1234567890'
  mac: 50:ec:50:aa:bb:cc
  market_brand: Xiaomi
  market_model: MJDP09YL, yeelink.light.mbulb3
  market_name: Mesh Bulb
  rssi_54ef44ccddff: -77
  rssi_6490c1ccddff: -52
  type: mesh
gateways: 54ef44ccddff, 6490c1ccddff
last_decode: 6m8s
last_decode_gw:
  fw_ver: 1.5.4_0090
  host: 192.168.1.123
  mac: 64:90:c1:cc:dd:ff
  model: lumi.gateway.mgl03
last_encode: 6m9s
last_report:
  light: false
model: 1771
payload:
  brightness: 255.0
  color_temp: 190
  flex_switch: true
  light: false
ttl: 20m
uid: 50ec50aabbcc

Stats table

type: custom:flex-table-card
clickable: true
columns:
  - data: name
    name: Name
  - data: device.uid
    name: UID
  - data: msg_received
    name: Recv
    modify: x+''
  - data: msg_missed
    name: Miss
    modify: x+''
  - data: device.extra.seq
    name: SEQ
    modify: x+''
  - data: device.extra.rssi
    name: RSSI
    modify: x+''
  - data: device.last_decode_gw.host
    name: Gateway
    modify: x+''
  - data: state
    name: Available
  - data: last_updated
    name: Last changed
entities:
  include:
    - binary_sensor.*_ble
    - binary_sensor.*_mesh
    - binary_sensor.*_zigbee

Attributes template

# before
xiaomi_gateway3:
  attributes_template: |  
    {% if attr in ('zigbee', 'ble', 'mesh') %}  
    {{{  
      "name": device.info.name,  
      "device_fw_ver": device.fw_ver,  
      "device_model": device.model,  
      "device_market_model": device.info.model,  
      "device_manufacturer": device.info.manufacturer,  
      "integration": "gw3",  
      "gate": gateway.info.name,  
      "gateway_model": gateway.info.model,  
      "gateway_fw_ver": gateway.fw_ver  
    }}}  
    {% elif attr == 'gateway' %}  
    {{{  
      "integration": "gw3",  
      "gate": gateway.info.name,  
      "gateway_model": gateway.info.model,  
      "gateway_fw_ver": gateway.fw_ver  
    }}}  
    {% elif attr == 'battery' %}  
    {{{  
      "integration": "gw3",  
      "name": device.info.name,  
      "gate": gateway.info.name,  
      "battery": "true"  
    }}}  
    {% endif %}

# after
xiaomi_gateway3:
  attributes_template: |
    {% if attr in ('zigbee', 'ble', 'mesh') %}
    {{{
      "integration": "gw3",
      "name": device.human_name,
      "device_fw_ver": device.firmware,
      "device_model": device.model,
      "device_market_model": device.human_model,
      "device_manufacturer": device.extra.market_brand,
      "gate": gateway.human_name,
      "gateway_model": gateway.model,
      "gateway_fw_ver": gateway.firmware
    }}}
    {% elif attr == 'gateway' %}
    {{{
      "integration": "gw3",
      "gate": gateway.human_name,
      "gateway_model": gateway.human_model,
      "gateway_fw_ver": gateway.firmware
    }}}
    {% elif attr == 'battery' %}
    {{{
      "integration": "gw3",
      "name": device.human_name,
      "gate": gateway.human_name,
      "battery": "true"
    }}}
    {% endif %}

Logging

Log format also have been reworked. Now you can control basic and mqtt logs from:

• Integration config (each type of data for each gateway)
• Global integration debug logs from Home Assistant Web UI
• configuration.yaml

logger:
  default: warning
  logs:
    custom_components.xiaomi_gateway3: warning
    custom_components.xiaomi_gateway3.gate: info
    custom_components.xiaomi_gateway3.mqtt: info
    custom_components.xiaomi_gateway3.gate.192.168.1.123: debug
    custom_components.xiaomi_gateway3.mqtt.192.168.1.234: debug

Command select

Now EVERY device has command select. Options list depends to device type:

• Device info - shows full device info in the Home Assistant notification. Same info you get in the stats sensors and in device diagnostics
• Device update - request state update from device. BLE devices and many battery Zigbee devices can't be requested
• Device delete - only Zigbee device option. Sends leave signal to Zigbee device

Cloud Integration

• Now supports multiple accounts
• Now supports reloading the integration to update data
• Shows more information about devices
• Fixed name support for new Zigbee devices

Zigbee force pairing

In the default "zigbee pairing" mode you can pair supported MiHome zigbee devices and 3rd party zigbee devices. But you can't pair unsupported MiHome zigbee devices.
In this mode you can pair any zigbee device, but it won't be displayed in the MiHome even if supported.

RSSI

Now supported for BLE and Mesh. Data is logged separately for each gateway.

Mesh groups

Now Mesh groups is another device type. Groups of different models (light, cover) are also supported.

Matter

Support Matter child devices for Xiaomi Miltimode Gateway 2 (EU) on fw 1.0.7_0019.

Device triggers

Was changed from multiple types and multiple actions. To one type - action and multiple states.
If you used device triggers - them should be updated manually.

Delete device

Now deleting Home Assistant device won't delete it from Gateway. For deleting Zigbee device you can use "command select". For deleting other devices you should use MiHome.

Gateway alarm

Now has trigger for enabling and disabling.

Gateway disabling

Now gateways has disable and enable options via "command select". Them just for test, so you can check if your BLE/Mesh devices still can be controlled when some of your gateways down.