[RFC] Standard radio settings API

[puddly]

I propose changing the radio library API to separate high-level network operations from the low-level radio details:

• connect(): connects to the serial port and verifies that it's compatible with the radio library. Does not start the network. This would also allow probe to be moved into zigpy (or removed entirely), since it's basically connect() in a try: block.
• disconnect(): shut down the radio (if it's running) and release the serial port. Not all radios have watchdog timers and thus will continue to run after the serial port loses connection.
• load_network_info(load_devices=False): populates self.state.network_information and self.state.node_information. The load_devices keyword argument allows for "slow" operations that provide supplemental network information (like scanning TCLK tables) to be done only when necessary.
• write_network_info(network_info, node_info): Writes the provided network and node information to the radio hardware. Not all available information can be used by every radio so warnings should be logged where appropriate when information is ignored.
• start_network(): start the network with settings currently stored in the stick. Zigpy's startup would then essentially be something like this:

  await self.connect()
  
  try:
      network_info, node_info = await self.load_network_info(False)
  except NetworkNotFormed:
      if auto_form:
          await self.form_network()
      else:
          raise RuntimeError("Network is not formed")
  
  if info != info_in_db:
      if strict:
          raise RuntimeError("Network settings don't match what's in the database")
      else:
          LOGGER.warning("Network settings don't match what's in the database")
  
  await self.start_network()
  await self.permit(0)

This provides a standard interface for downstream applications (e.g. ZHA and zigpy-cli) to manage network settings independent of the radio hardware. It also removes the need to separately maintain identical network formation code in each radio library.

Below is an example network formation API for zigpy that maintains compatibility with the current YAML configuration options:

# Implemented by the radio library
def stack_specific_formation_settings(self):
    return {"zstack": {"tclk_seed": os.urandom(16).hex()}}

async def form_network(self, *, start_after_formation=False):
    # First, make the settings consistent and randomly generate missing values
    channel = self.config[conf.CONF_NWK][conf.CONF_NWK_CHANNEL]
    channels = self.config[conf.CONF_NWK][conf.CONF_NWK_CHANNELS]
    pan_id = self.config[conf.CONF_NWK][conf.CONF_NWK_PAN_ID]
    extended_pan_id = self.config[conf.CONF_NWK][conf.CONF_NWK_EXTENDED_PAN_ID]
    network_key = self.config[conf.CONF_NWK][conf.CONF_NWK_KEY]

    if pan_id is None:
        pan_id = random.SystemRandom().randint(0x0001, 0xFFFE + 1)

    if extended_pan_id is None:
        # TODO: exclude `FF:FF:FF:FF:FF:FF:FF:FF` and possibly a few more reserved EPIDs
        extended_pan_id = ExtendedPanId(os.urandom(8))

    if network_key is None:
        network_key = t.KeyData(os.urandom(16))

    # Override `channels` with a single channel if one is explicitly set
    if channel is not None:
        channels = t.Channels.from_channel_list([channel])

    network_info = zigpy.state.NetworkInformation(
        extended_pan_id=extended_pan_id,
        pan_id=pan_id,
        nwk_update_id=self.config[conf.CONF_NWK][conf.CONF_NWK_UPDATE_ID],
        nwk_manager_id=0x0000,
        channel=channel,
        channel_mask=channels,
        security_level=5,
        network_key=zigpy.state.Key(
            key=network_key,
            tx_counter=0,
            rx_counter=0,
            seq=self.config[conf.CONF_NWK][conf.CONF_NWK_KEY_SEQ],
            partner_ieee=None,
        ),
        tc_link_key=zigpy.state.Key(
            key=self.config[conf.CONF_NWK][conf.CONF_NWK_KEY],
            tx_counter=0,
            rx_counter=0,
            seq=0,
            partner_ieee=self.config[conf.CONF_NWK][conf.CONF_NWK_TC_ADDRESS],
        ),
        children=[],
        key_table=[],
        nwk_addresses={},
        stack_specific=self.stack_specific_formation_settings(),
    )

    node_info = zigpy.state.NodeInfo(
        nwk=0x0000,
        ieee=None,
        logical_type=zdo_t.LogicalType.Coordinator,
    )

    await self.write_network_info(network_info=network_info, node_info=node_info)

Thoughts?

[Hedda]

The API is provisional so if you think it needs to be changed, this would be the time to make those changes relating to network formation (or maybe adding set_tx_power and set_led as standard methods?).

@puddly Could maybe set_channel be made a standard method to allow changing Zigbee channels too? As related to -> #908

It would be great if implementations like ZHA could allow UI setting to change Zigbee channel independent of radio library used.

That is, I now understand that most Zigbee Coordinators actually support changing Zigbee channel and at least properly made Zigbee devices will update and automatically change to the new channel without the need to be manually re-paired. Some devices might however need up to 24-hours or more before changing over and most battery-powered Zigbee end-devices will need to be woken up (like in a state change on a window/door sensor) before they will communicate and then the detect channel change. The only clear limitation is that ZZL (Zigbee Light Link) devices just support Zigbee channels 15, 20, or 25 (and maybe also Zigbee channel 11 depending on exactly which manufacturers Zigbee stack and age of hardware/firmware that the device is based on).

As well wondering if an NVRAM reset of Zigbee Coordinator (erase all memory) would be considered too low-level and if so some other type of high-level reset that would remove the network, etc. to allow users to start over could be implemented?

Regardless, having set_tx_power and set_led as standard methods sounds like a great idea.

PS: I by the way also requested a standard method for getting devices LQI with with a unified scale and format here -> #886

[MattWestb]

ZLL is using the the primary channels 11, 15, 20, and 25 for commissioning =joining one new network. ZB3 devices is doing the same but if the first run is not OK then they is doing one fast scan on all other channels. The largest problem is some device have very bad Zigbee stack that cant using all the primary channels.

All thins in the end is one user thing and the system shall not limiting anything like that then and the best is writing more in the wiki about bad behaving devices.

If i getting time i shall testing the only IKEA ZLL device (the old motion sensor after the old CWS1 was getting ZB3 in the last update) and sniffing is its broadcasting on also the primary channels then i have time.

[puddly]

Could maybe set_channel be made a standard method to allow changing Zigbee channels too?

You can already change the network channel with write_network_info. Changing the coordinator's channel alone won't do very much, though.

If this is going to be done, it'll have to be with zdo.Mgmt_NWK_Update_req and will need to be implemented within zigpy, not the radio libraries. It's not a simple task. The only way I can see it working is by using the network topology to traverse the existing devices depth-first and migrate them one-by-one with unicast requests.

PS: I by the way also requested a standard method for getting devices LQI with with a unified scale and format here -> #886

I guess it is possible to rescale the LQI measurements reported by the coordinator for last-hop packets, but what about every other router on the network? They self-report LQI during scans and we can't tell what firmware is running on them.

[MattWestb]

@puddly If i remember right shall the new network key with higher sequences number being broadcaster to all devices (not all routers). Then you can waiting and sending it more times or doing unicast to problem devices. If having pull control working its possible sending it safe to SEDs with unicast.
Then later sending / broadcasting "network key rolling number X" and all devices that is getting the commands is doing the rolling and other devices that has not getting the command is later trying connecting and is finding the network key is rolled and start using the new key.

Problem is no Zigbee 3 devices with old buggy Zigbee stacks or with not implanted function and Xiaomi devices . . .
One good Silabs paper is AN1233: Zigbee Security 3.4 Network Key Updates.

In the end every device that is moving with then rolling the key is one lesser then forming on new network and must repairing all devices !!

Rolling network key is high on my priority list !!