Add support for expiring retained messages.

[dustin]

For most of my environmental monitoring use cases, I want the readings
retained so I can pick them up from clients at any time, but if the
sensor (or zigbee2mqtt) fails, I want the readings to go away so I can
tell the difference between a stale reading and a missing reading.

This is easily accomplished in MQTTv5 using the "message expiry
interval" property. To add that to zigbee2mqtt, I added a 'version'
option to the mqtt section so I can specify to connect with version 5
and added a 'retention' property to devices allowing me to specify how
long items should be retained.

e.g.

mqtt:
  base_topic: site/zigbee2mqtt
  server: 'mqtt://myserver'
  user: zigbee
  version: 5
serial:
  port: /dev/ttyACM0
devices:
  '0x00358d00022308da':
    friendly_name: someroom
    retain: true
    retention: 900

[branch-switcher]

Hello @dustin. The base branch of this pull request has been updated to the dev branch. Please revisit the changes and make sure that there are no conflicts with the new base branch. Thank you for your contributions.

[dustin]

Updated for dev branch.

[dustin]

It seems the tests are very sensitive to properties, requiring they never be present even when empty.

[dustin]

Added tests for the various retention bits.

[dustin]

Oddly, the test that required mqtt v5 wasn't enough to make coverage happy on the line that configures the connection for mqtt v5. Fixed.

[Koenkk]

Thanks for the PR looks good.

One question: what happens if the user set retention for a device without setting MQTT version to 5?

[dustin]

This is an mqtt v5 property. In the protocol implementation, any properties are ignored if the connection version isn't 5 (which makes sense because there's no place to put them). If you wanted to do semantic validation on the config, that's an example of something that wouldn't do what the user probably meant. But it wouldn't break anything.

…

On Sun, Mar 8, 2020, 13:10 Koen Kanters ***@***.***> wrote: Thanks for the PR looks good. One question: what happens if the user set retention for a device without setting MQTT version to 5? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3082?email_source=notifications&email_token=AAAAN45IMQ2W4C4IXSSIYIDRGP3Q5A5CNFSM4LDSBBJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEOFABSY#issuecomment-596246731>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAAN43DZP5ZXHPCHXFJMS3RGP3Q5ANCNFSM4LDSBBJA> .

[Koenkk]

In that case please add a check here: https://github.com/Koenkk/zigbee2mqtt/blob/master/lib/util/settings.js#L291 to throw an exception when retention is set for a device and version != 5.

Please also add this option to the schema: https://github.com/Koenkk/zigbee2mqtt/blob/master/lib/util/settings.js#L120

[dustin]

I added schema definitions and validation and subsequent required testing to keep coverage up.

I'd also attempted to rebase this to upstream dev, but those tests aren't passing for me.

[dustin]

The latest test failed on a line that isn't in my code...

[dustin]

(I'm still not entirely sure how this integrates, but I added a friendly name to my test case and it made the upstream happy)

[Koenkk]

Thanks!

[Zer0x00]

What does the value of retention (here for example 900) stands for?
Seconds? Minutes? Days?

[dustin]

MQTT expiry intervals are expressed in seconds.

Minutes could be reasonable, but days wouldn't be, as the unit both lacks granularity and consistency in duration.