Sync peer check should have a configurable grace period

[afreiberger]

We have an environment running chrony via the ntp charm on bionic that regularly drops and re-selects it's sync peer on random hosts which lasts from 1-30 minutes with system default chrony profiles using the default ubuntu pools and an additional local source IP.

We get alerts for ntpmon showing "no sync peer" but it typically clears within 30 minutes or an hour, if not shorter.

It would be very helpful for alerting noise reduction to have a configurable length of time in which a system can take to find a sync peer among available sources.

[paulgear]

Hi Drew, somehow I only just saw this - not sure what happened there. An appropriately-configured chrony shouldn't spend much time at all selecting a new sync peer if one goes away (I would have expected a few seconds more than a few minutes). I'd be interested to see some example logs which show that behaviour.

[paulgear]

I've discussed this with a couple other folks familiar with this problem (or similar ones), and I'm not prepared to add statefulness to the Nagios check at the moment. The recommended solution for this is to use the telegraf version of NTPmon, scrape the telegraf agent with prometheus, and use prometheus alerter to define the desired thresholds and grace periods.