pcp2zabbix: enable emitting "low-level-discovery" metrics for indoms

[fche]

When pcp2zabbix is told to pass along metrics with an indom, it'll transcribe the instance name after the metric name, like foo.bar[baz]. For instances where the domain is variable, it's not practical on the zabbix side to configure it for all possible values of baz, so such metrics get tossed away. Instead, so-called "low-level discovery" zabbix configuration is the method of choice.

The idea is that, for all indoms pcp2zabbix encounters, a dummy metric is also sent to zabbix, with a stylized JSON string value such as:

[ { "{#INDOMxyz}":"value1" }, {"#INDOMxyz}":"value2"} ]

I believe a single such dummy metric can enumerate all the indoms (with their distinct #INDOMxyz names). Then the {#INDOMxyz} values become macros by which zabbix users can automatically create metric/trigger/graph families for related metrics of the same indom.

https://www.zabbix.com/documentation/3.4/manual/discovery/low_level_discovery

[fche]

#474

[fche]

There is a problem with this implementation. It sense the LLD metrics only 'as required', namely upon startup, or when the instance domain of a particular metric changes. It turns out that this is not enough. If pcp2zabbix is already running (and thus transmitted the LLD set once), and only after this point does zabbix become configured to monitor those discovery keys, then zabbix will not get them. Or at least it won't get them until the next time the indom changes.

It seems as though we need to send the discovery information more regularly, regardless of indom change status. But sending them every report would not be great either (since the indom enumerations are large, larger than the underlying data). But maybe that's the simplest thing to do.

[fche]

On second thought, one could use pcp2zabbix's -s or -T to instruct it to commit seppuku periodically; a pmmgr-driven restart would be instant, automatic, and ripe with lld goodness. Probably good enough.