Add monitoring information about functions to Functionbeat

[kvch]

This PR adds a new counter which returns the number of triggered functions of Functionbeat.

Example snippet of the telemetry message containing info about function:

{
  "state": {
    "functionbeat": {
        "functions": {
             "count": 1,
        }
    }
}

Follow-up work is required on Kibana side to process the reported information correctly to get telemetry data.

[urso]

What is the purpose of triggerCount or eventCount? Telemetry is less about monitoring, we just want to know which features are used.

[kvch]

I was interested to see how many triggers users configure. I can remove both counters if we don't want those to be reported at all.

[urso]

libbeat declares that 'stat' registry for telemtry data. If it is used we will have the monitoring backend report metrics and telemetry information every now and so often.

[kvch]

I've only found a registry named "state". Is that what you mean?
https://github.com/elastic/beats/blob/master/libbeat/cmd/instance/beat.go#L338

[ycombinator]

I think @kvch is already using the stats registry here, by virtue of using monitoring.Default. See

beats/libbeat/monitoring/monitoring.go

Lines 50 to 55 in f35ffdc

	// Default is the global default metrics registry provided by the monitoring package.
	var Default = NewRegistry()
	func init() {
	GetNamespace("stats").SetRegistry(Default)
	}

.

[urso]

I mean "state". The 'stats' registry is for metrics, the 'state' registry is for telemetry. monitoring.Default should not be used here.

[urso]

One tricky problem might be, that we have no control when telemetry data are reported, or how often. If we remove the trigger counts we still can collect information like: functionbeat being used, provider in use, trigger types configured, and configured output type. Once a function is installed with a provider, it is some rather static information.

Instead of relying on some monitoring background task that might be reporting information in time or not (the beat is put to halt once all events have been ACKed), we might want to implement slightly different strategy. e.g. On startup there is a license check. The license is cached once queried. The same time we query the license we could also attempt to send the telemetry info. Functionbeat would have to wait for events being published + telemetry being send. If there was no telemetry update for N minutes we could send the same info again.

@ph @ycombinator WDYT? I also wonder about feasbility (would we have to implement our own reporting or can we reuse code from the reporter as is?).

[ycombinator]

@urso I like your proposal about sending the monitoring data as soon as the license is queried. We can do that once and then continue with the periodic sending thereafter. Obviously this will require some refactoring in the libbeat monitoring code first. Specifically, it looks like currently the reporter's constructor is responsible for spawning a goroutine to periodically send monitoring data. Instead we might want to expose a Start method on the Reporter interface and require the reporter's client to call it after constructing the reporter. Then we can instantiate the reporter where it is being done today but call the Start method from the license check.

One clarifying question: I assume you'd want to make this change (start monitoring reporter right after license check) for all beats, not just Functionbeat?

[urso]

This sounds like a rather big change we should do in another PR.

Telemetry should not be reported by an independent process. If all events are ACKed before Telemetry did finish, then the information is lost.

One clarifying question: I assume you'd want to make this change (start monitoring reporter right after license check) for all beats, not just Functionbeat?

It could be for all Beats. But because Functionbeat is so different from the other Beats it is a requirement for Functionbeat. Reporting must not necesarrily be tied to the License check, but should happen if the function starts up first. Maybe again if X the function is invoked with a new Batch of events and the last report is X minutes in the past. => Although reporting is done in the background, the if and when to trigger it should be under control of Functionbeat, not libbeat.

[urso]

do we want the mock to store invocations, so we can validate them in tests?

[ycombinator]

@kvch I have a draft PR up for the Kibana-side changes here: elastic/kibana#54267. Once you've updated the monitoring snippet for Functionbeat telemetry with the final one in this PR's description above, I will proceed with making the corresponding changes in the Kibana PR.

[urso]

Are we missing a file? make check is failing with symbol errors.

[kvch]

@urso No, I just forgot to add : before =... :D

[urso]

No need to a lock here. (*monitoring.Int).Inc() is atomic.

[urso]

All in all, LGTM. Please create a follow up issue allowing functionbeat to control when the telemetry reporter should send info.