Expose events as Prometheus metrics (counter)

[fujin]

Motivation

I would like to use falcosidekick to expose metrics to Prometheus in the same manner as falco-exporter, which will allow me to use Grafana to visualize as well as create AlertManager rules based on the Prometheus data (e.g. statistical analysis of event count) as opposed to relaying Falco event payloads directly to AlertManager.

The reason I am looking at falcosidekick for this, is it seems to be designed as a sidecar, and also does not (currently) enforce mTLS to communicate with the Falco gRPC server to acquire event stream.

Feature

falcosidekick implements /metrics endpoint with a Prometheus Counter type for events; ideally labelled with Falco rule, priority and pod's hostname. Summary or Histogram types may be appropriate for different fleet sizes or analysis.

Alternatives

n/a

Additional context
https://github.com/falcosecurity/falco-exporter/blob/master/pkg/exporter/exporter.go#L37-L47

[Issif]

@fujin I started to work on that feature.
With prometheus package, we need to set labels that will be used, in falco-exporter, we have rule, priority and hostname. I can't add hostname as the method for gathering events is different in falcosidekick, is that a problem for your use case?

[jonny-wg2]

For me, I am using falcosidekick as the sole alerting for falco. If this component goes down and I am not aware of this, I could miss important security alerts.

Thus, it would be great to have at least an endpoint to know if it's working and properly configured, as well as, a record for previously fired events would be awesome.

[Issif]

You can call /ping to know if falcosidekick is up and running. This endpoint is already used for readiness and liveness probes.

For previous events, I always wanted to create a simple proxy, with no data retention, my guess was it's more on falco's side to bufferize.

For prom metrics, I'll add a stats for each output, with 0 or 1 depending if the output is enabled or not. A "not" may be because a misconfiguration.

[epcim]

I would like to see sidekick would expose it's metrics about its state/outputs performance, error rate on its own.

[Issif]

@epcim That's exactly what I would like to do for next release. Currenlty I'm currently moving, no time to spend on falcosidekick but I will do for sure.