controller: Add metrics listener with initial event count metric as demo

[nickbp]

Adds endpoint for getting metrics from the controller. Defaults to port 8900 but can be configured or disabled via optional --metrics-port arg. Resolves the TODO METRIC as an example to start with.

Need to check if there's anything needed to hook up these metrics into the system tenant...

$ curl -v http://localhost:8900/metrics

# HELP process_cpu_user_seconds_total Total user CPU time spent in seconds.
# TYPE process_cpu_user_seconds_total counter
process_cpu_user_seconds_total 1.224548

# HELP process_cpu_system_seconds_total Total system CPU time spent in seconds.
# TYPE process_cpu_system_seconds_total counter
process_cpu_system_seconds_total 0.106098

# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 1.330646

# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1629868507

[...]

# HELP nodejs_eventloop_lag_seconds Lag of event loop in seconds.
# TYPE nodejs_eventloop_lag_seconds gauge
nodejs_eventloop_lag_seconds 0

# HELP nodejs_eventloop_lag_min_seconds The minimum recorded event loop delay.
# TYPE nodejs_eventloop_lag_min_seconds gauge
nodejs_eventloop_lag_min_seconds 0.003028992

# HELP nodejs_eventloop_lag_max_seconds The maximum recorded event loop delay.
# TYPE nodejs_eventloop_lag_max_seconds gauge
nodejs_eventloop_lag_max_seconds 0.088735743

[...]

# HELP informer_events Incremented when a new k8s/graphql event is received
# TYPE informer_events counter
informer_events{type="FETCH_K8S_NAMESPACES_SUCCESS"} 1
informer_events{type="FETCH_K8S_V1ALERTMANAGERS_SUCCESS"} 1
informer_events{type="FETCH_K8S_V1PODMONITORS_SUCCESS"} 1
informer_events{type="FETCH_K8S_V1PROMETHEUSRULES_SUCCESS"} 1
informer_events{type="FETCH_K8S_V1PROMETHEUSS_SUCCESS"} 1
[...]
informer_events{type="ON_ADDED_K8S_CONFIGMAPS_REQUEST"} 50
informer_events{type="FETCH_K8S_CRDS_SUCCESS"} 1
informer_events{type="ON_ADDED_K8S_CRDS"} 16
informer_events{type="ON_UPDATED_K8S_STATEFUL_SETS"} 6
informer_events{type="ON_UPDATED_K8S_NODES"} 2

Signed-off-by: Nick Parker nick@opstrace.com

[jgehrcke]

👍

[jgehrcke]

Nice! We may want to add an HTTP server terminator though to make sure that we shutdown the NodeJS HTTP server within predictable time boundaries.

See for example

opstrace/test/test-remote/looker/src/prommetrics.ts

Line 180 in cb7764a

const httpServerTerminator = createHttpTerminator({

[jgehrcke]

🚀 let's get this in!