Skip to content

v1.9.1

Latest

Choose a tag to compare

@sylvesterdamgaard sylvesterdamgaard released this 24 Sep 15:23

cbox-init 3.7.1, which stops alerting on processes you removed on purpose, and a Grafana dashboard that queries the labels cbox-init actually emits.

cbox-init 3.7.1

cbox-init left stale series behind whenever a process or instance went away on purpose. A live scale-down, lowering scale through a reload or API edit, removing a process, or turning one into a scheduled task all left cbox_init_process_up at 0 for good. So process_up == 0 alerts fired on something the operator removed on purpose.

Scheduled tasks also kept their old desired_scale and last health status, which set off scale-drift and health alerts. Each of these paths now drops the series. No metric or label names changed, so existing dashboards and alerts keep working.

The cbox-init dashboard shows data again

observability/grafana-cbox-init-dashboard.json, observability/README.md and the observability docs queried labels cbox-init does not emit, so the status, queue, health and CPU panels were empty.

  • The process, scale and health metrics label the process as name, not process. The health check type is type, not check_type. Only the resource metrics (cbox_init_process_cpu_percent, _memory_bytes, ...) use process.
  • cbox_init_process_cpu_seconds_total never existed. The CPU panel now uses cbox_init_process_cpu_percent. The memory panel shows RSS only, instead of RSS and VMS drawn under one legend.
  • The cbox_init_scheduled_task_* metrics, their dashboard row and their alert rule were never real. cbox-init has no Prometheus metrics for scheduled tasks; the docs now point to its schedule API.
  • The memory and CPU panels need CBOX_INIT_GLOBAL_RESOURCE_METRICS_ENABLED=true. cbox-init ships with resource metrics off.
  • Under Prometheus' default honor_labels: false, cbox-init's own instance label is stored as exported_instance. The docs now say so.

If you imported the dashboard or copied the alert rules, re-import the JSON and update any process= selectors on the process and health metrics to name=.