[libbeat] beat.cgroup.memory.mem.usage.bytes should be a gauge in metrics

[andrewkroh]

Periodic metrics logged by Beats can include a negative value:

2022-01-18T18:06:22.792Z INFO [monitoring] log/log.go:184 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cgroup":{"cpuacct":{"total":{"ns":193010167}},"memory":{"mem":{"usage":{"bytes":-327680}}}},"cpu":{"system":{"ticks":2630,"time":{"ms":64}},"total":{"ticks":9700,"time":{"ms":179},"value":9700},"user":{"ticks":7070,"time":{"ms":115}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":17},"info":{"ephemeral_id":"8d4d7833-addd-47c3-90e1-89e88ed8da2c","uptime":{"ms":2610049},"version":"7.16.3"},"memstats":{"gc_next":22981312,"memory_alloc":11671312,"memory_total":1348970528,"rss":138485760},"runtime":{"goroutines":99}},"filebeat":{"events":{"active":6,"added":1146,"done":1140},"harvester":{"open_files":5,"running":5}},"libbeat":{"config":{"module":{"running":1},"scans":6},"output":{"events":{"acked":1098,"active":0,"batches":30,"total":1098},"read":{"bytes":26068},"write":{"bytes":1690273}},"pipeline":{"clients":6,"events":{"active":20,"filtered":42,"published":1104,"total":1146},"queue":{"acked":1098}}},"registrar":{"states":{"current":6,"update":1140},"writes":{"success":51,"total":51}},"system":{"load":{"1":0.03,"15":0.13,"5":0.07,"norm":{"1":0.0019,"15":0.0081,"5":0.0044}}}}}}

"monitoring.metrics.beat.cgroup.memory.mem.usage.bytes": -327680,

These metrics work by diffing the previous value (from -30s ago) to the current value. So if the value is a gauge (goes up and down) rather than a counter you get this problem. There is a list of metric names that are gauges in the code at

beats/libbeat/monitoring/report/log/log.go

Lines 39 to 66 in 0c20dbb

	var gauges = map[string]bool{
	"libbeat.output.events.active": true,
	"libbeat.pipeline.events.active": true,
	"libbeat.pipeline.clients": true,
	"libbeat.config.module.running": true,
	"registrar.states.current": true,
	"filebeat.harvester.running": true,
	"filebeat.harvester.open_files": true,
	"beat.memstats.memory_total": true,
	"beat.memstats.memory_alloc": true,
	"beat.memstats.rss": true,
	"beat.memstats.gc_next": true,
	"beat.info.uptime.ms": true,
	"beat.cpu.user.ticks": true,
	"beat.cpu.system.ticks": true,
	"beat.cpu.total.value": true,
	"beat.cpu.total.ticks": true,
	"beat.handles.open": true,
	"beat.handles.limit.hard": true,
	"beat.handles.limit.soft": true,
	"beat.runtime.goroutines": true,
	"system.load.1": true,
	"system.load.5": true,
	"system.load.15": true,
	"system.load.norm.1": true,
	"system.load.norm.5": true,
	"system.load.norm.15": true,
	}

. That metric, beat.cgroup.memory.mem.usage.bytes is not listed there.

beat.cgroup.memory.mem.usage.bytes needs added to that list.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[boranx]

Hi 👋
if it's a one-liner change, I can propose a PR
However, could you shed some light on how I can test this behavior?
and which exact CHANGELOG I should use to explain the change

[cmacknz]

It should be a one liner or close to it. For the changelog entry you can use something like the following, which you'll have to update with the PR number after you create it.

The beat.cgroup.memory.mem.usage.bytes metric is now a gauge {pull}INSERT_YOUR_PR_NUMBER[INSERT_YOUR_PR_NUMBER]`