APM Agent on K8s take wrong service memory

[hramos8]

Describe the bug
When using APM agent on Kubernetes cluster, the memory reported in system.process.memory.size is incorrect.

To identify this situation whe have the suport of @tanquetav

To Reproduce
Steps to reproduce the behavior:

1. In kibana, search your service metrics:
   service.name : your-service-name and processor.event : metric
   this value isn't the service memory value, this value is too hight. Maybe this is the memory of all services on host?

Expected behavior
The correct value of the service.

[eyalkoren]

Thanks for reporting this.
Please provide some more info:

• versions of the Java agent, APM server, Elasticsearch and Kibana are you using
• some details about your k8s deployment - version, OS, containers (platform/version/base-image)
• please set debug_level to TRACE, let it run for a few minutes and upload the log
• does you cluster contain more than one pod running the same service?
• please describe or add a screenshot of your JVMs tab, and a specific JVM's metrics dashboard

[felixbarny]

We're getting the memory metrics from /proc/meminfo. See also

apm-agent-java/apm-agent-core/src/main/java/co/elastic/apm/agent/metrics/builtin/SystemMetrics.java

Lines 117 to 164 in d10d713

	if (memInfoFile.canRead()) {
	metricRegistry.addUnlessNan("system.memory.actual.free", Labels.EMPTY, new DoubleSupplier() {
	final List<WildcardMatcher> relevantLines = Arrays.asList(
	caseSensitiveMatcher("MemAvailable:*kB"),
	caseSensitiveMatcher("MemFree:*kB"),
	caseSensitiveMatcher("Buffers:*kB"),
	caseSensitiveMatcher("Cached:*kB"));
	@Override
	public double get() {
	Map<String, Long> memInfo = new HashMap<>();
	try (BufferedReader fileReader = new BufferedReader(new FileReader(memInfoFile))) {
	for (String memInfoLine = fileReader.readLine(); memInfoLine != null && !memInfoLine.isEmpty(); memInfoLine = fileReader.readLine()) {
	if (WildcardMatcher.isAnyMatch(relevantLines, memInfoLine)) {
	final String[] memInfoSplit = StringUtils.split(memInfoLine, ' ');
	memInfo.put(memInfoSplit[0], Long.parseLong(memInfoSplit[1]) * 1024);
	}
	}
	if (memInfo.containsKey("MemAvailable:")) {
	return memInfo.get("MemAvailable:");
	} else if (memInfo.containsKey("MemFree:")) {
	return memInfo.get("MemFree:") + memInfo.get("Buffers:") + memInfo.get("Cached:");
	} else {
	return Double.NaN;
	}
	} catch (Exception e) {
	return Double.NaN;
	}
	}
	});
	metricRegistry.addUnlessNan("system.memory.total", Labels.EMPTY, new DoubleSupplier() {
	@Override
	public double get() {
	try (BufferedReader fileReader = new BufferedReader(new FileReader(memInfoFile))) {
	for (String memInfoLine = fileReader.readLine(); memInfoLine != null && !memInfoLine.isEmpty(); memInfoLine = fileReader.readLine()) {
	if (memInfoLine.startsWith("MemTotal:")) {
	final String[] memInfoSplit = StringUtils.split(memInfoLine, ' ');
	return Long.parseLong(memInfoSplit[1]) * 1024;
	}
	}
	return Double.NaN;
	} catch (Exception e) {
	return Double.NaN;
	}
	}
	});
	} else {

This issue seems related: google/cadvisor#2042

[tanquetav]

Yes, the problem is related to reads memory from /proc/meminfo. Unfortunately this information is not cgroups compliant, and gives the information from host environment. The cgroup information is available on /sys/fs/cgroup/

[george@casa02 apm-agent-dotnet]$ docker run -it --rm -m 500m ubuntu:latest /bin/bash
root@0b787d6c9417:/# cat /proc/meminfo |grep Mem
MemTotal:       16269404 kB
MemFree:          282024 kB
MemAvailable:    4454260 kB
root@0b787d6c9417:/# cat /sys/fs/cgroup/memory/memory.limit_in_bytes
524288000

The -m flag tells docker to limit the container to 500Mb, but /proc/meminfo shows 16G. The correct information is available in cgroup environment.

I hope it can help to fix this issue

[eyalkoren]

@elastic/apm-agent-devs FYI, relevant to any agent that is relying on /proc/meminfo to get system memory. See also related PR.