feat: add metrics for total_active_file and total_inactive_file memory

[jrcichra]

The goal of this PR is to have additional cAdvisor metrics which
expose total_active_file and total_inactive_file.

Today working_set_bytes subtracts total_inactive_file in its calculation,
but there are situations where exposing these metrics directly is valuable.

For example, two containers sharing files in an emptyDir increases total_active_file over time. This is not tracked in the working_set memory.

Exposing total_active_file and total_inactive_file to the user
allows them to subtract out total_active_file or total_inactive_file
if they so choose in their alerts.

In the case of prometheus with a thanos sidecar, working_set can give
a false sense of high memory usage. The kernel counts thanos reading prometheus written files as "active_file" memory.
In that situation, a user may want to exclude active_file from their ContainerLowOnMemory alert.

Relates to: kubernetes/kubernetes#43916

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[k8s-ci-robot]

Hi @jrcichra. Thanks for your PR.

I'm waiting for a google member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[iwankgb]

/ok-to-test

[jrcichra]

CLA check is passing now.

[jrcichra]

@iwankgb Thanks for the ok-to-test. This is ready for review.

[SuperQ]

Hmm, this appears to be doing a calculation in cAdvisor. In Prometheus we prefer to expose raw metrics so that the user can do whatever math they want in PromQL.

IMO we should get rid of "working set" and simply expose all the byte gauges like total_active_file.

[jrcichra]

Hmm, this appears to be doing a calculation in cAdvisor. In Prometheus we prefer to expose raw metrics so that the user can do whatever math they want in PromQL.

IMO we should get rid of "working set" and simply expose all the byte gauges like total_active_file.

That would be good too. For the use case I'm describing, the solution would be to have a combination of three cgroup metrics:

1. memory.usage_in_bytes
2. total_active_file
3. total_inactive_file
4. ...all other cgroup raw memory metrics...

Then we could build a prometheus query + alert for memory_usage_in_bytes - total_active_file - total_inactive_file.

Then perhaps we do a similar calculation for kubelet calculating memory pressure instead of working_set. That would be done here:

https://github.com/kubernetes/kubernetes/blob/a309fadbac3339bc8db9ae0a928a33b8e81ef10f/pkg/kubelet/eviction/memory_threshold_notifier.go#L92

[jrcichra]

@SuperQ I've updated the PR to add raw metrics for total_active_file and total_inactive_file so the users (alert authors, kubelet, etc) can make their own memory calculations (e.g container_memory_usage_bytes - container_memory_total_active_file_bytes - container_memory_total_inactive_file_bytes).