server/status: read current process's cgroup to determine available memory

[ajwerner]

Prior to this commit we would consult the system memory and the root cgroup
to determine the available memory. This is problematic because almost always
when systems use cgroups they use a hierarchy to create limits for specific
sets of processes. After this change we'll now determine the deepest child
hierarchy which contains this process and examine all of the relevant
limits to determine the memory limit.

Release note (bug fix): On linux machines we now respect the available memory
limit as dictated by the cgroup limits which apply to the cockroach process.

[cockroach-teamcity]

This change is 

[derekwaynecarr]

to save future headaches, is it possible to support reading both cgroup v1 file or v2 file?

in Kubernetes, we are starting to work toward cgroup v2 enablement so adding support for memory.max in addition to what is checked here would help prevent future revisions.

[ajwerner]

@bdarnell I didn't know about #32067 when I typed this. Do we prefer that patch? It adds a dependency on runc just to read the /proc/<pid>/cgroup file though that dependency makes for cleaner code. I think that patch still has a problem with regards to limits in a parent of the hierarchy being smaller than the limit in the child.

Relates to #31750.

[bdarnell]

Do we prefer that patch? It adds a dependency on runc just to read the /proc//cgroup file though that dependency makes for cleaner code

My thinking is that if it's just the small amount of code in this PR, I'd rather parse the file ourselves and not have the dependency. But given the "TODO: support v2" comments, it seems useful to take on a dependency and get (hopefully) more complete and future-proof support for it. (runc does seem like a rather large dependency for this purpose though).

[ajwerner]

But given the "TODO: support v2" comments, it seems useful to take on a dependency and get (hopefully) more complete and future-proof support for it.

I buy that. I went fishing around for a better library than the one in runc and didn't come up with too much. https://github.com/containerd/cgroups seems like it could be okay though on first glance the API seems pretty heavy weight. The logic required to support v2 is rather small. It's just detecting that there's a v2 hierarchy when parsing the /proc/<pid>/cgroup file which would be a row with a 0::$PATH line and then reading the memory.max file instead of the memory.limit_in_bytes file. The bigger win from adopting the dependency is that it parses the mount points to find out where the cgroups are mounted (also a TODO).

If I read your comment correctly, I should take a stab at adopting a dependency to achieve the goals and only if I have a bad time should we revisit this more manual approach.

[bdarnell]

Yeah, I'd try with the dependency first.

[ajwerner]

Yeah, I'd try with the dependency first.

I ran in to trouble with the dependencies. I decided to just support v2. I wrote tests and tried it out in a variety of settings on VMs. It seems to work.

@vladdy would you be willing to give this a review?

[ajwerner]

I suspect the code needs more of a gate on the OS, perhaps by consulting runtime.GOOS at runtime or by using compilation directives. Otherwise it is RFAL.

[vladdy]

Reviewed 4 of 4 files at r1.
Reviewable status: complete! 0 of 0 LGTMs obtained

[ajwerner]

TFTR!

bors r=vladdy

[craig]

Build succeeded

• GitHub CI (Cockroach)