Expose limits imposed by cgroups via port library APIs

[ashu-mehra]

Runtimes like JVM rely on resource limits like memory and cpus to adjust their internal parameters which includes heap size, GC helper threads, JIT helper threads etc.
Cgroups in Linux provide a mechanism to impose limits on such resources. Therefore, when running as part of a cgroup, the runtime should adjust its internal parameters based on the resource constraints imposed by the cgroup.
Currently the information about resource constraints imposed by cgroup is not provided by the port library in any way. If a runtime wants to take into consideration the limits imposed by the cgroup, it would have to figure this out on its own.
It would be useful for the runtimes if port library can provide APIs that would enable them to query the resource constraints imposed by the cgroup.
The proposal is to add mechanism for following tasks:

1. Add capability to read /proc/<pid>/cgroup file.
   Using this file, the process can figure out the cgroup it belongs to for a given subsystem.
2. Expose APIs for providing memory limits and CPU limits imposed by the cgroup. For start only these two limits would be exposed. More can be added later as and when required.
3. Now, the above mechanism would work when running on bare metal or a VM, but would break when running in a docker container. This is because in a docker container the /proc/<pid>/cgroup gives the cgroup of the process on the host, and not the one in the container. AFAIK when running in container, the cgroup would always be the root cgroup. Therefore, we also need to detect whether the process is running in a docker container or not, and use correct cgroup to get the resource limits.

As an example, if I log in to a docker container, and cat /proc/1/cgroup, I get following contents:

$ docker exec -it dfee854963d4575d263785ddee0f3c398355b354e6608c5777a3a2e40fef3612 /bin/bash
root@dfee854963d4:/# cat /proc/1/cgroup
11:pids:/docker/dfee854963d4575d263785ddee0f3c398355b354e6608c5777a3a2e40fef3612
10:perf_event:/docker/dfee854963d4575d263785ddee0f3c398355b354e6608c5777a3a2e40fef3612
9:freezer:/docker/dfee854963d4575d263785ddee0f3c398355b354e6608c5777a3a2e40fef3612
8:cpuset:/docker/dfee854963d4575d263785ddee0f3c398355b354e6608c5777a3a2e40fef3612
7:blkio:/docker/dfee854963d4575d263785ddee0f3c398355b354e6608c5777a3a2e40fef3612
6:memory:/docker/dfee854963d4575d263785ddee0f3c398355b354e6608c5777a3a2e40fef3612
5:hugetlb:/docker/dfee854963d4575d263785ddee0f3c398355b354e6608c5777a3a2e40fef3612
4:devices:/docker/dfee854963d4575d263785ddee0f3c398355b354e6608c5777a3a2e40fef3612
3:net_prio,net_cls:/docker/dfee854963d4575d263785ddee0f3c398355b354e6608c5777a3a2e40fef3612
2:cpuacct,cpu:/docker/dfee854963d4575d263785ddee0f3c398355b354e6608c5777a3a2e40fef3612
1:name=systemd:/docker/dfee854963d4575d263785ddee0f3c398355b354e6608c5777a3a2e40fef3612

which shows the process is part of /docker/dfee854963d4575d263785ddee0f3c398355b354e6608c5777a3a2e40fef3612 cgroup in memory subsystem. However, if I go to /sys/fs/cgroup/memory I don't see any such cgroup, but the same cgroup is present in the host. Probably the /sys/fs/cgroup/memory/docker/<id> cgroup on the host is mapped as the root cgroup inside the container.
In short, /proc/<pid>/cgroup cannot be used for getting the cgroup when running in a docker container.

Breaking down this work into following lists of tasks:

• Add support for reading files for detecting resource limits for cgroup v1
• Determine if the we are running in container or not
• Add port library API for exposing memory limit imposed by cgroup
• Add port library API for exposing cpu limit imposed by cgroup
• Detect if the system is using cgroup v1 or cgroup v2
• Add support for reading files for detecting resource limits for cgroup v2

[ashu-mehra]

fyi @charliegracie @DanHeidinga

[DanHeidinga]

Thanks @ashu-mehra for starting on this - it's an important feature and definitely needed in OMR!

The API for this should make it easy to make existing code cgroup-aware without having to explicitly code a cgroup vs non-cgroup path in all code requesting i.e. sys max memory.

A very rough sketch of what I'm thinking would be:

/* Determine if running in a cgroup namespace */
BOOLEAN omr_sysinfo_cgroup_in_cgroup()

/* Indicate to port library that limits should be taken from the cgroup, not from the machine.
 * @return previous value of this setting
 */
BOOLEAN omr_sysinfo_cgroup_enable_cgroup_limits()

/* Should the port library respect cgroups in its normal API calls */
BOOLEAN omr_sysinfo_cgroup_limits_enabled()

/* Special accessors to get the cgroup-related limit for YYYY.
 * ie: long omr_sysinfo_cgroup_get_max_memory
 */
XXXX omr_sysinfo_cgroup_get_max_YYYYY()

And existing apis would be updated to be cgroup aware:

XXXX omr_sysinfo_get_max_memory() {
   if (omr_sysinfo_cgroup_limits_enabled()) {
      return omr_sysinfo_get_max_memory();
   } else {
      ....
      /* existing code path */
      .....
   }

This makes it easy for all consuming code to be aware of cgroup limits without having to be modified. Does this seem like a reasonable (though very rough) api sketch?

[ashu-mehra]

@DanHeidinga Thanks for jotting down the structure. I have exactly similar usage pattern in mind and I agree existing APIs should be updated to use cgroups when available.

Couple of points:

1. Do we need to make omr_sysinfo_cgroup_limits_enabled() an API, or just an internal helper function?
2. Same for omr_sysinfo_cgroup_in_cgroup(). It is checking if we are in cgroup or not, which would be needed internally by omr_sysinfo_cgroup_enable_cgroup_limits(). So would it be better to just have it as helper function?

[DanHeidinga]

Do we need to make omr_sysinfo_cgroup_limits_enabled() an API, or just an internal helper function?

My gut says to make it an API so that reporting / monitoring tools such as a javacore file can report which mode the port library has been put in.

Same for omr_sysinfo_cgroup_in_cgroup().

For the same kinds of reasons, we want this to be query-able by the runtime using the port library.

And to be clear, I'm not sold on the method names proposed in the rough sketch. If you think of something better, go with it - these were off the cuff examples.

[ashu-mehra]

As there are quite a few things to be done to fully support cgroups, I have tried to break it down into smaller items and created a task list in the first comment.
I will try to create pull requests targeting these tasks in the list.

If I have missed anything, feel free to add tasks in the list in the first comment.

[ashu-mehra]

@DanHeidinga @charliegracie @dinogun

For the task of determining if we are running inside a container or not, I have following thoughts:

Since there are multiple container types, not just Docker (Rkt is another going around), having a port library API omrsysinfo_get_container_type() would be relevant.
Detecting container type is not straight forward and not well documented. We would have to use all sorts of mechanisms to figure that out.
We can propose to set an env variable "CONTAINER_TYPE" to be set by the user which would indicate the container in which the runtime is running.
For Docker, mechanism described in this stackeoverflow thread can be used:
The most reliable way is to check /proc/1/cgroup. It will tell you the control groups of the init process, and when you are not in a container, that will be / for all hierarchies. When you are inside a container, you will see the name of the anchor point; which, with LXC/Docker containers, will be something like /lxc/<containerid> or /docker/<containerid> respectively.

Based on above thoughts I have following code structure. Let me know if this makes sense.

enum ContainerType {
	DOCKER,
	RKT,
        ...,
	UNSUPPORTED,
	NONE
}
ContainerType omrsysinfo_get_container_type()

omrsysinfo_get_container_type() {
	type = getenv("CONTAINER_TYPE");
	if (NULL != type) {
		if (type == "docker" )
			return DOCKER;
		else if (type == "rkt")
			return RKT;
		else 
			return UNSUPPORTED;
	} else {
		if (isDockerContainer()) {
			return DOCKER;
		} else if (isRktContainer()) {
			return RKT;
		} else {
			return NONE;
		}
	}
}

isDockerContainer() {
	cgroup = getCgroupForPID(1)
	if (cgroup == "/") {
		return FALSE;
	} else if (cgroup.contains("docker")) {
		return TRUE;
	}
}

isRktContainer() {
      ...
}

[dinogun]

+1.
This is as per the discussion that we had earlier today and makes sense to me. This is also how we have designed the isVirtual API as well.