Optimization updating memory usage via mincore syscall is expensive

[zhuangel]

When sentry update memory usage, mincore syscall will be used to check whether pages are resident in memory, this is expensive for CPU utilization, because syscall in sentry kernel will invoke a vmexit and kernel syscall.

Mincore syscall will cause about 1%~%5 CPU utilization in all Guest ring 0 and Host ring 0 CPU utilization of our Java / Go application container.

[zhuangel]

The issue could be emulated with following case, a memory allocate and free program, a meminfo read program, run this two program.

memory allocate and free program

#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <unistd.h>

#define MAX_SIZE (102410241024)
#define PAGE_SIZE 4096
#define PAGE_NUM (MAX_SIZE/PAGE_SIZE)

void main(void)
{
unsigned char *ptr[PAGE_NUM];
unsigned long idx;

while (1) {
	for (idx=0;idx<PAGE_NUM;idx++)
		ptr[idx] = (unsigned char *) malloc(4*1024);
	usleep(10);
	for (idx=0;idx<PAGE_NUM;idx++) {
		ptr[idx][idx%4096] = idx;
	}
	usleep(10);
	for (idx=0;idx<PAGE_NUM;idx++)
		free(ptr[idx]);
	usleep(10);
}

}

/proc/meminfo reading program

#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>
#include <stdio.h>

int main(void){
int fd;
size_t size, total;
unsigned char buf[4096];

while (1) {
	fd = open("/proc/meminfo",O_RDONLY);
	size = read(fd, buf, 4096);
	close(fd);
	total += size;
	usleep(10);
}
return total;

}

[amscanne]

There are likely other implications (e.g. huge pages) but I’d be curious what happens if the MapUnit in kvm is set to usermem.PageSize.

In theory, we should be able to actually mark this as used on the fault itself, and there should be no mincore required.

[zhuangel]

usermem.PageSize

Thanks @amscanne , I tried to change MapUnit into usermem.PageSize, test in my test environment, the mincore stil has the same cost of CPU.