task_mem info missing from /proc/<pid>/status inside a cgroup (and sometimes elsewhere)

[aconz2]

I'm trying to use bpftrace to log memory usage for a group of processes (any descendant process of a make -j) as each process exits. This command works (sometimes) for the whole system:

> sudo bpftrace -b -e 'tracepoint:syscalls:sys_enter_exit,tracepoint:syscalls:sys_enter_exit_group { cat("/proc/%d/status", pid); printf("---\n"); }'
Name:	firefox
Umask:	0022
State:	R (running)
...
VmPeak:	19649048 kB
VmSize:	 4717856 kB
...
---

I occasionally don't see VmPeak stats and friends in some of the processes, but most of them do. But when running in a cgroup (to easily filter the set of processes to look at), I never see VmPeak stats in the output. I've tried running bpftrace both inside and outside the cgroup with no difference.

Looking at the source, it looks like the task_struct->mm isn't available or locked or something else? I had tried using curtask->mm->hiwater_rss in the bpf program but this leads me to believe that is an invalid thing to do if the kernel doesn't want to do it in the first place.

Is sys_exit_enter too late in the process lifetime to capture this data? Is there a different tracepoint I could use for this?

[danobi]

I believe kernel tasks are mm == NULL. Not sure why a kernel thread is hitting a syscall tracepoint, though. There might be other cases I don't know about. You might have better luck asking the memory management kernel mailing list.

[fbs]

The cat doesn't happen in the kernel, the bpftrace userspace daemon does that AFTER it gets a signal from the probe to do so. So you're basically racing the kernel here which can probably (assumption here) cause a weird intermediate state until it goes full zombie.

Once its a zombie only limit data is available, until the parent waits for it

Name:	python3
State:	Z (zombie)
Tgid:	9189
Ngid:	0
Pid:	9189
PPid:	9188
TracerPid:	0
Uid:	0	0	0	0
Gid:	0	0	0	0
FDSize:	0
Groups:	0
Threads:	1
SigQ:	0/14947
SigPnd:	0000000000000000
ShdPnd:	0000000000000000
SigBlk:	0000000000000000
SigIgn:	0000000001001000
SigCgt:	0000000188000000
CapInh:	0000000000000000
CapPrm:	0000001fffffffff
CapEff:	0000001fffffffff
CapBnd:	0000001fffffffff
CapAmb:	0000000000000000
NoNewPrivs:	0
Seccomp:	0
Speculation_Store_Bypass:	vulnerable
Cpus_allowed:	f
Cpus_allowed_list:	0-3
Mems_allowed:	00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000001
Mems_allowed_list:	0
voluntary_ctxt_switches:	2
nonvoluntary_ctxt_switches:	2

I don't know a better way, except for wrapping the command in something like gnu time.

[aconz2]

Interesting, thanks for pointing out that cat runs in userspace, that makes a lot of sense!

So then would it still be valid to use curtask->mm->hiwater_rss since that is kernel side? Only reason I was going for /proc/pid/status is that I wasn't trusting the numbers I was getting and wanted to verify them -- but I guess I can do that other ways too. Will try some more and see if I get something working.

[fbs]

Yes you can manually recreate what the procfs read does, you can validate the numbers with e.g. gnu time

# /usr/bin/time --verbose bash -c 'man bash | awk "{ count++ } END { print count }"'
2713
	Command being timed: "bash -c man bash | awk "{ count++ } END { print count }""
	User time (seconds): 0.21
	System time (seconds): 0.03
	Percent of CPU this job got: 163%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.15
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 3692
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 0
	Minor (reclaiming a frame) page faults: 8555
	Voluntary context switches: 580
	Involuntary context switches: 10
	Swaps: 0
	File system inputs: 0
	File system outputs: 0
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

Or you can write your own wrapper using wait4 as they have access to rusage. Or if you want to go mad you can write your own wrapper that ptraces the processes and prints /proc on exit.

[aconz2]

I've made a bit of progress and have a repo up here in case anyone is interested