Failed to find a process in proc_info_map

[yanivagman]

Description

In environments with a big amount of CPUs (96 CPUs) we got the following warnings:

"{"level":"warn","ts":1695678822.5690784,"msg":"Failed to find a map element","id":2,"type":"BPF_LOG_ID_MAP_LOOKUP_ELEM","ret":0,"cpu":82,"file":"./pkg/ebpf/c/tracee.bpf.c","line":603,"count":1}

"{"level":"warn","ts":1710099814.7776074,"msg":"Failed to find a map element","id":2,"type":"BPF_LOG_ID_MAP_LOOKUP_ELEM","ret":0,"cpu":70,"file":"./pkg/ebpf/c/tracee.bpf.c","line":695,"count":1}

These warnings then repeat themselves a lot of times.

Output of tracee version:

v0.11.1

Output of uname -a:

(paste your output here)

Additional details

It seems that in such environments the current map size of 10240 entries is not enough, and gets filled quickly.
A quick mitigation then will be to set a bigger map size.

The root cause of this issue is that we assume that if task_info exists for some task in the task_info_map, then also the proc_info of the process to which the task belongs to also has an entry in the map, but that is not the case.
So the real fix to this issue will be to change all the places that try to get an entry from proc_info_map and make this assumption. Instead, we should reinitialize proc_info_t with the matching pid. One problem to implement the reinitialization is that we already lost some of the information like if it was a "new process", the binary name and for which scopes this process should be followed (used by the follow filter).

[geyslan]

It is reproducible when decreasing proc_info_map to 100 entries (for example):

sudo ./dist/tracee -e sched_process_exec
TIME             UID    COMM             PID     TID     RET              EVENT                     ARGS
14:07:18:109802  1000   sed              3028873 3028873 0                sched_process_exec        cmdpath: /usr/bin/sed, pathname: /usr/bin/sed, dev: 271581187, inode: 14943488, ctime: 1704474426733346008, inode_mode: 33261, interpreter_pathname: /usr/lib/ld-linux-x86-64.so.2, interpreter_dev: 271581187, interpreter_inode: 14953455, interpreter_ctime: 1708634766762078251, argv: [sed -n s/^cpu\s//p /proc/stat], interp: /usr/bin/sed, stdin_type: S_IFSOCK, stdin_path: UNIX-STREAM, invoked_from_kernel: 0, env: <nil>
{"level":"warn","ts":1710263238.1111948,"msg":"Failed to find a map element","id":2,"type":"BPF_LOG_ID_MAP_LOOKUP_ELEM","ret":0,"cpu":11,"file":"./pkg/ebpf/c/tracee.bpf.c","line":531,"count":1}
{"level":"warn","ts":1710263238.1115654,"msg":"Failed to find a map element","id":2,"type":"BPF_LOG_ID_MAP_LOOKUP_ELEM","ret":0,"cpu":0,"file":"./pkg/ebpf/c/tracee.bpf.c","line":1233,"count":1}
{"level":"warn","ts":1710263238.1120112,"msg":"Failed to find a map element","id":2,"type":"BPF_LOG_ID_MAP_LOOKUP_ELEM","ret":0,"cpu":3,"file":"./pkg/ebpf/c/tracee.bpf.c","line":531,"count":1}
{"level":"warn","ts":1710263238.1126502,"msg":"Failed to find a map element","id":2,"type":"BPF_LOG_ID_MAP_LOOKUP_ELEM","ret":0,"cpu":8,"file":"./pkg/ebpf/c/tracee.bpf.c","line":1233,"count":1}
{"level":"warn","ts":1710263238.1134667,"msg":"Failed to find a map element","id":2,"type":"BPF_LOG_ID_MAP_LOOKUP_ELEM","ret":0,"cpu":4,"file":"./pkg/ebpf/c/tracee.bpf.c","line":531,"count":1}
{"level":"warn","ts":1710263238.1137593,"msg":"Failed to find a map element","id":2,"type":"BPF_LOG_ID_MAP_LOOKUP_ELEM","ret":0,"cpu":5,"file":"./pkg/ebpf/c/tracee.bpf.c","line":1233,"count":1}
14:07:18:617060  1000   docker           3028880 3028880 0                sched_process_exec        cmdpath: /usr/bin/docker, pathname: /usr/bin/docker, dev: 271581187, inode: 14967037, ctime: 1710188848527547782, inode_mode: 33261, interpreter_pathname: /usr/lib/ld-linux-x86-64.so.2, interpreter_dev: 271581187, interpreter_inode: 14953455, interpreter_ctime: 1708634766762078251, argv: [docker context ls --format {{json .}}], interp: /usr/bin/docker, stdin_type: S_IFSOCK, stdin_path: UNIX-STREAM, invoked_from_kernel: 0, env: <nil>
14:07:18:745352  0      nmcli            3028889 3028889 0                sched_process_exec        cmdpath: /usr/bin/nmcli, pathname: /usr/bin/nmcli, dev: 271581187, inode: 14973392, ctime: 1710188855744427510, inode_mode: 33261, interpreter_pathname: /usr/lib/ld-linux-x86-64.so.2, interpreter_dev: 271581187, interpreter_inode: 14953455, interpreter_ctime: 1708634766762078251, argv: [nmcli --terse --fields active,ssid,bssid,mode,chan,freq,signal,security,wpa-flags,rsn-flags,device device wifi], interp: /usr/bin/nmcli, stdin_type: S_IFSOCK, stdin_path: UNIX-STREAM, invoked_from_kernel: 0, env: <nil>
{"level":"warn","ts":1710263240.6293547,"msg":"Failed to find a map element","id":2,"type":"BPF_LOG_ID_MAP_LOOKUP_ELEM","ret":0,"cpu":24,"file":"./pkg/ebpf/c/tracee.bpf.c","line":531,"count":1}
{"level":"warn","ts":1710263240.636572,"msg":"Failed to find a map element","id":2,"type":"BPF_LOG_ID_MAP_LOOKUP_ELEM","ret":0,"cpu":6,"file":"./pkg/ebpf/c/tracee.bpf.c","line":1233,"count":1}
{"level":"warn","ts":1710263240.6382625,"msg":"Failed to find a map element","id":2,"type":"BPF_LOG_ID_MAP_LOOKUP_ELEM","ret":0,"cpu":6,"file":"./pkg/ebpf/c/tracee.bpf.c","line":531,"count":1}
{"level":"warn","ts":1710263240.6384227,"msg":"Failed to find a map element","id":2,"type":"BPF_LOG_ID_MAP_LOOKUP_ELEM","ret":0,"cpu":26,"file":"./pkg/ebpf/c/tracee.bpf.c","line":531,"count":1}
{"level":"warn","ts":1710263240.6406047,"msg":"Failed to find a map element","id":2,"type":"BPF_LOG_ID_MAP_LOOKUP_ELEM","ret":0,"cpu":29,"file":"./pkg/ebpf/c/tracee.bpf.c","line":531,"count":1}
{"level":"warn","ts":1710263240.6406913,"msg":"Failed to find a map element","id":2,"type":"BPF_LOG_ID_MAP_LOOKUP_ELEM","ret":0,"cpu":5,"file":"./pkg/ebpf/c/tracee.bpf.c","line":531,"count":1}
14:07:21:755254  1000   sh               3028903 3028903 0                sched_process_exec        cmdpath: /bin/sh, pathname: /usr/bin/bash, dev: 271581187, inode: 14942358, ctime: 1708634769445470347, inode_mode: 33261, interpreter_pathname: /usr/lib/ld-linux-x86-64.so.2, interpreter_dev: 271581187, interpreter_inode: 14953455, interpreter_ctime: 1708634766762078251, argv: [/bin/sh -c which ps], interp: /bin/sh, stdin_type: S_IFSOCK, stdin_path: UNIX-STREAM, invoked_from_kernel: 0, env: <nil>
14:07:21:756811  1000   which            3028903 3028903 0                sched_process_exec        cmdpath: /usr/bin/which, pathname: /usr/bin/which, dev: 271581187, inode: 14943707, ctime: 1704474426940012677, inode_mode: 33261, interpreter_pathname: /usr/lib/ld-linux-x86-64.so.2, interpreter_dev: 271581187, interpreter_inode: 14953455, interpreter_ctime: 1708634766762078251, argv: [which ps], interp: /usr/bin/which, stdin_type: S_IFSOCK, stdin_path: UNIX-STREAM, invoked_from_kernel: 0, env: <nil>