forked from iovisor/bcc
/
exitsnoop_example.txt
150 lines (128 loc) · 6.22 KB
/
exitsnoop_example.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
Demonstrations of exitsnoop.
This Linux tool traces all process terminations and reason, it
- is implemented using BPF, which requires CAP_SYS_ADMIN and
should therefore be invoked with sudo
- traces sched_process_exit tracepoint in kernel/exit.c
- includes processes by root and all users
- includes processes in containers
- includes processes that become zombie
The following example shows the termination of the 'sleep' and 'bash' commands
when run in a loop that is interrupted with Ctrl-C from the terminal:
# ./exitsnoop.py > exitlog &
[1] 18997
# for((i=65;i<100;i+=5)); do bash -c "sleep 1.$i;exit $i"; done
^C
# fg
./exitsnoop.py > exitlog
^C
# cat exitlog
PCOMM PID PPID TID AGE(s) EXIT_CODE
sleep 19004 19003 19004 1.65 0
bash 19003 17656 19003 1.65 code 65
sleep 19007 19006 19007 1.70 0
bash 19006 17656 19006 1.70 code 70
sleep 19010 19009 19010 1.75 0
bash 19009 17656 19009 1.75 code 75
sleep 19014 19013 19014 0.23 signal 2 (INT)
bash 19013 17656 19013 0.23 signal 2 (INT)
#
The output shows the process/command name (PCOMM), the PID,
the process that will be notified (PPID), the thread (TID), the AGE
of the process with hundredth of a second resolution, and the reason for
the process exit (EXIT_CODE).
A -t option can be used to include a timestamp column, it shows local time
by default. The --utc option shows the time in UTC. The --label
option adds a column indicating the tool that generated the output,
'exit' by default. If other tools follow this format their outputs
can be merged into a single trace with a simple lexical sort
increasing in time order with each line labeled to indicate the event,
e.g. 'exec', 'open', 'exit', etc. Time is displayed with millisecond
resolution. The -x option will show only non-zero exits and fatal
signals, which excludes processes that exit with 0 code:
# ./exitsnoop.py -t --utc -x --label= > exitlog &
[1] 18289
# for((i=65;i<100;i+=5)); do bash -c "sleep 1.$i;exit $i"; done
^C
# fg
./exitsnoop.py -t --utc -x --label= > exitlog
^C
# cat exitlog
TIME-UTC LABEL PCOMM PID PPID TID AGE(s) EXIT_CODE
13:20:22.997 exit bash 18300 17656 18300 1.65 code 65
13:20:24.701 exit bash 18303 17656 18303 1.70 code 70
13:20:26.456 exit bash 18306 17656 18306 1.75 code 75
13:20:28.260 exit bash 18310 17656 18310 1.80 code 80
13:20:30.113 exit bash 18313 17656 18313 1.85 code 85
13:20:31.495 exit sleep 18318 18317 18318 1.38 signal 2 (INT)
13:20:31.495 exit bash 18317 17656 18317 1.38 signal 2 (INT)
#
USAGE message:
# ./exitsnoop.py -h
usage: exitsnoop.py [-h] [-t] [--utc] [-p PID] [--label LABEL] [-x] [--per-thread]
Trace all process termination (exit, fatal signal)
optional arguments:
-h, --help show this help message and exit
-t, --timestamp include timestamp (local time default)
--utc include timestamp in UTC (-t implied)
-p PID, --pid PID trace this PID only
--label LABEL label each line
-x, --failed trace only fails, exclude exit(0)
--per-thread trace per thread termination
examples:
exitsnoop # trace all process termination
exitsnoop -x # trace only fails, exclude exit(0)
exitsnoop -t # include timestamps (local time)
exitsnoop --utc # include timestamps (UTC)
exitsnoop -p 181 # only trace PID 181
exitsnoop --label=exit # label each output line with 'exit'
exitsnoop --per-thread # trace per thread termination
Exit status:
0 EX_OK Success
2 argparse error
70 EX_SOFTWARE syntax error detected by compiler, or
verifier error from kernel
77 EX_NOPERM Need sudo (CAP_SYS_ADMIN) for BPF() system call
About process termination in Linux
----------------------------------
A program/process on Linux terminates normally
- by explicitly invoking the exit( int ) system call
- in C/C++ by returning an int from main(),
...which is then used as the value for exit()
- by reaching the end of main() without a return
...which is equivalent to return 0 (C99 and C++)
Notes:
- Linux keeps only the least significant eight bits of the exit value
- an exit value of 0 means success
- an exit value of 1-255 means an error
A process terminates abnormally if it
- receives a signal which is not ignored or blocked and has no handler
... the default action is to terminate with optional core dump
- is selected by the kernel's "Out of Memory Killer",
equivalent to being sent SIGKILL (9), which cannot be ignored or blocked
Notes:
- any signal can be sent asynchronously via the kill() system call
- synchronous signals are the result of the CPU detecting
a fault or trap during execution of the program, a kernel handler
is dispatched which determines the cause and the corresponding
signal, examples are
- attempting to fetch data or instructions at invalid or
privileged addresses,
- attempting to divide by zero, unmasked floating point exceptions
- hitting a breakpoint
Linux keeps process termination information in 'exit_code', an int
within struct 'task_struct' defined in <linux/sched.c>
- if the process terminated normally:
- the exit value is in bits 15:8
- the least significant 8 bits of exit_code are zero (bits 7:0)
- if the process terminates abnormally:
- the signal number (>= 1) is in bits 6:0
- bit 7 indicates a 'core dump' action, whether a core dump was
actually done depends on ulimit.
Success is indicated with an exit value of zero.
The meaning of a non zero exit value depends on the program.
Some programs document their exit values and their meaning.
This script uses exit values as defined in <include/sysexits.h>
References:
https://github.com/torvalds/linux/blob/master/kernel/exit.c
https://github.com/torvalds/linux/blob/master/arch/x86/include/uapi/asm/signal.h
https://code.woboq.org/userspace/glibc/misc/sysexits.h.html