forked from iovisor/bcc
/
drsnoop_example.txt
149 lines (124 loc) · 5 KB
/
drsnoop_example.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
Demonstrations of drsnoop, the Linux eBPF/bcc version.
drsnoop traces the direct reclaim system-wide, and prints various details.
Example output:
# ./drsnoop
COMM PID LAT(ms) PAGES
summond 17678 0.19 143
summond 17669 0.55 313
summond 17669 0.15 145
summond 17669 0.27 237
summond 17669 0.48 111
summond 17669 0.16 75
head 17821 0.29 339
head 17825 0.17 109
summond 17669 0.14 73
summond 17496 104.84 40
summond 17678 0.32 167
summond 17678 0.14 106
summond 17678 0.16 67
summond 17678 0.29 267
summond 17678 0.27 69
summond 17678 0.32 46
base64 17816 0.16 85
summond 17678 0.43 283
summond 17678 0.14 182
head 17736 0.57 135
^C
While tracing, the processes alloc pages，due to insufficient memory available
in the system, direct reclaim events happened, which will increase the waiting
delay of the processes.
drsnoop can be useful for discovering when allocstall(/proc/vmstat) continues to increase,
whether it is caused by some critical processes or not.
The -p option can be used to filter on a PID, which is filtered in-kernel. Here
I've used it with -T to print timestamps:
# ./drsnoop -Tp 17491
TIME(s) COMM PID LAT(ms) PAGES
107.364115000 summond 17491 0.24 50
107.364550000 summond 17491 0.26 38
107.365266000 summond 17491 0.36 72
107.365753000 summond 17491 0.22 49
^C
This shows the summond process allocs pages, and direct reclaim events happening,
and the delays are not affected much.
The -U option include UID on output:
# ./drsnoop -U
UID COMM PID LAT(ms) PAGES
1000 summond 17678 0.32 46
0 base64 17816 0.16 85
1000 summond 17678 0.43 283
1000 summond 17678 0.14 182
0 head 17821 0.29 339
0 head 17825 0.17 109
^C
The -u option filtering UID:
# ./drsnoop -Uu 1000
UID COMM PID LAT(ms) PAGES
1000 summond 17678 0.19 143
1000 summond 17669 0.55 313
1000 summond 17669 0.15 145
1000 summond 17669 0.27 237
1000 summond 17669 0.48 111
1000 summond 17669 0.16 75
1000 summond 17669 0.14 73
1000 summond 17678 0.32 167
^C
A maximum tracing duration can be set with the -d option. For example, to trace
for 2 seconds:
# ./drsnoop -d 2
COMM PID LAT(ms) PAGES
head 21715 0.15 195
The -n option can be used to filter on process name using partial matches:
# ./drsnoop -n mond
COMM PID LAT(ms) PAGES
summond 10271 0.03 51
summond 10271 0.03 51
summond 10259 0.05 51
summond 10269 319.41 37
summond 10270 111.73 35
summond 10270 0.11 78
summond 10270 0.12 71
summond 10270 0.03 35
summond 10277 111.62 41
summond 10277 0.08 45
summond 10277 0.06 32
^C
This caught the 'summond' command because it partially matches 'mond' that's passed
to the '-n' option.
The -v option can be used to show system memory state (now only free mem) at
the beginning of direct reclaiming:
# ./drsnoop.py -v
COMM PID LAT(ms) PAGES FREE(KB)
base64 34924 0.23 151 86260
base64 34962 0.26 149 86260
head 34931 0.24 150 86260
base64 34902 0.19 148 86260
head 34963 0.19 151 86228
base64 34959 0.17 151 86228
head 34965 0.29 190 86228
base64 34957 0.24 152 86228
summond 34870 0.15 151 86080
summond 34870 0.12 115 86184
USAGE message:
# ./drsnoop -h
usage: drsnoop.py [-h] [-T] [-U] [-p PID] [-t TID] [-u UID] [-d DURATION]
[-n NAME]
Trace direct reclaim
optional arguments:
-h, --help show this help message and exit
-T, --timestamp include timestamp on output
-U, --print-uid print UID column
-p PID, --pid PID trace this PID only
-t TID, --tid TID trace this TID only
-u UID, --uid UID trace this UID only
-d DURATION, --duration DURATION
total duration of trace in seconds
-n NAME, --name NAME only print process names containing this name
examples:
./drsnoop # trace all direct reclaim
./drsnoop -T # include timestamps
./drsnoop -U # include UID
./drsnoop -p 181 # only trace PID 181
./drsnoop -t 123 # only trace TID 123
./drsnoop -u 1000 # only trace UID 1000
./drsnoop -d 10 # trace for 10 seconds only
./drsnoop -n main # only print process names containing "main"