/
kvmexit_example.txt
250 lines (217 loc) · 11.6 KB
/
kvmexit_example.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
Demonstrations of kvm exit reasons, the Linux eBPF/bcc version.
Considering virtual machines' frequent exits can cause performance problems,
this tool aims to locate the frequent exited reasons and then find solutions
to reduce or even avoid the exit, by displaying the detail exit reasons and
the counts of each vm exit for all vms running on one physical machine.
Features of this tool
=====================
- Although there is a patch: [KVM: x86: add full vm-exit reason debug entries]
(https://patchwork.kernel.org/project/kvm/patch/1555939499-30854-1-git-send-email-pizhenwei@bytedance.com/)
trying to fill more vm-exit reason debug entries, just as the comments said,
the code allocates lots of memory that may never be consumed, misses some
arch-specific kvm causes, and can not do kernel aggregation. Instead bcc, as
a user space tool, can implement all these functions more easily and flexibly.
- The bcc python logic could provide nice kernel aggregation and custom output,
like collpasing all tids for one pid (e.i. one vm's qemu process id) with exit
reasons sorted in descending order. For more information, see the following
#USAGE message.
- The bpf in-kernel percpu_array and percpu_cache further improves performance.
For more information, see the following #Help to understand.
Limited
=======
In view of the hardware-assisted virtualization technology of
different architectures, currently we only adapt on vmx in intel.
And the amd feature is on the road..
Example output:
===============
# ./kvmexit.py
Display kvm exit reasons and statistics for all threads... Hit Ctrl-C to end.
PID TID KVM_EXIT_REASON COUNT
^C1273551 1273568 EXIT_REASON_HLT 12
1273551 1273568 EXIT_REASON_MSR_WRITE 6
1274253 1274261 EXIT_REASON_EXTERNAL_INTERRUPT 1
1274253 1274261 EXIT_REASON_HLT 12
1274253 1274261 EXIT_REASON_MSR_WRITE 4
# ./kvmexit.py 6
Display kvm exit reasons and statistics for all threads after sleeping 6 secs.
PID TID KVM_EXIT_REASON COUNT
1273903 1273922 EXIT_REASON_EXTERNAL_INTERRUPT 175
1273903 1273922 EXIT_REASON_CPUID 10
1273903 1273922 EXIT_REASON_HLT 6043
1273903 1273922 EXIT_REASON_IO_INSTRUCTION 24
1273903 1273922 EXIT_REASON_MSR_WRITE 15025
1273903 1273922 EXIT_REASON_PAUSE_INSTRUCTION 11
1273903 1273922 EXIT_REASON_EOI_INDUCED 12
1273903 1273922 EXIT_REASON_EPT_VIOLATION 6
1273903 1273922 EXIT_REASON_EPT_MISCONFIG 380
1273903 1273922 EXIT_REASON_PREEMPTION_TIMER 194
1273551 1273568 EXIT_REASON_EXTERNAL_INTERRUPT 18
1273551 1273568 EXIT_REASON_HLT 989
1273551 1273568 EXIT_REASON_IO_INSTRUCTION 10
1273551 1273568 EXIT_REASON_MSR_WRITE 2205
1273551 1273568 EXIT_REASON_PAUSE_INSTRUCTION 1
1273551 1273568 EXIT_REASON_EOI_INDUCED 5
1273551 1273568 EXIT_REASON_EPT_MISCONFIG 61
1273551 1273568 EXIT_REASON_PREEMPTION_TIMER 14
# ./kvmexit.py -p 1273795 5
Display kvm exit reasons and statistics for PID 1273795 after sleeping 5 secs.
KVM_EXIT_REASON COUNT
MSR_WRITE 13467
HLT 5060
PREEMPTION_TIMER 345
EPT_MISCONFIG 264
EXTERNAL_INTERRUPT 169
EPT_VIOLATION 18
PAUSE_INSTRUCTION 6
IO_INSTRUCTION 4
EOI_INDUCED 2
# ./kvmexit.py -p 1273795 5 -a
Display kvm exit reasons and statistics for PID 1273795 and its all threads after sleeping 5 secs.
TID KVM_EXIT_REASON COUNT
1273819 EXTERNAL_INTERRUPT 64
1273819 HLT 2802
1273819 IO_INSTRUCTION 4
1273819 MSR_WRITE 7196
1273819 PAUSE_INSTRUCTION 2
1273819 EOI_INDUCED 2
1273819 EPT_VIOLATION 6
1273819 EPT_MISCONFIG 162
1273819 PREEMPTION_TIMER 194
1273820 EXTERNAL_INTERRUPT 78
1273820 HLT 2054
1273820 MSR_WRITE 5199
1273820 EPT_VIOLATION 2
1273820 EPT_MISCONFIG 77
1273820 PREEMPTION_TIMER 102
# ./kvmexit.py -p 1273795 -v 0
Display kvm exit reasons and statistics for PID 1273795 VCPU 0... Hit Ctrl-C to end.
KVM_EXIT_REASON COUNT
^CMSR_WRITE 2076
HLT 795
PREEMPTION_TIMER 86
EXTERNAL_INTERRUPT 20
EPT_MISCONFIG 10
PAUSE_INSTRUCTION 2
IO_INSTRUCTION 2
EPT_VIOLATION 1
EOI_INDUCED 1
# ./kvmexit.py -p 1273795 -v 0 4
Display kvm exit reasons and statistics for PID 1273795 VCPU 0 after sleeping 4 secs.
KVM_EXIT_REASON COUNT
MSR_WRITE 4726
HLT 1827
PREEMPTION_TIMER 78
EPT_MISCONFIG 67
EXTERNAL_INTERRUPT 28
IO_INSTRUCTION 4
EOI_INDUCED 2
PAUSE_INSTRUCTION 2
# ./kvmexit.py -p 1273795 -v 4 4
Traceback (most recent call last):
File "tools/kvmexit.py", line 306, in <module>
raise Exception("There's no v%s for PID %d." % (tgt_vcpu, args.pid))
Exception: There's no vCPU 4 for PID 1273795.
# ./kvmexit.py -t 1273819 10
Display kvm exit reasons and statistics for TID 1273819 after sleeping 10 secs.
KVM_EXIT_REASON COUNT
MSR_WRITE 13318
HLT 5274
EPT_MISCONFIG 263
PREEMPTION_TIMER 171
EXTERNAL_INTERRUPT 109
IO_INSTRUCTION 8
PAUSE_INSTRUCTION 5
EOI_INDUCED 4
EPT_VIOLATION 2
# ./kvmexit.py -T '1273820,1273819'
Display kvm exit reasons and statistics for TIDS ['1273820', '1273819']... Hit Ctrl-C to end.
TIDS KVM_EXIT_REASON COUNT
^C1273819 EXTERNAL_INTERRUPT 300
1273819 HLT 13718
1273819 IO_INSTRUCTION 26
1273819 MSR_WRITE 37457
1273819 PAUSE_INSTRUCTION 13
1273819 EOI_INDUCED 13
1273819 EPT_VIOLATION 53
1273819 EPT_MISCONFIG 654
1273819 PREEMPTION_TIMER 958
1273820 EXTERNAL_INTERRUPT 212
1273820 HLT 9002
1273820 MSR_WRITE 25495
1273820 PAUSE_INSTRUCTION 2
1273820 EPT_VIOLATION 64
1273820 EPT_MISCONFIG 396
1273820 PREEMPTION_TIMER 268
Help to understand
==================
We use a PERCPU_ARRAY: pcpuArrayA and a percpu_hash: hashA to collaboratively
store each kvm exit reason and its count. The reason is there exists a rule when
one vcpu exits and re-enters, it tends to continue to run on the same physical
cpu (pcpu as follows) as the last cycle, which is also called 'cache hit'. Thus
we turn to use a PERCPU_ARRAY to record the 'cache hit' situation to speed
things up; and for other cases, then use a percpu_hash.
BTW, we originally use a common hash to do this, with a u64(exit_reason)
key and a struct exit_info {tgid_pid, exit_reason} value. But due to
the big lock in bpf_hash, each updating is quite performance consuming.
Now imagine here is a pid_tgidA (vcpu A) exits and is going to run on
pcpuArrayA, the BPF code flow is as follows:
pid_tgidA keeps running on the same pcpu
// \\
// \\
// Y N \\
// \\
a. cache_hit b. cache_miss
(cacheA's pid_tgid matches pid_tgidA) ||
| ||
| ||
"increase percpu exit_ct and return" ||
[*Note*] ||
pid_tgidA ever been exited on pcpuArrayA?
// \\
// \\
// \\
// Y N \\
// \\
b.a load_last_hashA b.b initialize_hashA_with_zero
\ /
\ /
\ /
"increase percpu exit_ct"
||
||
is another pid_tgid been running on pcpuArrayA?
// \\
// Y N \\
// \\
b.*.a save_theLastHit_hashB do_nothing
\\ //
\\ //
\\ //
b.* save_to_pcpuArrayA
[*Note*] we do not update the table in above "a.", in case the vcpu hit the same
pcpu again when exits next time, instead we only update until this pcpu is not
hitted by the same tgidpid(vcpu) again, which is in "b.*.a" and "b.*".
USAGE message:
==============
# ./kvmexit.py -h
usage: kvmexit.py [-h] [-p PID [-v VCPU | -a] ] [-t TID | -T 'TID1,TID2'] [duration]
Display kvm_exit_reason and its statistics at a timed interval
optional arguments:
-h, --help show this help message and exit
-p PID, --pid PID display process with this PID only, collpase all tids with exit reasons sorted in descending order
-v VCPU, --v VCPU display this VCPU only for this PID
-a, --alltids display all TIDS for this PID
-t TID, --tid TID display thread with this TID only with exit reasons sorted in descending order
-T 'TID1,TID2', --tids 'TID1,TID2'
display threads for a union like {395490, 395491}
duration duration of display, after sleeping several seconds
examples:
./kvmexit # Display kvm_exit_reason and its statistics in real-time until Ctrl-C
./kvmexit 5 # Display in real-time after sleeping 5s
./kvmexit -p 3195281 # Collpase all tids for pid 3195281 with exit reasons sorted in descending order
./kvmexit -p 3195281 20 # Collpase all tids for pid 3195281 with exit reasons sorted in descending order, and display after sleeping 20s
./kvmexit -p 3195281 -v 0 # Display only vcpu0 for pid 3195281, descending sort by default
./kvmexit -p 3195281 -a # Display all tids for pid 3195281
./kvmexit -t 395490 # Display only for tid 395490 with exit reasons sorted in descending order
./kvmexit -t 395490 20 # Display only for tid 395490 with exit reasons sorted in descending order after sleeping 20s
./kvmexit -T '395490,395491' # Display for a union like {395490, 395491}