This is a very (very!) basic guide on how to enable and configure kdump and sos to collect and analyze vmcore and sosreport log files. Also, an overview on how to make a very simple performance troubleshooting.
- RH342 - Red Hat Enterprise Linux Diagnostics and Troubleshooting
- RH442 - Red Hat Performance Tuning: Linux in Physical, Virtual, and Cloud
- Monitoring and managing system status and performance
- Chapter 10. Installing kdump
- How to use the SysRq facility to collect information from a RHEL server
- Vmcore analysis techniques
- Kernel Oops Analyzer
- Kdump Helper
- A Guide to Unexpected System Restarts
- How to analyze and interpret sar data
- xsos -- a tool for sysadmins and support techs
- Brendan Gregg - Linux Performance
[root@demo ~]# rpm -q kexec-tools
kexec-tools-2.0.20-68.el8.x86_64
[root@demo ~]# systemctl status kdump.service
● kdump.service - Crash recovery kernel arming
Loaded: loaded (/usr/lib/systemd/system/kdump.service; enabled; vendor preset: enabled)
Active: active (exited) since Wed 2022-08-17 17:30:31 -03; 1 day 20h ago
Main PID: 1326 (code=exited, status=0/SUCCESS)
Tasks: 0 (limit: 23644)
Memory: 0B
CGroup: /system.slice/kdump.service
ago 17 17:30:29 demo.example.local systemd[1]: Starting Crash recovery kernel arming...
ago 17 17:30:31 demo.example.local kdumpctl[1338]: kdump: kexec: loaded kdump kernel
ago 17 17:30:31 demo.example.local kdumpctl[1338]: kdump: Starting kdump: [OK]
ago 17 17:30:31 demo.example.local systemd[1]: Started Crash recovery kernel arming.
[root@demo ~]# grep crash /etc/default/grub
GRUB_CMDLINE_LINUX="crashkernel=auto resume=/dev/mapper/VG_01-swap rd.lvm.lv=VG_01/root rd.lvm.lv=VG_01/swap rhgb quiet"
[root@demo ~]# kdumpctl estimate
Reserved crashkernel: 192M
Recommended crashkernel: 192M
Kernel image size: 54M
Kernel modules size: 8M
Initramfs size: 26M
Runtime reservation: 64M
Large modules:
xfs: 1556480
[root@demo ~]# kdumpctl showmem
kdump: Reserved 192MB memory for crash kernel
RHEL 8 (Manually open and edit the grub config file in the kernel line with the crashkernel parameter)
[root@demo ~]# vi /etc/default/grub
[root@demo ~]# grep crash /etc/default/grub
GRUB_CMDLINE_LINUX="crashkernel=192M resume=/dev/mapper/VG_01-swap rd.lvm.lv=VG_01/root rd.lvm.lv=VG_01/swap rhgb quiet"
[root@demo ~]# grub2-mkconfig -o /boot/grub2/grub.cfg
[root@demo ~]# reboot
[root@demo ~]# kdumpctl reset-crashkernel --kernel=ALL
[root@demo ~]# grubby --update-kernel ALL --args "crashkernel=192M”
[root@demo ~]# grep crash /etc/default/grub
GRUB_CMDLINE_LINUX="crashkernel=192M resume=/dev/mapper/VG_01-swap rd.lvm.lv=VG_01/root rd.lvm.lv=VG_01/swap rhgb quiet"
[root@demo ~]# reboot
[root@demo ~]# echo c > /proc/sysrq-trigger
[root@demo ~]# ls /var/crash/127.0.0.1-2022-08-16-16\:32\:51/
kexec-dmesg.log vmcore vmcore-dmesg.txt
[root@demo ~]# grep 'Command line' /var/log/messages
Aug 17 15:01:47 demo kernel: Command line: BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-372.16.1.el8_6.x86_64 root=/dev/mapper/VG_01-root ro crashkernel=auto resume=/dev/mapper/VG_01-swap rd.lvm.lv=VG_01/root rd.lvm.lv=VG_01/swap rhgb quiet
Aug 17 15:32:14 demo kernel: Command line: BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-348.20.1.el8_5.x86_64 root=/dev/mapper/VG_01-root ro crashkernel=auto resume=/dev/mapper/VG_01-swap rd.lvm.lv=VG_01/root rd.lvm.lv=VG_01/swap rhgb quiet
Aug 17 15:49:50 demo kernel: Command line: BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-372.16.1.el8_6.x86_64 root=/dev/mapper/VG_01-root ro crashkernel=auto resume=/dev/mapper/VG_01-swap rd.lvm.lv=VG_01/root rd.lvm.lv=VG_01/swap rhgb quiet
Aug 17 16:36:30 demo kernel: Command line: BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-372.19.1.el8_6.x86_64 root=/dev/mapper/VG_01-root ro crashkernel=auto resume=/dev/mapper/VG_01-swap rd.lvm.lv=VG_01/root rd.lvm.lv=VG_01/swap rhgb quiet
Aug 17 17:23:39 demo kernel: Command line: BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-372.19.1.el8_6.x86_64 root=/dev/mapper/VG_01-root ro crashkernel=auto resume=/dev/mapper/VG_01-swap rd.lvm.lv=VG_01/root rd.lvm.lv=VG_01/swap rhgb quiet
Aug 17 17:30:22 demo kernel: Command line: BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-372.19.1.el8_6.x86_64 root=/dev/mapper/VG_01-root ro crashkernel=auto resume=/dev/mapper/VG_01-swap rd.lvm.lv=VG_01/root rd.lvm.lv=VG_01/swap rhgb quiet
[root@demo ~]# last | grep reboot
reboot system boot 4.18.0-372.19.1. Wed Aug 17 17:30 still running
reboot system boot 4.18.0-372.19.1. Wed Aug 17 17:23 - 17:30 (00:06)
reboot system boot 4.18.0-372.19.1. Wed Aug 17 16:36 - 17:23 (00:46)
reboot system boot 4.18.0-372.16.1. Wed Aug 17 15:49 - 16:36 (00:46)
reboot system boot 4.18.0-348.20.1. Wed Aug 17 15:32 - 15:49 (00:17)
reboot system boot 4.18.0-372.16.1. Wed Aug 17 15:01 - 15:31 (00:30)
Basic vmcore analysis (PS. If it's in another system, make sure you have the same kernel version of the generated vmcore)
[root@demo ~]# dnf repolist
Updating Subscription Management repositories.
repo id repo name
advanced-virt-for-rhel-8-x86_64-rpms Advanced Virtualization for RHEL 8 x86_64 (RPMs)
codeready-builder-for-rhel-8-x86_64-rpms Red Hat CodeReady Linux Builder for RHEL 8 x86_64 (RPMs)
epel Extra Packages for Enterprise Linux 8 - x86_64
epel-modular Extra Packages for Enterprise Linux Modular 8 - x86_64
rhel-8-for-x86_64-appstream-debug-rpms Red Hat Enterprise Linux 8 for x86_64 - AppStream (Debug RPMs)
rhel-8-for-x86_64-appstream-rpms Red Hat Enterprise Linux 8 for x86_64 - AppStream (RPMs)
rhel-8-for-x86_64-baseos-debug-rpms Red Hat Enterprise Linux 8 for x86_64 - BaseOS (Debug RPMs)
rhel-8-for-x86_64-baseos-rpms Red Hat Enterprise Linux 8 for x86_64 - BaseOS (RPMs)
rhel-8-for-x86_64-highavailability-rpms Red Hat Enterprise Linux 8 for x86_64 - High Availability (RPMs)
rhel-8-for-x86_64-resilientstorage-rpms Red Hat Enterprise Linux 8 for x86_64 - Resilient Storage (RPMs)
rhel-8-for-x86_64-rt-rpms Red Hat Enterprise Linux 8 for x86_64 - Real Time (RPMs)
rhel-8-for-x86_64-supplementary-debug-rpms Red Hat Enterprise Linux 8 for x86_64 - Supplementary (Debug RPMs)
rhel-8-for-x86_64-supplementary-rpms Red Hat Enterprise Linux 8 for x86_64 - Supplementary (RPMs)
rsawaroha rsaw aroha rpms for Fedora/RHEL6+
satellite-client-6-for-rhel-8-x86_64-rpms Red Hat Satellite Client 6 for RHEL 8 x86_64 (RPMs)
[root@demo ~]# rpm -qa | grep kernel
kernel-modules-extra-4.18.0-372.19.1.el8_6.x86_64
kernel-modules-4.18.0-372.16.1.el8_6.x86_64
kernel-devel-4.18.0-372.16.1.el8_6.x86_64
kernel-debug-modules-4.18.0-372.16.1.el8_6.x86_64
kernel-tools-4.18.0-372.19.1.el8_6.x86_64
kernel-core-4.18.0-372.19.1.el8_6.x86_64
kernel-debuginfo-common-x86_64-4.18.0-372.19.1.el8_6.x86_64
kernel-debug-modules-extra-4.18.0-372.19.1.el8_6.x86_64
kernel-debuginfo-4.18.0-372.16.1.el8_6.x86_64
kernel-debug-modules-4.18.0-372.19.1.el8_6.x86_64
kernel-debuginfo-4.18.0-372.19.1.el8_6.x86_64
kernel-4.18.0-372.19.1.el8_6.x86_64
kernel-devel-4.18.0-372.19.1.el8_6.x86_64
kernel-headers-4.18.0-372.19.1.el8_6.x86_64
kernel-debuginfo-common-x86_64-4.18.0-372.16.1.el8_6.x86_64
kernel-4.18.0-372.16.1.el8_6.x86_64
kernel-modules-extra-4.18.0-372.16.1.el8_6.x86_64
kernel-debug-modules-extra-4.18.0-372.16.1.el8_6.x86_64
kernel-modules-4.18.0-372.19.1.el8_6.x86_64
kernel-core-4.18.0-372.16.1.el8_6.x86_64
kernel-debug-core-4.18.0-372.19.1.el8_6.x86_64
kernel-debug-core-4.18.0-372.16.1.el8_6.x86_64
kernel-tools-libs-4.18.0-372.19.1.el8_6.x86_64
[root@demo ~]# rpm -qa | grep crash
crash-7.3.1-5.el8.x86_64
[root@demo ~]# crash /usr/lib/debug/lib/modules/4.18.0-372.16.1.el8_6.x86_64/vmlinux /var/crash/127.0.0.1-2022-08-16-16\:32\:51/vmcore
crash 7.3.1-5.el8
Copyright (C) 2002-2021 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011, 2020-2021 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...
WARNING: kernel relocated [744MB]: patching 105274 gdb minimal_symbol values
KERNEL: /usr/lib/debug/lib/modules/4.18.0-372.16.1.el8_6.x86_64/vmlinux
DUMPFILE: /var/crash/127.0.0.1-2022-08-16-16:32:51/vmcore [PARTIAL DUMP]
CPUS: 2
DATE: Tue Aug 16 16:32:41 -03 2022
UPTIME: 22:44:27
LOAD AVERAGE: 0.03, 0.03, 0.02
TASKS: 170
NODENAME: demo.example.local
RELEASE: 4.18.0-372.16.1.el8_6.x86_64
VERSION: #1 SMP Tue Jun 28 03:02:21 EDT 2022
MACHINE: x86_64 (2112 Mhz)
MEMORY: 4 GB
PANIC: "sysrq: SysRq : Trigger a crash"
PID: 32874
COMMAND: "bash"
TASK: ffff8931c7f4d200 [THREAD_INFO: ffff8931c7f4d200]
CPU: 1
STATE: TASK_RUNNING (SYSRQ)
crash> help
* extend log rd task
alias files mach repeat timer
ascii foreach mod runq tree
bpf fuser mount search union
bt gdb net set vm
btop help p sig vtop
dev ipcs ps struct waitq
dis irq pte swap whatis
eval kmem ptob sym wr
exit list ptov sys q
crash version: 7.3.1-5.el8 gdb version: 7.6
For help on any command above, enter "help <command>".
For help on input options, enter "help input".
For help on output options, enter "help output".
crash> log
...skipping...
[81674.123533] 15922 pages in swap cache
[81674.123534] Swap cache stats: add 170452, delete 154532, find 38295/42165
[81674.123535] Free swap = 964296kB
[81674.123545] Total swap = 1048572kB
[81674.123546] 1048443 pages RAM
[81674.123547] 0 pages HighMem/MovableOnly
[81674.123548] 93115 pages reserved
[81674.123548] 0 pages hwpoisoned
[81867.366572] sysrq: SysRq : Trigger a crash
[81867.366731] Kernel panic - not syncing: sysrq triggered crash
[81867.366988] CPU: 1 PID: 32874 Comm: bash Kdump: loaded Not tainted 4.18.0-372.16.1.el8_6.x86_64 #1
[81867.367139] Hardware name: Red Hat KVM, BIOS 1.11.0-2.el7 04/01/2014
[81867.367280] Call Trace:
[81867.367439] dump_stack+0x41/0x60
[81867.367582] panic+0xe7/0x2ac
[81867.369759] ? printk+0x58/0x6f
[81867.369905] sysrq_handle_crash+0x11/0x20
[81867.370037] __handle_sysrq.cold.13+0x48/0xfb
[81867.370169] write_sysrq_trigger+0x2b/0x30
[81867.370296] proc_reg_write+0x39/0x60
[81867.370439] vfs_write+0xa5/0x1a0
[81867.370568] ksys_write+0x4f/0xb0
[81867.370694] do_syscall_64+0x5b/0x1a0
[81867.370827] entry_SYSCALL_64_after_hwframe+0x65/0xca
[81867.370958] RIP: 0033:0x7f64ad4c55a8
[81867.371088] Code: 89 02 48 c7 c0 ff ff ff ff eb b3 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 f5 3f 2a 00 8b 00 85 c0 75 17 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 58 c3 0f 1f 80 00 00 00 00 41 54 49 89 d4 55
[81867.371372] RSP: 002b:00007fff0ea6b318 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
[81867.371512] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f64ad4c55a8
[81867.371659] RDX: 0000000000000002 RSI: 0000557c58a8f8e0 RDI: 0000000000000001
[81867.371794] RBP: 0000557c58a8f8e0 R08: 000000000000000a R09: 00007f64ad525800
[81867.371933] R10: 000000000000000a R11: 0000000000000246 R12: 00007f64ad7656e0
[81867.372064] R13: 0000000000000002 R14: 00007f64ad760860 R15: 0000000000000002
crash> bt
PID: 32874 TASK: ffff8931c7f4d200 CPU: 1 COMMAND: "bash"
#0 [ffffa72f80d9fcd0] machine_kexec at ffffffffaf86504e
#1 [ffffa72f80d9fd28] __crash_kexec at ffffffffaf9a53ad
#2 [ffffa72f80d9fdf0] panic at ffffffffaf8ed877
#3 [ffffa72f80d9fe70] sysrq_handle_crash at ffffffffafdc4801
#4 [ffffa72f80d9fe78] __handle_sysrq.cold.13 at ffffffffafdc50c8
#5 [ffffa72f80d9fea8] write_sysrq_trigger at ffffffffafdc4f8b
#6 [ffffa72f80d9feb8] proc_reg_write at ffffffffafbc3289
#7 [ffffa72f80d9fed0] vfs_write at ffffffffafb40475
#8 [ffffa72f80d9ff00] ksys_write at ffffffffafb406ef
#9 [ffffa72f80d9ff38] do_syscall_64 at ffffffffaf80430b
#10 [ffffa72f80d9ff50] entry_SYSCALL_64_after_hwframe at ffffffffb02000ad
RIP: 00007f64ad4c55a8 RSP: 00007fff0ea6b318 RFLAGS: 00000246
RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f64ad4c55a8
RDX: 0000000000000002 RSI: 0000557c58a8f8e0 RDI: 0000000000000001
RBP: 0000557c58a8f8e0 R8: 000000000000000a R9: 00007f64ad525800
R10: 000000000000000a R11: 0000000000000246 R12: 00007f64ad7656e0
R13: 0000000000000002 R14: 00007f64ad760860 R15: 0000000000000002
ORIG_RAX: 0000000000000001 CS: 0033 SS: 002b
crash> ps
PID PPID CPU TASK ST %MEM VSZ RSS COMM
...skipping...
> 32874 32873 1 ffff8931c7f4d200 RU 0.1 28236 4256 bash
41290 2 0 ffff8931c1cad200 ID 0.0 0 0 [kworker/u4:2]
42813 2 0 ffff8931c0c5a900 ID 0.0 0 0 [kworker/0:3]
43042 2 1 ffff89323bd90000 RU 0.0 0 0 [kworker/1:0]
43043 2 1 ffff8931c2bcd200 ID 0.0 0 0 [kworker/u4:1]
43045 2 1 ffff8931c4a35200 ID 0.0 0 0 [kworker/1:2]
43048 1 1 ffff8931c5620000 IN 0.5 143424 32624 pmproxy
43659 2 0 ffff8931c33a2900 ID 0.0 0 0 [kworker/0:0]
43716 2 0 ffff8931c326d200 ID 0.0 0 0 [kworker/u4:0]
43724 915 0 ffff8931c575d200 IN 0.0 7312 1848 sleep
43725 2 0 ffff8931c2982900 ID 0.0 0 0 [kworker/0:1]
crash> task 32874
PID: 32874 TASK: ffff8931c7f4d200 CPU: 1 COMMAND: "bash"
struct task_struct {
thread_info = {
flags = 2147483776,
status = 0
},
{
{
__state = 0
},
rh_kabi_hidden_742 = {
state = 0
},
{<No data fields>}
},
stack = 0xffffa72f80d9c000,
{
usage = {
refs = {
counter = 1
}
},
rh_kabi_hidden_751 = {
usage = {
counter = 1
}
},
{<No data fields>}
},
...skipping...
crash> files
PID: 32874 TASK: ffff8931c7f4d200 CPU: 1 COMMAND: "bash"
ROOT: / CWD: /root
FD FILE DENTRY INODE TYPE PATH
0 ffff8931c2f92b00 ffff8931c8137d80 ffff8931fb280cd0 CHR /dev/pts/0
1 ffff8931c585a200 ffff8931c3a8d9c0 ffff8931c8311f90 REG /proc/sysrq-trigger
2 ffff8931c2f92b00 ffff8931c8137d80 ffff8931fb280cd0 CHR /dev/pts/0
3 ffff8931c54bd700 ffff8931c8029cc0 ffff8931c801ebb0 REG /var/lib/sss/mc/passwd
4 ffff8931c54bd200 ffff8931c811f240 ffff8931c8266970 SOCK UNIX
10 ffff8931c2f92b00 ffff8931c8137d80 ffff8931fb280cd0 CHR /dev/pts/0
255 ffff8931c2f92b00 ffff8931c8137d80 ffff8931fb280cd0 CHR /dev/pts/0
crash> dis -r ffffffffaf9a53ad
0xffffffffaf9a5340 <__crash_kexec>: nopl 0x0(%rax,%rax,1) [FTRACE NOP]
0xffffffffaf9a5345 <__crash_kexec+5>: push %rbp
0xffffffffaf9a5346 <__crash_kexec+6>: push %rbx
0xffffffffaf9a5347 <__crash_kexec+7>: mov %rdi,%rbx
0xffffffffaf9a534a <__crash_kexec+10>: mov $0xffffffffb1435e40,%rdi
0xffffffffaf9a5351 <__crash_kexec+17>: sub $0xb0,%rsp
0xffffffffaf9a5358 <__crash_kexec+24>: mov %gs:0x28,%rax
0xffffffffaf9a5361 <__crash_kexec+33>: mov %rax,0xa8(%rsp)
0xffffffffaf9a5369 <__crash_kexec+41>: xor %eax,%eax
0xffffffffaf9a536b <__crash_kexec+43>: callq 0xffffffffb01a2020 <mutex_trylock>
0xffffffffaf9a5370 <__crash_kexec+48>: test %eax,%eax
0xffffffffaf9a5372 <__crash_kexec+50>: je 0xffffffffaf9a53b9 <__crash_kexec+121>
0xffffffffaf9a5374 <__crash_kexec+52>: cmpq $0x0,0x2401e7c(%rip) # 0xffffffffb1da71f8
0xffffffffaf9a537c <__crash_kexec+60>: je 0xffffffffaf9a53ad <__crash_kexec+109>
0xffffffffaf9a537e <__crash_kexec+62>: test %rbx,%rbx
0xffffffffaf9a5381 <__crash_kexec+65>: je 0xffffffffaf9a53da <__crash_kexec+154>
0xffffffffaf9a5383 <__crash_kexec+67>: mov %rsp,%rbp
0xffffffffaf9a5386 <__crash_kexec+70>: mov $0x15,%ecx
0xffffffffaf9a538b <__crash_kexec+75>: mov %rbx,%rsi
0xffffffffaf9a538e <__crash_kexec+78>: mov %rbp,%rdi
0xffffffffaf9a5391 <__crash_kexec+81>: rep movsq %ds:(%rsi),%es:(%rdi)
0xffffffffaf9a5394 <__crash_kexec+84>: callq 0xffffffffaf9a4e00 <crash_save_vmcoreinfo>
0xffffffffaf9a5399 <__crash_kexec+89>: mov %rbp,%rdi
0xffffffffaf9a539c <__crash_kexec+92>: callq 0xffffffffaf857950 <machine_crash_shutdown>
0xffffffffaf9a53a1 <__crash_kexec+97>: mov 0x2401e50(%rip),%rdi # 0xffffffffb1da71f8
0xffffffffaf9a53a8 <__crash_kexec+104>: callq 0xffffffffaf864e90 <machine_kexec>
0xffffffffaf9a53ad <__crash_kexec+109>: mov $0xffffffffb1435e40,%rdi
crash> kmem -i
PAGES TOTAL PERCENTAGE
TOTAL MEM 955328 3.6 GB ----
FREE 849606 3.2 GB 88% of TOTAL MEM
USED 105722 413 MB 11% of TOTAL MEM
SHARED 8237 32.2 MB 0% of TOTAL MEM
BUFFERS 0 0 0% of TOTAL MEM
CACHED 42874 167.5 MB 4% of TOTAL MEM
SLAB 11013 43 MB 1% of TOTAL MEM
TOTAL HUGE 0 0 ----
HUGE FREE 0 0 0% of TOTAL HUGE
TOTAL SWAP 262143 1024 MB ----
SWAP USED 21069 82.3 MB 8% of TOTAL SWAP
SWAP FREE 241074 941.7 MB 91% of TOTAL SWAP
COMMIT LIMIT 739807 2.8 GB ----
COMMITTED 107082 418.3 MB 14% of TOTAL LIMIT
[root@demo ~]# rpm -qa | grep sos
sos-4.2-20.el8_6.noarch
[root@demo ~]# sos report --allow-system-changes
[root@demo ~]# ls -ltr /var/tmp/
total 37188
-rw-------. 1 root root 18976176 ago 19 09:57 sosreport-demo-2022-08-19-wvihyks.tar.xz
-rw-r--r--. 1 root root 65 ago 19 09:57 sosreport-demo-2022-08-19-wvihyks.tar.xz.sha256
[root@demo ~]# dnf install http://people.redhat.com/rsawhill/rpms/latest-rsawaroha-release.rpm
[root@demo ~]# dnf install xsos rsar
[root@demo ~]# tar -Jxvf /var/tmp/sosreport-demo-2022-08-19-wvihyks.tar.xz -C /var/tmp/
[root@demo ~]# xsos -ay /var/tmp/sosreport-demo-2022-08-19-wvihyks
DMIDECODE
BIOS:
Vend: SeaBIOS
Vers: 1.11.0-2.el7
Date: 04/01/2014
BIOS Rev: 0.0
FW Rev:
System:
Mfr: Red Hat
Prod: KVM
Vers: RHEL-7.4.0 PC (Q35 + ICH9, 2009)
Ser: Not Specified
UUID: f459afe5-6d04-42ae-8b4d-7d81b94c5414
CPU:
2 of 2 CPU sockets populated, 1 cores/1 threads per CPU
2 total cores, 2 total threads
Mfr: Red Hat
Fam:
Freq: 2000 MHz
Vers: RHEL-7.4.0 PC (Q35 + ICH9, 2009)
Memory:
Total: 4096 MiB (4 GiB)
DIMMs: 1 of 1 populated
MaxCapacity: 4096 MiB (4 GiB / 0.00 TiB)
OS
Hostname: demo.example.local
Distro: [redhat-release] Red Hat Enterprise Linux release 8.6 (Ootpa)
[os-release] Red Hat Enterprise Linux 8.6 (Ootpa) 8.6 (Ootpa)
RHN: (missing)
RHSM: hostname = satellite.example.local
proxy_hostname =
YUM: 3 enabled plugins: debuginfo-install, product-id, subscription-manager
SELinux: enforcing (default enforcing)
...skipping...
[root@demo ~]# ls /var/tmp/sosreport-demo-2022-08-19-wvihyks
boot df etc installed-rpms lib lspci pmstat pstree run sos_reports uname var
chkconfig dmidecode free ip_addr lsmod mount proc root sos_commands sos_strings uptime version.txt
date environment hostname last lsof netstat ps route sos_logs sys usr vgdisplay
[root@demo ~]# ls /var/tmp/sosreport-demo-2022-08-19-wvihyks/sos_commands/
alternatives console dnf hardware kvm md numa processor scsi sysvipc yum
ata cron dracut host libraries memory pam python selinux targetcli
auditd crypto filesys i18n libvirt multipath pci redis services tuned
block date firewalld iscsi login networking pcp rpm subscription_manager vdo
boot dbus firewall_tables kdump logrotate networkmanager perl rpmostree sunrpc vhostmd
cgroups devicemapper gluster kernel logs nfs podman runc systemd virsh
cockpit devices grub2 keyutils lvm2 nis process sar systemtap xfs
[root@demo ~]# ls /var/tmp/sosreport-demo-2022-08-19-wvihyks/var/log/
anaconda dnf.librepo.log dnf.rpm.log kdump.log messages-20220609 redis secure-20220602 spooler
audit dnf.log firewalld maillog messages-20220810 rhsm secure-20220609 sssd
boot.log dnf.log.1 glusterfs messages messages-20220817 sa secure-20220810 tuned
cron dnf.log.2 insights-client messages-20220602 pcp secure secure-20220817
[root@demo ~]# tc qdisc show
qdisc noqueue 0: dev lo root refcnt 2
qdisc fq_codel 0: dev enp1s0 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64
tc qdisc add dev enp1s0 root netem delay 100ms
tc qdisc add dev enp1s0 root netem delay 100ms 50ms distribution normal
tc qdisc add dev enp1s0 root netem loss 30%
tc qdisc add dev enp1s0 root netem loss 30% 50%
tc qdisc add dev enp1s0 root duplicate 50%
tc qdisc add dev enp1s0 root netem corrupt 30%
tc qdisc add dev enp1s0 root netem rate 10Mbit
tc qdisc delete dev enp1s0 root
[root@demo ~]# rpm -qa | grep stress
stress-ng-0.13.10-1.el8.x86_64
[root@demo ~]# stress-ng -c 2 -l 100 --timeout 15
stress-ng: info: [118293] setting to a 15 second run per stressor
stress-ng: info: [118293] dispatching hogs: 2 cpu
stress-ng: info: [118293] successful run completed in 15.00s
[root@demo ~]# stress-ng -m 1 --vm-bytes 3.2G --vm-keep --timeout 15
stress-ng: info: [118346] setting to a 15 second run per stressor
stress-ng: info: [118346] dispatching hogs: 1 vm
stress-ng: info: [118346] successful run completed in 15.15s
[root@demo ~]# dd if=/dev/zero of=./teste bs=1k count=2000000
2000000+0 records in
2000000+0 records out
2048000000 bytes (2,0 GB, 1,9 GiB) copied, 21,1111 s, 97,0 MB/s
# fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=random_read_write.fio --bs=4k --iodepth=64 --size=5G --readwrite=randrw --rwmixread=75
test: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [m(1)][100.0%][r=80.1MiB/s,w=26.6MiB/s][r=20.5k,w=6814 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=15851: Fri Aug 19 16:09:52 2022
read: IOPS=50.7k, BW=198MiB/s (208MB/s)(3837MiB/19380msec)
bw ( KiB/s): min=69472, max=273984, per=100.00%, avg=205411.24, stdev=70600.54, samples=38
iops : min=17368, max=68496, avg=51352.76, stdev=17650.16, samples=38
write: IOPS=16.9k, BW=66.2MiB/s (69.4MB/s)(1283MiB/19380msec)
bw ( KiB/s): min=23840, max=91120, per=100.00%, avg=68651.13, stdev=23560.99, samples=38
iops : min= 5960, max=22780, avg=17162.76, stdev=5890.25, samples=38
cpu : usr=16.84%, sys=51.13%, ctx=189346, majf=0, minf=29
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=982350,328370,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=198MiB/s (208MB/s), 198MiB/s-198MiB/s (208MB/s-208MB/s), io=3837MiB (4024MB), run=19380-19380msec
WRITE: bw=66.2MiB/s (69.4MB/s), 66.2MiB/s-66.2MiB/s (69.4MB/s-69.4MB/s), io=1283MiB (1345MB), run=19380-19380msec
Disk stats (read/write):
dm-2: ios=978719/327335, merge=0/0, ticks=821461/96582, in_queue=919952, util=99.31%, aggrios=983671/338127, aggrmerge=0/0, aggrticks=834551/123768, aggrin_queue=960986, aggrutil=99.18%
dm-0: ios=983671/338127, merge=0/0, ticks=834551/123768, in_queue=960986, util=99.18%, aggrios=983670/338028, aggrmerge=1/99, aggrticks=816020/111708, aggrin_queue=927728, aggrutil=99.37%
nvme0n1: ios=983670/338028, merge=1/99, ticks=816020/111708, in_queue=927728, util=99.37%
[root@demo ~]# ping -c4 www.google.com.br
PING www.google.com.br (172.253.115.94) 56(84) bytes of data.
64 bytes from bg-in-f94.1e100.net (172.253.115.94): icmp_seq=1 ttl=99 time=128 ms
64 bytes from bg-in-f94.1e100.net (172.253.115.94): icmp_seq=2 ttl=99 time=124 ms
64 bytes from bg-in-f94.1e100.net (172.253.115.94): icmp_seq=3 ttl=99 time=134 ms
64 bytes from bg-in-f94.1e100.net (172.253.115.94): icmp_seq=4 ttl=99 time=123 ms
--- www.google.com.br ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3005ms
rtt min/avg/max/mdev = 122.845/127.274/133.620/4.190 ms
# ethtool enp11s0u1
Settings for enp11s0u1:
Supported ports: [ TP MII ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Half 1000baseT/Full
Supported pause frame use: No
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised pause frame use: Symmetric Receive-only
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Link partner advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Link partner advertised pause frame use: Symmetric Receive-only
Link partner advertised auto-negotiation: Yes
Link partner advertised FEC modes: Not reported
Speed: 1000Mb/s
Duplex: Full
Port: MII
PHYAD: 32
Transceiver: internal
Auto-negotiation: on
Supports Wake-on: pumbg
Wake-on: d
Current message level: 0x00007fff (32767)
drv probe link timer ifdown ifup rx_err tx_err tx_queued intr tx_done rx_status pktdata hw wol
Link detected: yes
# ethtool -k enp11s0u1
Features for enp11s0u1:
rx-checksumming: on
tx-checksumming: on
tx-checksum-ipv4: on
tx-checksum-ip-generic: off [fixed]
tx-checksum-ipv6: on
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: off [fixed]
scatter-gather: on
tx-scatter-gather: on
tx-scatter-gather-fraglist: on
tcp-segmentation-offload: on
tx-tcp-segmentation: on
tx-tcp-ecn-segmentation: off [fixed]
tx-tcp6-segmentation: on
tx-tcp-mangleid-segmentation: off
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: off [fixed]
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-ipip-segmentation: off [fixed]
tx-sit-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
busy-poll: off [fixed]
tx-gre-csum-segmentation: off [fixed]
tx-udp_tnl-csum-segmentation: off [fixed]
tx-gso-partial: off [fixed]
tx-sctp-segmentation: off [fixed]
rx-gro-hw: off [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: off [fixed]
rx-udp_tunnel-port-offload: off [fixed]
[root@demo ~]# netstat -s
Ip:
Forwarding: 2
3109429 total packets received
163 with invalid addresses
0 forwarded
0 incoming packets discarded
3103122 incoming packets delivered
3737699 requests sent out
1 outgoing packets dropped
178 dropped because of missing route
385820 fragments received ok
8873860 fragments created
Icmp:
111 ICMP messages received
0 input ICMP message failed
ICMP input histogram:
echo requests: 49
echo replies: 62
182 ICMP messages sent
0 ICMP messages failed
ICMP output histogram:
destination unreachable: 3
echo requests: 130
echo replies: 49
IcmpMsg:
InType0: 62
InType8: 49
OutType0: 49
OutType3: 3
OutType8: 130
Tcp:
1740 active connection openings
73 passive connection openings
0 failed connection attempts
17 connection resets received
10 connections established
2888465 segments received
106828701 segments sent out
282 segments retransmitted
1 bad segments received
3427 resets sent
Udp:
214543 packets received
2 packets to unknown port received
0 packet receive errors
600373 packets sent
0 receive buffer errors
1 send buffer errors
UdpLite:
TcpExt:
56 TCP sockets finished time wait in fast timer
1 packets rejected in established connections because of timestamp
11685 delayed acks sent
14 delayed acks further delayed because of locked socket
Quick ack mode was activated 62 times
288274 packet headers predicted
216120 acknowledgments not containing data payload received
1938402 predicted acknowledgments
TCPSackRecovery: 31
TCPDSACKUndo: 2
7 congestion windows recovered without slow start after partial ack
TCPLostRetransmit: 52
4 timeouts in loss state
51 fast retransmits
38 retransmits in slow start
TCPTimeouts: 36
TCPLossProbes: 147
TCPLossProbeRecovery: 29
TCPSackRecoveryFail: 9
TCPBacklogCoalesce: 600218
TCPDSACKOldSent: 62
TCPDSACKRecv: 56
1671 connections reset due to unexpected data
4 connections reset due to early user close
1 connections aborted due to timeout
6 times unable to send RST due to no memory
TCPDSACKIgnoredNoUndo: 22
TCPSackMerged: 20
TCPSackShiftFallback: 44
IPReversePathFilter: 57
TCPRcvCoalesce: 16406
TCPOFOQueue: 416
TCPChallengeACK: 1
TCPSYNChallenge: 1
TCPSpuriousRtxHostQueues: 9
TCPAutoCorking: 300823
TCPFromZeroWindowAdv: 205
TCPToZeroWindowAdv: 205
TCPWantZeroWindowAdv: 271
TCPSynRetrans: 7
TCPOrigDataSent: 106710996
TCPHystartTrainDetect: 41
TCPHystartTrainCwnd: 3488
TCPHystartDelayDetect: 1
TCPHystartDelayCwnd: 29
TCPKeepAlive: 8329
TCPDelivered: 106708303
TCPAckCompressed: 403
IpExt:
InBcastPkts: 10
InOctets: 1024918700
OutOctets: 167146767096
InBcastOctets: 3196
InNoECTPkts: 3109515
# iperf3 -s -p 19766
-----------------------------------------------------------
Server listening on 19766
-----------------------------------------------------------
[root@demo ~]# iperf3 -c 192.168.1.1 -p 19766
Connecting to host 192.168.1.1, port 19766
[ 5] local 192.168.1.27 port 55096 connected to 192.168.1.1 port 19766
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 2.95 GBytes 25.3 Gbits/sec 0 1.42 MBytes
[ 5] 1.00-2.00 sec 2.78 GBytes 23.9 Gbits/sec 0 2.51 MBytes
[ 5] 2.00-3.00 sec 3.07 GBytes 26.4 Gbits/sec 0 2.51 MBytes
[ 5] 3.00-4.00 sec 3.27 GBytes 28.1 Gbits/sec 0 2.76 MBytes
[ 5] 4.00-5.00 sec 3.03 GBytes 26.0 Gbits/sec 0 2.76 MBytes
[ 5] 5.00-6.00 sec 3.24 GBytes 27.9 Gbits/sec 0 2.76 MBytes
[ 5] 6.00-7.00 sec 3.02 GBytes 26.0 Gbits/sec 0 3.04 MBytes
[ 5] 7.00-8.00 sec 3.21 GBytes 27.6 Gbits/sec 0 3.04 MBytes
[ 5] 8.00-9.00 sec 2.91 GBytes 25.0 Gbits/sec 0 3.04 MBytes
[ 5] 9.00-10.00 sec 3.01 GBytes 25.9 Gbits/sec 0 3.04 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 30.5 GBytes 26.2 Gbits/sec 0 sender
[ 5] 0.00-10.00 sec 30.5 GBytes 26.2 Gbits/sec receiver
iperf Done.
# qperf
[root@demo ~]# qperf -v 192.168.1.1 --ip_port 19766 --time 5 tcp_bw tcp_lat udp_bw udp_lat
tcp_bw:
bw = 3.49 GB/sec
msg_rate = 53.2 K/sec
port = 19,766
time = 5 sec
send_cost = 238 ms/GB
recv_cost = 1.29 sec/GB
send_cpus_used = 83 % cpus
recv_cpus_used = 450 % cpus
tcp_lat:
latency = 20.6 us
msg_rate = 48.6 K/sec
port = 19,766
time = 5 sec
loc_cpus_used = 48.8 % cpus
rem_cpus_used = 346 % cpus
udp_bw:
send_bw = 742 MB/sec
recv_bw = 727 MB/sec
msg_rate = 22.2 K/sec
port = 19,766
time = 5 sec
send_cost = 933 ms/GB
recv_cost = 5.25 sec/GB
send_cpus_used = 69.2 % cpus
recv_cpus_used = 382 % cpus
udp_lat:
latency = 22 us
msg_rate = 45.4 K/sec
port = 19,766
time = 5 sec
loc_cpus_used = 43.6 % cpus
rem_cpus_used = 397 % cpus
[root@demo ~]# tcpdump -i enp1s0 port not 22 and port not 3260 and host 192.168.1.1 -vv
dropped privs to tcpdump
tcpdump: listening on enp1s0, link-type EN10MB (Ethernet), capture size 262144 bytes
15:55:33.622720 IP (tos 0x0, ttl 64, id 20676, offset 0, flags [DF], proto ICMP (1), length 84)
_gateway > demo: ICMP echo request, id 12430, seq 1, length 64
15:55:33.622856 IP (tos 0x0, ttl 64, id 58321, offset 0, flags [none], proto ICMP (1), length 84)
demo > _gateway: ICMP echo reply, id 12430, seq 1, length 64
15:55:33.623766 IP (tos 0x0, ttl 64, id 64332, offset 0, flags [DF], proto UDP (17), length 71)
demo.39409 > _gateway.domain: [bad udp cksum 0x83b1 -> 0x554d!] 2282+ PTR? 27.1.168.192.in-addr.arpa. (43)
15:55:33.623883 IP (tos 0x0, ttl 64, id 35667, offset 0, flags [DF], proto UDP (17), length 89)
_gateway.domain > demo.39409: [bad udp cksum 0x83c3 -> 0xdf0e!] 2282* q: PTR? 27.1.168.192.in-addr.arpa. 1/0/0 27.1.168.192.in-addr.arpa. PTR demo. (61)
15:55:33.624039 IP (tos 0x0, ttl 64, id 64333, offset 0, flags [DF], proto UDP (17), length 70)
demo.56391 > _gateway.domain: [bad udp cksum 0x83b0 -> 0xba92!] 22674+ PTR? 1.1.168.192.in-addr.arpa. (42)
15:55:33.773438 IP (tos 0x0, ttl 64, id 35753, offset 0, flags [DF], proto UDP (17), length 70)
_gateway.domain > demo.56391: [bad udp cksum 0x83b0 -> 0x3a10!] 22674 ServFail q: PTR? 1.1.168.192.in-addr.arpa. 0/0/0 (42)
# ping -c1 demo.example.local
PING demo (192.168.1.27) 56(84) bytes of data.
64 bytes from demo (192.168.1.27): icmp_seq=1 ttl=64 time=0.251 ms
--- demo ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.251/0.251/0.251/0.000 ms
[root@demo ~]# iptraf-ng
[root@demo ~]# perf stat ls
anaconda-ks.cfg demo-certs tshoot-tools
Performance counter stats for 'ls':
1,26 msec task-clock # 0,672 CPUs utilized
0 context-switches # 0,000 /sec
0 cpu-migrations # 0,000 /sec
95 page-faults # 75,210 K/sec
<not supported> cycles
<not supported> instructions
<not supported> branches
<not supported> branch-misses
0,001878325 seconds time elapsed
0,000172000 seconds user
0,001759000 seconds sys
[root@demo ~]# free -ht
total used free shared buff/cache available
Mem: 3,6Gi 395Mi 2,4Gi 5,0Mi 903Mi 3,0Gi
Swap: 1,0Gi 121Mi 902Mi
Total: 4,6Gi 516Mi 3,3Gi
[root@demo ~]# top
top - 17:55:31 up 2 days, 25 min, 2 users, load average: 0,01, 0,02, 0,03
Tasks: 186 total, 2 running, 184 sleeping, 0 stopped, 0 zombie
%Cpu0 : 0,3 us, 0,0 sy, 0,0 ni, 99,7 id, 0,0 wa, 0,0 hi, 0,0 si, 0,0 st
%Cpu1 : 0,0 us, 0,3 sy, 0,0 ni, 96,0 id, 3,3 wa, 0,3 hi, 0,0 si, 0,0 st
MiB Mem : 3731,8 total, 1419,7 free, 529,1 used, 1783,0 buff/cache
MiB Swap: 1024,0 total, 906,1 free, 117,9 used. 2887,7 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
128858 root 20 0 0 0 0 R 0,3 0,0 0:00.02 kworker/1:3-xfs-buf/dm-0
128859 root 20 0 275232 4704 4048 R 0,3 0,1 0:00.03 top
1 root 20 0 243712 13704 7496 S 0,0 0,4 0:24.68 systemd
2 root 20 0 0 0 0 S 0,0 0,0 0:00.15 kthreadd
3 root 0 -20 0 0 0 I 0,0 0,0 0:00.00 rcu_gp
...skipping...
[root@demo ~]# systemd-cgtop
Control Group Tasks %CPU Memory Input/s Output/s
/ 227 2.0 2.2G - -
/user.slice 12 1.7 1.5G - -
/system.slice 80 0.3 422.9M - -
/system.slice/tuned.service 5 0.2 15.7M - -
/system.slice/redis.service 4 0.1 146.0M - -
/system.slice/rhsm.service 2 0.0 24.5M - -
/system.slice/pmproxy.service 1 0.0 30.8M - -
/system.slice/multipathd.service 7 0.0 11.5M - -
/system.slice/NetworkManager.service 3 0.0 4.6M - -
/system.slice/rsyslog.service 3 0.0 2.9M - -
/system.slice/sssd.service 3 0.0 11.7M - -
/init.scope 1 - 26.4M - -
/system.slice/auditd.service 4 - 1.9M - -
/system.slice/boot.mount - - 4.0K - -
/system.slice/cockpit-wsinstance-http.socket - - 4.0K - -
/system.slice/cockpit-wsinstance-https-factory.socket - - 4.0K - -
/system.slice/cockpit.service 7 - 2.5M - -
/system.slice/cockpit.socket - - 60.0K - -
/system.slice/crond.service 1 - 1.1M - -
/system.slice/dbus.service 2 - 1.9M - -
/system.slice/dev-hugepages.mount - - 4.0K - -
/system.slice/dev-mapper-VG_01\x2dswap.swap - - 56.0K - -
/system.slice/dev-mqueue.mount - - 4.0K - -
/system.slice/firewalld.service 2 - 23.1M - -
/system.slice/gssproxy.service 6 - 1008.0K - -
/system.slice/irqbalance.service 2 - 920.0K - -
/system.slice/iscsid.service 1 - 13.4M - -
/system.slice/ksmtuned.service 2 - 6.3M - -
/system.slice/osbuild-composer.socket - - 4.0K - -
/system.slice/pmcd.service 6 - 12.9M - -
[root@demo ~]# iostat -cdyx 1 1
Linux 4.18.0-372.19.1.el8_6.x86_64 (demo.example.local) 19/08/2022 _x86_64_ (2 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
0,00 0,00 0,00 0,00 0,00 0,00
Device r/s w/s rkB/s wkB/s rrqm/s wrqm/s %rrqm %wrqm r_await w_await aqu-sz rareq-sz wareq-sz svctm %util
vda 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00
scd0 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00
dm-0 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00
dm-1 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00
sda 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00
sdb 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00
dm-2 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00
dm-3 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00
[root@demo ~]# mpstat -P ALL 1 1
Linux 4.18.0-372.19.1.el8_6.x86_64 (demo.example.local) 19/08/2022 _x86_64_ (2 CPU)
16:31:09 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
16:31:10 all 1,49 0,00 1,00 13,43 0,00 0,00 0,00 0,00 0,00 84,08
16:31:10 0 1,98 0,00 0,99 26,73 0,00 0,00 0,00 0,00 0,00 70,30
16:31:10 1 1,00 0,00 1,00 0,00 0,00 0,00 0,00 0,00 0,00 98,00
Average: CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
Average: all 1,49 0,00 1,00 13,43 0,00 0,00 0,00 0,00 0,00 84,08
Average: 0 1,98 0,00 0,99 26,73 0,00 0,00 0,00 0,00 0,00 70,30
Average: 1 1,00 0,00 1,00 0,00 0,00 0,00 0,00 0,00 0,00 98,00
[root@demo ~]# iotop
Total DISK READ : 0.00 B/s | Total DISK WRITE : 0.00 B/s
Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 0.00 B/s
TID PRIO USER DISK READ DISK WRITE> COMMAND
1 be/4 root 0.00 B/s 0.00 B/s systemd --switched-root --system --deserialize 18
2 be/4 root 0.00 B/s 0.00 B/s [kthreadd]
3 be/0 root 0.00 B/s 0.00 B/s [rcu_gp]
4 be/0 root 0.00 B/s 0.00 B/s [rcu_par_gp]
...skipping...
[root@demo ~]# iftop
12,5Kb 25,0Kb 37,5Kb 50,0Kb 62,5Kb
└────────────────────────────┴─────────────────────────────┴─────────────────────────────┴─────────────────────────────┴─────────────────────────────
demo => _gateway 1,59Kb 2,85Kb 2,85Kb
<= 608b 981b 981b
demo => dns 0b 56b 56b
<= 0b 102b 102
[root@demo ~]# dstat -c -C total -l -p -m -s -g -d -D total -r
----total-usage---- ---load-avg--- ---procs--- ------memory-usage----- ----swap--- ---paging-- -dsk/total- --io/total-
usr sys idl wai stl| 1m 5m 15m |run blk new| used free buf cach| used free| in out | read writ| read writ
|0.10 0.05 0.09|2.0 0 | 412M 2409M 1372k 868M| 120M 904M| | |
0 0 100 0 0|0.10 0.05 0.09|1.0 0 0| 412M 2409M 1372k 868M| 120M 904M| 0 0 | 0 40k| 0 3.00
0 0 100 0 0|0.10 0.05 0.09| 0 0 0| 412M 2409M 1372k 868M| 120M 904M| 0 0 | 0 0 | 0 1.50
0 0 99 0 0|0.10 0.05 0.09| 0 0 0| 412M 2409M 1372k 868M| 120M 904M| 0 0 | 0 0 | 0 0
0 0 99 0 0|0.10 0.05 0.09| 0 0 0| 412M 2409M 1372k 868M| 120M 904M| 0 0 | 0 0 | 0 0
2 2 98 0 0|0.25 0.08 0.10| 0 0 8.0| 412M 2409M 1372k 868M| 120M 904M| 0 0 | 0 0 | 0 0
...skipping...
[root@demo ~]# sar -BbdqrSuvW 1 1
Linux 4.18.0-372.19.1.el8_6.x86_64 (demo.example.local) 19/08/2022 _x86_64_ (2 CPU)
16:53:37 CPU %user %nice %system %iowait %steal %idle
16:53:38 all 0,00 0,00 0,50 0,00 0,00 99,50
16:53:37 pswpin/s pswpout/s
16:53:38 0,00 0,00
16:53:37 pgpgin/s pgpgout/s fault/s majflt/s pgfree/s pgscank/s pgscand/s pgsteal/s %vmeff
16:53:38 0,00 0,00 0,00 0,00 35,00 0,00 0,00 0,00 0,00
16:53:37 tps rtps wtps bread/s bwrtn/s
16:53:38 0,00 0,00 0,00 0,00 0,00
16:53:37 kbmemfree kbavail kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty
16:53:38 2476012 3150664 1345300 35,21 1372 889064 594740 12,21 663432 448084 44
16:53:37 kbswpfree kbswpused %swpused kbswpcad %swpcad
16:53:38 926068 122504 11,68 68400 55,83
16:53:37 dentunusd file-nr inode-nr pty-nr
16:53:38 17942 2784 38368 1
16:53:37 runq-sz plist-sz ldavg-1 ldavg-5 ldavg-15 blocked
16:53:38 1 226 0,03 0,03 0,06 0
16:53:37 DEV tps rkB/s wkB/s areq-sz aqu-sz await svctm %util
16:53:38 dev252-0 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00
16:53:38 dev11-0 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00
16:53:38 dev253-0 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00
16:53:38 dev253-1 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00
16:53:38 dev8-0 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00
16:53:38 dev8-16 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00
16:53:38 dev253-2 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00
16:53:38 dev253-3 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00
Average: CPU %user %nice %system %iowait %steal %idle
Average: all 0,00 0,00 0,50 0,00 0,00 99,50
Average: pswpin/s pswpout/s
Average: 0,00 0,00
Average: pgpgin/s pgpgout/s fault/s majflt/s pgfree/s pgscank/s pgscand/s pgsteal/s %vmeff
Average: 0,00 0,00 0,00 0,00 35,00 0,00 0,00 0,00 0,00
Average: tps rtps wtps bread/s bwrtn/s
Average: 0,00 0,00 0,00 0,00 0,00
Average: kbmemfree kbavail kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty
Average: 2476012 3150664 1345300 35,21 1372 889064 594740 12,21 663432 448084 44
Average: kbswpfree kbswpused %swpused kbswpcad %swpcad
Average: 926068 122504 11,68 68400 55,83
Average: dentunusd file-nr inode-nr pty-nr
Average: 17942 2784 38368 1
Average: runq-sz plist-sz ldavg-1 ldavg-5 ldavg-15 blocked
Average: 1 226 0,03 0,03 0,06 0
Average: DEV tps rkB/s wkB/s areq-sz aqu-sz await svctm %util
Average: dev252-0 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00
Average: dev11-0 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00
Average: dev253-0 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00
Average: dev253-1 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00
Average: dev8-0 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00
Average: dev8-16 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00
Average: dev253-2 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00
Average: dev253-3 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00
[root@demo ~]# cifsiostat 1 3
Linux 4.18.0-372.19.1.el8_6.x86_64 (demo.example.local) 19/08/2022 _x86_64_ (2 CPU)
Filesystem rB/s wB/s rops/s wops/s fo/s fc/s fd/s
Filesystem rB/s wB/s rops/s wops/s fo/s fc/s fd/s
Filesystem rB/s wB/s rops/s wops/s fo/s fc/s fd/s
[root@demo ~]# pidstat
Linux 4.18.0-372.19.1.el8_6.x86_64 (demo.example.local) 19/08/2022 _x86_64_ (2 CPU)
16:55:55 UID PID %usr %system %guest %wait %CPU CPU Command
16:55:55 0 1 0,00 0,01 0,00 0,00 0,01 0 systemd
16:55:55 0 2 0,00 0,00 0,00 0,00 0,00 0 kthreadd
16:55:55 0 12 0,00 0,00 0,00 0,01 0,00 0 ksoftirqd/0
16:55:55 0 13 0,00 0,02 0,00 0,13 0,02 0 rcu_sched
16:55:55 0 14 0,00 0,00 0,00 0,00 0,00 0 migration/0
16:55:55 0 15 0,00 0,00 0,00 0,00 0,00 0 watchdog/0
16:55:55 0 18 0,00 0,00 0,00 0,00 0,00 1 watchdog/1
16:55:55 0 19 0,00 0,00 0,00 0,00 0,00 1 migration/1
16:55:55 0 20 0,00 0,00 0,00 0,01 0,00 1 ksoftirqd/1
16:55:55 0 27 0,00 0,00 0,00 0,00 0,00 0 kauditd
16:55:55 0 28 0,00 0,00 0,00 0,00 0,00 0 khungtaskd
16:55:55 0 29 0,00 0,00 0,00 0,00 0,00 1 oom_reaper
16:55:55 0 31 0,00 0,00 0,00 0,00 0,00 1 kcompactd0
16:55:55 0 33 0,00 0,01 0,00 0,00 0,01 1 khugepaged
16:55:55 0 42 0,00 0,00 0,00 0,00 0,00 0 kworker/0:1H-kblockd
16:55:55 0 76 0,00 0,00 0,00 0,00 0,00 1 kswapd0
16:55:55 0 192 0,00 0,00 0,00 0,00 0,00 1 irq/30-pciehp
16:55:55 0 314 0,00 0,00 0,00 0,00 0,00 1 kworker/1:1H-kblockd
16:55:55 0 492 0,00 0,00 0,00 0,00 0,00 0 scsi_eh_0
16:55:55 0 629 0,00 0,01 0,00 0,02 0,01 1 xfsaild/dm-0
16:55:55 0 725 0,00 0,00 0,00 0,00 0,00 0 systemd-journal
16:55:55 0 765 0,00 0,00 0,00 0,00 0,00 0 systemd-udevd
16:55:55 32 894 0,00 0,00 0,00 0,00 0,00 1 rpcbind
16:55:55 0 897 0,00 0,00 0,00 0,00 0,00 0 auditd
16:55:55 0 899 0,00 0,00 0,00 0,00 0,00 0 sedispatch
16:55:55 0 939 0,00 0,00 0,00 0,00 0,00 1 sssd
16:55:55 0 940 0,00 0,00 0,00 0,00 0,00 1 smartd
16:55:55 998 942 0,00 0,00 0,00 0,00 0,00 1 polkitd
16:55:55 81 943 0,00 0,00 0,00 0,01 0,00 0 dbus-daemon
16:55:55 0 947 0,00 0,00 0,00 0,00 0,00 0 systemd-machine
16:55:55 0 949 0,00 0,00 0,00 0,00 0,00 1 irqbalance
16:55:55 0 966 0,00 0,00 0,00 0,00 0,00 1 ksmtuned
16:55:55 0 972 0,00 0,00 0,00 0,00 0,00 1 firewalld
16:55:55 0 979 0,00 0,00 0,00 0,00 0,00 1 NetworkManager
16:55:55 0 988 0,00 0,00 0,00 0,00 0,00 0 sshd
16:55:55 0 989 0,14 0,06 0,00 0,02 0,21 0 tuned
16:55:55 979 995 0,08 0,08 0,00 0,01 0,15 1 redis-server
16:55:55 0 1016 0,00 0,00 0,00 0,00 0,00 1 sssd_be
16:55:55 0 1037 0,00 0,00 0,00 0,00 0,00 0 sssd_nss
16:55:55 0 1111 0,00 0,00 0,00 0,00 0,00 0 systemd-logind
16:55:55 0 1243 0,00 0,00 0,00 0,00 0,00 0 rsyslogd
16:55:55 0 1253 0,00 0,00 0,00 0,00 0,00 1 iscsid
16:55:55 0 1339 0,00 0,00 0,00 0,00 0,00 1 crond
16:55:55 980 1532 0,00 0,00 0,00 0,00 0,00 1 pmcd
16:55:55 0 1547 0,00 0,00 0,00 0,00 0,00 1 pmdaroot
16:55:55 0 1556 0,00 0,01 0,00 0,00 0,01 1 pmdaproc
16:55:55 0 1561 0,00 0,00 0,00 0,00 0,00 1 pmdaxfs
16:55:55 0 1564 0,00 0,00 0,00 0,00 0,01 1 pmdalinux
16:55:55 0 1575 0,00 0,00 0,00 0,00 0,00 1 pmdakvm
16:55:55 980 1580 0,02 0,00 0,00 0,00 0,02 0 pmproxy
16:55:55 980 2019 0,00 0,00 0,00 0,00 0,00 0 pmlogger
16:55:55 0 2379 0,00 0,00 0,00 0,00 0,00 0 systemd
16:55:55 985 2987 0,00 0,00 0,00 0,00 0,00 0 cockpit-tls
16:55:55 0 3029 0,00 0,00 0,00 0,00 0,00 1 dbus-daemon
16:55:55 0 4592 0,01 0,00 0,00 0,00 0,02 1 rhsm-service
16:55:55 0 16234 0,00 0,00 0,00 0,00 0,00 1 agetty
16:55:55 0 66652 0,00 0,00 0,00 0,00 0,00 0 multipathd
16:55:55 0 117704 0,00 0,00 0,00 0,00 0,00 1 snmpd
16:55:55 0 118314 0,00 0,00 0,00 0,00 0,00 0 sshd
16:55:55 0 118317 0,00 0,00 0,00 0,00 0,00 0 sshd
16:55:55 0 118318 0,00 0,00 0,00 0,00 0,00 0 bash
16:55:55 0 119672 0,00 0,00 0,00 0,00 0,00 1 kworker/u4:0-events_unbound
16:55:55 0 119701 0,00 0,00 0,00 0,00 0,00 0 kworker/0:0-cgroup_pidlist_destroy
16:55:55 0 119749 0,00 0,00 0,00 0,00 0,00 1 kworker/1:0-events_power_efficient
16:55:55 0 119783 0,00 0,00 0,00 0,00 0,00 0 kworker/0:2-cgroup_pidlist_destroy
16:55:55 0 119835 0,00 0,00 0,00 0,00 0,00 0 kworker/0:1-cgroup_destroy
16:55:55 0 120250 0,00 0,00 0,00 0,00 0,00 1 kworker/1:1-events
16:55:55 0 120347 0,00 0,00 0,00 0,00 0,00 1 pidstat
[root@demo ~]# for i in $(pidof sshd); do pidstat -p $i; done
Linux 4.18.0-372.19.1.el8_6.x86_64 (demo.example.local) 19/08/2022 _x86_64_ (2 CPU)
16:59:34 UID PID %usr %system %guest %wait %CPU CPU Command
16:59:34 0 120378 0,00 0,00 0,00 0,00 0,00 1 sshd
Linux 4.18.0-372.19.1.el8_6.x86_64 (demo.example.local) 19/08/2022 _x86_64_ (2 CPU)
16:59:34 UID PID %usr %system %guest %wait %CPU CPU Command
16:59:34 0 120375 0,00 0,00 0,00 0,00 0,00 0 sshd
Linux 4.18.0-372.19.1.el8_6.x86_64 (demo.example.local) 19/08/2022 _x86_64_ (2 CPU)
16:59:34 UID PID %usr %system %guest %wait %CPU CPU Command
16:59:34 0 118317 0,00 0,00 0,00 0,00 0,00 1 sshd
Linux 4.18.0-372.19.1.el8_6.x86_64 (demo.example.local) 19/08/2022 _x86_64_ (2 CPU)
16:59:34 UID PID %usr %system %guest %wait %CPU CPU Command
16:59:34 0 118314 0,00 0,00 0,00 0,00 0,00 0 sshd
Linux 4.18.0-372.19.1.el8_6.x86_64 (demo.example.local) 19/08/2022 _x86_64_ (2 CPU)
16:59:34 UID PID %usr %system %guest %wait %CPU CPU Command
16:59:34 0 988 0,00 0,00 0,00 0,00 0,00 0 sshd
[root@demo ~]# ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.3 243712 13636 ? Ss ago17 0:21 /usr/lib/systemd/systemd --switched-root --system --deserialize 18
root 2 0.0 0.0 0 0 ? S ago17 0:00 [kthreadd]
root 3 0.0 0.0 0 0 ? I< ago17 0:00 [rcu_gp]
root 4 0.0 0.0 0 0 ? I< ago17 0:00 [rcu_par_gp]
root 6 0.0 0.0 0 0 ? I< ago17 0:00 [kworker/0:0H-events_highpri]
root 9 0.0 0.0 0 0 ? I< ago17 0:00 [mm_percpu_wq]
...skipping...
[root@demo ~]# /usr/share/bcc/tools/biolatency
Tracing block device I/O... Hit Ctrl-C to end.
^C
usecs : count distribution
0 -> 1 : 0 | |
2 -> 3 : 0 | |
4 -> 7 : 0 | |
8 -> 15 : 0 | |
16 -> 31 : 0 | |
32 -> 63 : 0 | |
64 -> 127 : 3 |********** |
128 -> 255 : 12 |****************************************|
256 -> 511 : 3 |********** |
512 -> 1023 : 4 |************* |
1024 -> 2047 : 7 |*********************** |
2048 -> 4095 : 0 | |
4096 -> 8191 : 0 | |
8192 -> 16383 : 0 | |
16384 -> 32767 : 0 | |
32768 -> 65535 : 0 | |
65536 -> 131071 : 0 | |
131072 -> 262143 : 0 | |
262144 -> 524287 : 1 |*** |
[root@demo ~]# /usr/share/bcc/tools/cachestat -T
TIME HITS MISSES DIRTIES HITRATIO BUFFERS_MB CACHED_MB
17:03:48 0 0 0 0.00% 6 1454
17:03:50 1 0 0 100.00% 6 1454
17:03:51 0 0 0 0.00% 6 1454
17:03:52 0 0 0 0.00% 6 1454
17:03:53 0 0 0 0.00% 6 1454
17:03:54 0 0 0 0.00% 6 1454
17:03:55 0 0 0 0.00% 6 1454
17:03:56 0 0 0 0.00% 6 1454
17:03:57 0 0 0 0.00% 6 1454
17:03:58 0 0 0 0.00% 6 1454
17:03:59 0 0 0 0.00% 6 1454
17:04:00 0 0 0 0.00% 6 1454
17:04:01 0 0 0 0.00% 6 1454
17:04:02 0 0 0 0.00% 6 1454
17:04:03 0 0 0 0.00% 6 1454
^C17:04:04 0 0 0 0.00% 6 1454
Detaching...