LinuxKI 7.9

LinuxKI version 7.9 is now available (04/19/2024)

This version of LinuxKI has supports Linux Kernel versions through 6.5.13 and RHEL 9.3 as well as new ioctl() commands for Device Multipath (DM), SG_IO (SCSI IO), and KVM. For example:

0.000371 cpu=48 pid=107507 tgid=107507 ioctl [16] ret=0x1 syscallbeg= 0.000008 fd=17 request=KVM_SIGNAL_MSI/0x4020aea5 arg2=0x7ffc360c7e00 arg3=0xfee02
0.003722 cpu=57 pid=3133768 tgid=0 ioctl [16] ret=0x0 syscallbeg= 0.001332 fd=19 request=SG_IO/0x2285 arg2=0x7f9ed2ffc180 arg3=0x7f9ef46c1fd3
0.080747 cpu=16 pid=26751 tgid=26732 ioctl [16] ret=0x0 syscallbeg= 0.000052 fd=5 request=DM_TABLE_STATUS/0xc138fd0c arg2=0x7fbe841f2730 arg3=0x0

Other fixes and enhancements include:

• Added warning to runki script is -U option is used but perf is not found in /usr/bin/perf
• Fix coredump from parse_irqlist() due to long input lines
• Collect kallsyms early so symbols are available if the trace gets aborted or system crashes.
• Fix CPU Freq when base frequency is not available in cpuinfo or lscpu. Now, use dmidecode to get the base frequency
• Fixes system call filenames when lsof output has TASKCMD column
• Added SR932i controller for CACHE_BYPASS waring
• Fixed missing per-PID CPU time due to IRQ events prior to first hardclock event
• Fixed random/sequential io counters for disk IO when block_rq_insert events are not found
• Expanded Avwait and Avserv for better precision, help for for low latency IO as fields are in milliseconds and many drives can complete IO un under 100 microseconds.
• WinKI - Enhance HardIRQ/SoftIRQ names to include the driver filename along with the hex address if the symbol table cannot be found
• WinKI - Include the following PDB files for processing - NDIS.SYS, CLASSPNP.SYS, WdFilter.sys, dxgkrnl.sys, vmswitch.sys, vmkbkmclr.sys, vmbusr.sys
• WinKI - Added ReadyThread stack traces on Kipid reports
• WinKI - Fixed SQL Thread names if SQLThreadList file had a NULL instance name
• Decode NTSTATUS return code for Windows SysClEvents

LinuxKI 7.8

LinuxKI version 7.8 is now available (11/09/23)

This version of LinuxKI has been enhanced to be more container friendly with non-Docker container environments and also provides additional details on NVIDIA GPU ioctl() calls to help troubleshoot issues on servers using GPUs. For example, the KI ASCII trace files will decode the ioctl() command:

0.004007 cpu=39 pid=6166 tgid=6166 ioctl [16] entry fd=4 request=NVIDIA_IOCTL_CALL/0xc020462a arg2=0x7ffe142a6d60 arg3=0x493975e8

Below is a list of changes and fixes in the 7.8 release:

• Enhanced LinuxKI to be more Container-aware, not just Docker-aware, by using the filenames from /var/log/pods to get container names for K8s pods.
• For NVIDIA ioctl() calls, print the ioctl command name used along with the hex value.
• For ARM64 traces, skip IRQ hardclock functions in the stack trace
• Have runki look for likit.ko, in /opt/linuxki since /usr is not writable. This is to support LinuxKI in Openshift environments
• Print c-state name along with cstate number in kirunq and kparse files
• Fix coredump when running kiinfo (kiall script) due to truncated ki.bin files with KI dumps created with ftrace (runki -f)
• Fix coredump when formatting traces due to unitialized varaiable in hc_print_stktrc()
• Fixed incorrect library symbol names in symlookup() when library file has non-zero p_off field
• Fixed Multipath device name in kidsk/kipid/kparse output due to variable format of the multipath -l output

LinuxKI 7.7

LinuxKI version 7.7 is now available (07/29/2023)

LinuxKI has been enhanced to work with Linux kernel versions through 6.4.3 kernels, RHEL 9.2, and SLES 15 SP5.

• Enable Advanced CPU metrics for Intel SapphireRapids processors
• Improved ARM hardclock traces
• Check for SecureBoot during LinuxKI install and print warning message if enabled
• Enhance runki script to collect some miscellaneous parameters, such as tracing/tracing_on, sched_features, and transparent_hugepages.
• More reliable ftrace dumps by draining trace_pipe before a new collection
• Kparse enhancements
  o Add Kparse warning to detect large number of CPU cores in a NUMA node
  o Expand Kparse Warning for NUMA page migrations to look at CPU profile data
  o Collect lscpu output with runki script and parse to get system information for more accurate NUMA node reporting in Kparse and Kirunq reports
  o Get Product Name and Vendor from dmidecode and display in Kparse report
• Bug Fixes:
  o Fixed LinuxKI for ARM and Ubuntu kernels above 5.2
  o Fixed Hardclock output showing UNKNOWN SYS by keeping track of UNKNOWN USER and UNKNOWN SYS events
  o Fixed ftrace tracing if ncpus is > 512
  o Fixed ftrace tracing if hotplug vCPUs are used in VM environment
  o Fixed Bad HBA names with live tracing
  o Fixed bad Global File CPU metrics with kiinfo -live in real-time mode

LinuxKI 7.6

LinuxKI version 7.6 is now available (02/24/2023)

LinuxKI version 7.6 has been enhanced to work with Linux kernel versions through 6.1.12 kernels as well as RHEL 9.1. Below are other fixes and enhancements in this version:

• Enabled Advance CPU metrics for IceLake processors
• Removed Visualization capability due to security and maintenance concerns
• Bug fixes
  o Process irqlist file to get irq names longer than 16 characters
  o Fixed PPID for threads cloned during the trace
  o Remove collecting /proc/slabinfo from runki
• RunQ Enhancements
  o Added MaxRqTm stat to Top PIDS waiting for CPU in kirunq and kparse report.
  o Included RunQ histogram in kipid report by default
  o Added Kparse warning for RunQ delay detection
• WinKI enhancements
  o Add spinlock information to WinKI reports (kparse, kipid)
  o Fixed Process DCStart event to show the executable and command line arguments
  o Added several events previously shown as “unknown”

LinuxKI 7.5

LinuxKI version 7.5 is now available (10/14/2022).

LinuxKI version 7.5 has been enhanced to work with Linux kernel versions through 5.19.15 kernels. LinuxKI has also been tested on the RL300 server (ARM64) running Redhat 8.6. Other versions of Linux on the RL300 are likely to work, but have not been tested.

Also, LinuxKI for Windows has been enhanced to provide thread WaitReason to the Sleep and System Call report. For example:

Kernel Functions calling sleep() - Top 20 Functions
   Count     Pct    SlpTime    Slp% TotalTime%   Msec/Slp   MaxMsecs  Func
    2679  98.93%    19.7960  99.22%     93.77%      7.389    508.916  KeWaitForSingleObject
     306  11.30%    19.6285  98.39%     92.98%     64.146    508.916      WrUserRequest
    2305  85.12%     0.1205   0.60%      0.57%      0.052      0.783      WrResource
       1   0.04%     0.0260   0.13%      0.12%     26.032     26.032      Unknown
      13   0.48%     0.0157   0.08%      0.07%      1.211      6.844      UserRequest
      51   1.88%     0.0046   0.02%      0.02%      0.090      1.942      WrLpcReply
       2   0.07%     0.0006   0.00%      0.00%      0.288      0.490      Executive
       1   0.04%     0.0000   0.00%      0.00%      0.005      0.005      WrPushLock

Other fixes included in 7.5 include:

• Fixed MaxMses in Windows PID Sleep Report
• Fixed Windows "Process Start" events
• Filtered out EtwTraceContextSwap function from CSwitch processing
• Fixed duplicate stack traces in Linux Sleep Reports in kipid output

LinuxKI 7.4

LinuxKI version 7.4 is now available (05/26/22).

LinuxKI version 7.4 has been enhanced to work with RHEL 8.6 and RHEL 9.0. It also includes the following minor fixes and enhancements:

• Fixed runki script to capture all devices in the block_params file
• Fixed runki script to properly collect the network bond configuration
• Fixed runki script to remove the tcpdump files after archiving the ki_all.*.tgz file
• enhanced the error message for the sched_setscheduler() error to provide additional information
• Enhance LiKI DLKM and kiinfo to collect the call_function traces (runki -e all), which can help with low-latency debugging
• Enhance kiinfo for ftrace data collection (runki -f) to collect per-cpu trace data on all cores when kiinfo is affinitized to a subset of cores.
• Enhance WinKI data collection (runki.cmd) to collect additional SQL statistics, such as spinlock and wait statistics

LinuxKI 7.3

LinuxKI version 7.3 is now available (12/21/21).

LinuxKI version 7.3 includes changes to support Linux Kernels through 5.14.21.

LinuxKI 7.2

LinuxKI version 7.2 is now available (11/09/21).

LinuxKI version 7.2 has a few minor fixes and enhancements:

• Reduced Windows overhead by eliminating Stack Traces to CSwitch, ReadyThread, and SampleProfile events.
• Increased Windows trace size to 2GB.
• Added .NET PDB (clr.pdb) to list of PDBs collected during Windows trace analysis
• Fix issue with nested system calls in Windows traces which mess up the system call stats. Now, outer nested system calls times will be 0.
• Enabled FileIo FileRundown trace events and increased the file object hash size to improve filename lookups.
• Fixed issue in runki script where only first FC interface is listed in the fc_linkspeed file
• Collect per-NUMA node meminfo data into the numa_meminfo file.
• Print warning when sysignore file is used with kiinfo when doing trace dump analysis. The sysignore file only ignores the system calls during the actual tracing of the events.
• Updated link on Windows TCP Timeout issue
• Fix runki script to suppress errors when collecting block_param data

LinuxKI version 7.1-1

LinuxKI version 7.1 is now available (07/09/2021).

LinuxKI verison 7.1 adds a number of new enhancements for the WinKI feature, including:

• File Activity Report (kifile.*.txt) as well as file information in kparse, kipid, and the curses-based kiinfo -live
• Socket Acitivity Report (kisock.*.txt) as well as network socket information in kparse, kipid, and the curses-based kiinfo -live
• Added SQL thread names in the KI report
• Decoded thread wait_reason and mode in KI ASCII trace events
• Added CSV files for Windows data
• Added new Kparse Warnings to detect the following issues:
• Windows TCP Timeouts
• excessive CPU usage with SQL auto statistics

LinuxKI version 7.1 also added some new enhancements and bug fixes with the Linux features:

• Added IO Controllers section in Kparse report
• Added Memory DIMM section in Kparse report
• Added new Kparse warnings to detect the following issues:
  • KVM host page faults
  • Max_sectors_kbs settings for HPE Smart Array Controllers
  • Excessive CPU usage with Oracle column stats
  • Low memory on a NUMA node
  • Memory imbalance across NUMA nodes
• Bug fixes
  • Fixed kiinfo Makefile to get better stack traces for Oracle Enterprise Linux (OEL)
  • Fixed LiKI to get IP addresses after Linux version 4.17.0 due to Linux changes with getname() function
  • fixed coredump when using runki -I due to bug introduced in version 7.0
  • Fixed runki script to append to fc_linkspeed file
  • Fixed bug with kiall -f if you have odd filenames with bash option prefixes in them, such as "--file"
  • Fixed many issues with the Visualizations

For more information, please refer to the LinuxKI MasterClass.

LinuxKI version 7.0-1

LinuxKI for Windows? Absolutely! LinuxKI version 7.0 is now available (03/12/2021).

With version 7.0, LinuxKI can now read Windows ETL trace files, bringing the power and rich features of LinuxKI to analyze Windows ETL trace data the same way LinuxKI data is analyzed. For more information on collecting and analyzing Windows ETL trace files, see Section 12 of the LinuxKI MasterClass.

The following fixes and enhancements are included in this release:

• Ability to read Windows ETL trace files
• Support for Linux kernels through 5.9.16
• Fixed kiinfo coredump when analyzing KI data from LinuxKI ftrace dumps from Linux 5.0 kernels and later
• Fixed select() fds_bytes on syscall exit records
• Fixed file open modes to print RDONLY, WRONLY, or RDRW on open() trace records
• Fixed kiinfo coredump in print_hardclock_rec()/print_stacktrace() due to uninitialized variable in parse_kallsyms()
• Fixed various formatting and alignment issues