Permalink
| Demonstrations of profile, the Linux eBPF/bcc version. | |
| This is a CPU profiler. It works by taking samples of stack traces at timed | |
| intervals, and frequency counting them in kernel context for efficiency. | |
| Example output: | |
| # ./profile | |
| Sampling at 49 Hertz of all threads by user + kernel stack... Hit Ctrl-C to end. | |
| ^C | |
| filemap_map_pages | |
| handle_mm_fault | |
| __do_page_fault | |
| do_page_fault | |
| page_fault | |
| [unknown] | |
| - cp (9036) | |
| 1 | |
| [unknown] | |
| [unknown] | |
| - sign-file (8877) | |
| 1 | |
| __clear_user | |
| iov_iter_zero | |
| read_iter_zero | |
| __vfs_read | |
| vfs_read | |
| sys_read | |
| entry_SYSCALL_64_fastpath | |
| read | |
| - dd (25036) | |
| 4 | |
| func_a | |
| main | |
| __libc_start_main | |
| [unknown] | |
| - func_ab (13549) | |
| 5 | |
| [...] | |
| native_safe_halt | |
| default_idle | |
| arch_cpu_idle | |
| default_idle_call | |
| cpu_startup_entry | |
| rest_init | |
| start_kernel | |
| x86_64_start_reservations | |
| x86_64_start_kernel | |
| - swapper/0 (0) | |
| 72 | |
| native_safe_halt | |
| default_idle | |
| arch_cpu_idle | |
| default_idle_call | |
| cpu_startup_entry | |
| start_secondary | |
| - swapper/1 (0) | |
| 75 | |
| The output was long; I truncated some lines ("[...]"). | |
| This default output prints stack traces, followed by a line to describe the | |
| process (a dash, the process name, and a PID in parenthesis), and then an | |
| integer count of how many times this stack trace was sampled. | |
| The output above shows the most frequent stack was from the "swapper/1" | |
| process (PID 0), running the native_safe_halt() function, which was called | |
| by default_idle(), which was called by arch_cpu_idle(), and so on. This is | |
| the idle thread. Stacks can be read top-down, to follow ancestry: child, | |
| parent, grandparent, etc. | |
| The func_ab process is running the func_a() function, called by main(), | |
| called by __libc_start_main(), and called by "[unknown]" with what looks | |
| like a bogus address (1st column). That's evidence of a broken stack trace. | |
| It's common for user-level software that hasn't been compiled with frame | |
| pointers (in this case, libc). | |
| The dd process has called read(), and then enters the kernel via | |
| entry_SYSCALL_64_fastpath(), calling sys_read(), and so on. Yes, I'm now | |
| reading it bottom up. That way follows the code flow. | |
| The dd process is actually "dd if=/dev/zero of=/dev/null": it's a simple | |
| workload to analyze that just moves bytes from /dev/zero to /dev/null. | |
| Profiling just that process: | |
| # ./profile -p 25036 | |
| Sampling at 49 Hertz of PID 25036 by user + kernel stack... Hit Ctrl-C to end. | |
| ^C | |
| [unknown] | |
| [unknown] | |
| - dd (25036) | |
| 1 | |
| __write | |
| - dd (25036) | |
| 1 | |
| read | |
| - dd (25036) | |
| 1 | |
| [...] | |
| [unknown] | |
| [unknown] | |
| - dd (25036) | |
| 2 | |
| entry_SYSCALL_64_fastpath | |
| __write | |
| [unknown] | |
| - dd (25036) | |
| 3 | |
| entry_SYSCALL_64_fastpath | |
| read | |
| - dd (25036) | |
| 3 | |
| __clear_user | |
| iov_iter_zero | |
| read_iter_zero | |
| __vfs_read | |
| vfs_read | |
| sys_read | |
| entry_SYSCALL_64_fastpath | |
| read | |
| [unknown] | |
| - dd (25036) | |
| 3 | |
| __clear_user | |
| iov_iter_zero | |
| read_iter_zero | |
| __vfs_read | |
| vfs_read | |
| sys_read | |
| entry_SYSCALL_64_fastpath | |
| read | |
| - dd (25036) | |
| 7 | |
| Again, I've truncated some lines. Now we're just analyzing the dd process. | |
| The filtering is performed in kernel context, for efficiency. | |
| This output has some "[unknown]" frames that probably have valid addresses, | |
| but we're lacking the symbol translation. This is a common for all profilers | |
| on Linux, and is usually fixable. See the DEBUGGING section of the profile(8) | |
| man page. | |
| Lets add delimiters between the user and kernel stacks, using -d: | |
| # ./profile -p 25036 -d | |
| ^C | |
| __vfs_write | |
| sys_write | |
| entry_SYSCALL_64_fastpath | |
| -- | |
| __write | |
| - dd (25036) | |
| 1 | |
| -- | |
| [unknown] | |
| [unknown] | |
| - dd (25036) | |
| 1 | |
| iov_iter_init | |
| __vfs_read | |
| vfs_read | |
| sys_read | |
| entry_SYSCALL_64_fastpath | |
| -- | |
| read | |
| - dd (25036) | |
| 1 | |
| [...] | |
| __clear_user | |
| iov_iter_zero | |
| read_iter_zero | |
| __vfs_read | |
| vfs_read | |
| sys_read | |
| entry_SYSCALL_64_fastpath | |
| -- | |
| read | |
| - dd (25036) | |
| 9 | |
| In this mode, the delimiters are "--". | |
| Here's another example, a func_ab program that runs two functions, func_a() and | |
| func_b(). Profiling it for 5 seconds: | |
| # ./profile -p `pgrep -n func_ab` 5 | |
| Sampling at 49 Hertz of PID 2930 by user + kernel stack for 5 secs. | |
| func_a | |
| main | |
| __libc_start_main | |
| [unknown] | |
| - func_ab (2930) | |
| 2 | |
| func_b | |
| main | |
| __libc_start_main | |
| [unknown] | |
| - func_ab (2930) | |
| 3 | |
| func_a | |
| main | |
| __libc_start_main | |
| [unknown] | |
| - func_ab (2930) | |
| 5 | |
| func_b | |
| main | |
| __libc_start_main | |
| [unknown] | |
| - func_ab (2930) | |
| 12 | |
| func_b | |
| main | |
| __libc_start_main | |
| [unknown] | |
| - func_ab (2930) | |
| 19 | |
| func_a | |
| main | |
| __libc_start_main | |
| [unknown] | |
| - func_ab (2930) | |
| 22 | |
| func_b | |
| main | |
| __libc_start_main | |
| [unknown] | |
| - func_ab (2930) | |
| 64 | |
| func_a | |
| main | |
| __libc_start_main | |
| [unknown] | |
| - func_ab (2930) | |
| 72 | |
| Note that the same stack (2nd column) seems to be repeated. Weren't we doing | |
| frequency counting and only printing unique stacks? We are, but in terms of | |
| the raw addresses, not the symbols. See the 1st column: those stacks are | |
| all unique. | |
| We can output in "folded format", which puts the stack trace on one line, | |
| separating frames with semi-colons. Eg: | |
| # ./profile -f -p `pgrep -n func_ab` 5 | |
| func_ab;[unknown];__libc_start_main;main;func_a 2 | |
| func_ab;[unknown];__libc_start_main;main;func_b 2 | |
| func_ab;[unknown];__libc_start_main;main;func_a 11 | |
| func_ab;[unknown];__libc_start_main;main;func_b 12 | |
| func_ab;[unknown];__libc_start_main;main;func_a 23 | |
| func_ab;[unknown];__libc_start_main;main;func_b 28 | |
| func_ab;[unknown];__libc_start_main;main;func_b 57 | |
| func_ab;[unknown];__libc_start_main;main;func_a 64 | |
| I find this pretty useful for writing to files and later grepping. | |
| Folded format can also be used by flame graph stack visualizers, including | |
| the original implementation: | |
| https://github.com/brendangregg/FlameGraph | |
| I'd include delimiters, -d. For example: | |
| # ./profile -df -p `pgrep -n func_ab` 5 > out.profile | |
| # git clone https://github.com/brendangregg/FlameGraph | |
| # ./FlameGraph/flamegraph.pl < out.profile > out.svg | |
| (Yes, I could pipe profile directly into flamegraph.pl, however, I like to | |
| keep the raw folded profiles around: can be useful for regenerating flamegraphs | |
| with different options, and, for differential flame graphs.) | |
| Some flamegraph.pl palettes recognize kernel annotations, which can be added | |
| with -a. It simply adds a "_[k]" at the end of kernel function names. | |
| For example: | |
| # ./profile -adf -p `pgrep -n dd` 10 | |
| dd;[unknown] 1 | |
| dd;[unknown];[unknown] 1 | |
| dd;[unknown];[unknown] 1 | |
| dd;[unknown];read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];rw_verify_area_[k];security_file_permission_[k];__fsnotify_parent_[k] 1 | |
| dd;[unknown];__write;-;entry_SYSCALL_64_fastpath_[k];sys_write_[k];vfs_write_[k];__fsnotify_parent_[k] 1 | |
| dd;[unknown];read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];__fdget_pos_[k] 1 | |
| dd;read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];rw_verify_area_[k];apparmor_file_permission_[k] 1 | |
| dd;[unknown] 1 | |
| dd;[unknown];[unknown] 1 | |
| dd;__write;-;entry_SYSCALL_64_fastpath_[k];sys_write_[k];vfs_write_[k] 1 | |
| dd;[unknown] 1 | |
| dd;read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];__fget_light_[k] 1 | |
| dd;read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];rw_verify_area_[k];__fsnotify_parent_[k] 1 | |
| dd;read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];__vfs_read_[k];read_iter_zero_[k];iov_iter_zero_[k] 1 | |
| dd;[unknown];__write;-;entry_SYSCALL_64_fastpath_[k];sys_write_[k];__fget_light_[k] 1 | |
| dd;[unknown];[unknown] 1 | |
| dd;read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k] 1 | |
| dd;[unknown];[unknown] 1 | |
| dd;read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];__vfs_read_[k];read_iter_zero_[k];iov_iter_zero_[k] 1 | |
| dd;[unknown];read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];__vfs_read_[k];read_iter_zero_[k];iov_iter_zero_[k] 1 | |
| dd;read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];read_iter_zero_[k] 1 | |
| dd;[unknown];read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];__fsnotify_parent_[k] 1 | |
| dd;[unknown];[unknown] 1 | |
| dd;read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];rw_verify_area_[k];security_file_permission_[k];apparmor_file_permission_[k];common_file_perm_[k] 1 | |
| dd;read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];__fsnotify_parent_[k] 1 | |
| dd;__write;-;entry_SYSCALL_64_fastpath_[k];sys_write_[k];vfs_write_[k];fsnotify_[k] 1 | |
| dd;__write;-;entry_SYSCALL_64_fastpath_[k];sys_write_[k];vfs_write_[k];rw_verify_area_[k];security_file_permission_[k] 1 | |
| dd;[unknown];__write;-;entry_SYSCALL_64_fastpath_[k];sys_write_[k];__fdget_pos_[k] 1 | |
| dd;[unknown];__write;-;entry_SYSCALL_64_fastpath_[k] 1 | |
| dd;[unknown];[unknown] 1 | |
| dd;read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];rw_verify_area_[k];security_file_permission_[k] 1 | |
| dd;__write;-;entry_SYSCALL_64_fastpath_[k];sys_write_[k];vfs_write_[k] 1 | |
| dd;[unknown];read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];__fget_light_[k] 1 | |
| dd;[unknown] 1 | |
| dd;[unknown];__write;-;entry_SYSCALL_64_fastpath_[k];sys_write_[k];vfs_write_[k] 1 | |
| dd;read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];rw_verify_area_[k];security_file_permission_[k];apparmor_file_permission_[k] 1 | |
| dd;[unknown];[unknown] 1 | |
| dd;read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];rw_verify_area_[k];security_file_permission_[k] 1 | |
| dd;[unknown];read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];rw_verify_area_[k];__fsnotify_parent_[k] 1 | |
| dd;read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];__vfs_read_[k];read_iter_zero_[k];iov_iter_zero_[k] 1 | |
| dd;__write;-;entry_SYSCALL_64_fastpath_[k];sys_write_[k];vfs_write_[k];rw_verify_area_[k];security_file_permission_[k] 1 | |
| dd;read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];__vfs_read_[k];read_iter_zero_[k];iov_iter_zero_[k];__clear_user_[k] 1 | |
| dd;[unknown];[unknown] 1 | |
| dd;__write;-;entry_SYSCALL_64_fastpath_[k];sys_write_[k];vfs_write_[k] 1 | |
| dd;[unknown];[unknown] 1 | |
| dd;read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];rw_verify_area_[k];security_file_permission_[k];apparmor_file_permission_[k];common_file_perm_[k] 1 | |
| dd;read 1 | |
| dd;__write;-;entry_SYSCALL_64_fastpath_[k];sys_write_[k];vfs_write_[k];security_file_permission_[k] 1 | |
| dd;read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];fsnotify_[k] 1 | |
| dd;read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];fsnotify_[k] 1 | |
| dd;__write;-;entry_SYSCALL_64_fastpath_[k];sys_write_[k];vfs_write_[k];rw_verify_area_[k];apparmor_file_permission_[k] 1 | |
| dd;[unknown];__write;-;entry_SYSCALL_64_fastpath_[k];sys_write_[k];vfs_write_[k];__fsnotify_parent_[k] 1 | |
| dd;[unknown];__write;-;entry_SYSCALL_64_fastpath_[k];sys_write_[k];vfs_write_[k];rw_verify_area_[k];apparmor_file_permission_[k] 1 | |
| dd;[unknown];read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k] 1 | |
| dd;read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];__vfs_read_[k];iov_iter_init_[k] 1 | |
| dd;[unknown];read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];rw_verify_area_[k];security_file_permission_[k];__fsnotify_parent_[k] 1 | |
| dd;[unknown];__write;-;entry_SYSCALL_64_fastpath_[k];sys_write_[k];vfs_write_[k];__vfs_write_[k];write_null_[k] 1 | |
| dd;[unknown];read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];__vfs_read_[k];read_iter_zero_[k];__clear_user_[k] 1 | |
| dd;[unknown];read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];security_file_permission_[k] 1 | |
| dd;read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];rw_verify_area_[k];security_file_permission_[k];apparmor_file_permission_[k];common_file_perm_[k] 1 | |
| dd;read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];__vfs_read_[k];read_iter_zero_[k] 1 | |
| dd;[unknown];__write;-;entry_SYSCALL_64_fastpath_[k];sys_write_[k];vfs_write_[k] 1 | |
| dd;__write;-;entry_SYSCALL_64_fastpath_[k];sys_write_[k];vfs_write_[k] 1 | |
| dd;[unknown];read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];rw_verify_area_[k] 1 | |
| dd;read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];__fget_light_[k] 1 | |
| dd;[unknown];__write;-;entry_SYSCALL_64_fastpath_[k];sys_write_[k];vfs_write_[k];rw_verify_area_[k] 1 | |
| dd;read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];__vfs_read_[k] 1 | |
| dd;[unknown];read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];__vfs_read_[k] 1 | |
| dd;[unknown];__write;-;entry_SYSCALL_64_fastpath_[k];sys_write_[k];vfs_write_[k];__vfs_write_[k] 1 | |
| dd;[unknown] 1 | |
| dd;[unknown];read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];__vfs_read_[k];read_iter_zero_[k];iov_iter_zero_[k] 1 | |
| dd;[unknown] 1 | |
| dd;read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];__vfs_read_[k] 1 | |
| dd;__write;-;entry_SYSCALL_64_fastpath_[k];sys_write_[k];__fsnotify_parent_[k] 1 | |
| dd;__write;-;entry_SYSCALL_64_fastpath_[k];sys_write_[k];vfs_write_[k] 1 | |
| dd;[unknown];[unknown] 1 | |
| dd;read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];__vfs_read_[k];read_iter_zero_[k];iov_iter_zero_[k] 1 | |
| dd;read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];__vfs_read_[k];read_iter_zero_[k];iov_iter_zero_[k] 1 | |
| dd;[unknown];__write;-;sys_write_[k] 1 | |
| dd;read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];__fsnotify_parent_[k] 1 | |
| dd;read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];rw_verify_area_[k];security_file_permission_[k];common_file_perm_[k] 1 | |
| dd;[unknown];read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];__vfs_read_[k];read_iter_zero_[k];iov_iter_zero_[k] 1 | |
| dd;[unknown];[unknown] 1 | |
| dd;read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];__vfs_read_[k];read_iter_zero_[k];iov_iter_zero_[k];__clear_user_[k] 1 | |
| dd;__write;-;entry_SYSCALL_64_fastpath_[k];sys_write_[k];vfs_write_[k];rw_verify_area_[k];security_file_permission_[k];apparmor_file_permission_[k] 1 | |
| dd;__write;-;entry_SYSCALL_64_fastpath_[k];sys_write_[k];__fget_light_[k] 1 | |
| dd;read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k] 1 | |
| dd;read;-;entry_SYSCALL_64_fastpath_[k];vfs_read_[k] 1 | |
| dd;__write 1 | |
| dd;read;-;entry_SYSCALL_64_fastpath_[k];vfs_read_[k] 1 | |
| dd;__write;-;entry_SYSCALL_64_fastpath_[k];sys_write_[k];vfs_write_[k];rw_verify_area_[k];security_file_permission_[k];apparmor_file_permission_[k];common_file_perm_[k] 1 | |
| dd;read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];__vfs_read_[k];read_iter_zero_[k];iov_iter_zero_[k] 1 | |
| dd;[unknown];__write;-;entry_SYSCALL_64_fastpath_[k];sys_write_[k];__fget_light_[k] 1 | |
| dd;[unknown];[unknown] 1 | |
| dd;[unknown] 1 | |
| dd;[unknown];read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];__vfs_read_[k];read_iter_zero_[k];iov_iter_zero_[k] 1 | |
| dd;[unknown] 1 | |
| dd;[unknown] 1 | |
| dd;[unknown];[unknown] 1 | |
| dd;__write;-;entry_SYSCALL_64_fastpath_[k];sys_write_[k];vfs_write_[k] 1 | |
| dd;[unknown];read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];__vfs_read_[k];read_iter_zero_[k];iov_iter_zero_[k] 1 | |
| dd;read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k] 1 | |
| dd;[unknown];read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];__vfs_read_[k];read_iter_zero_[k] 1 | |
| dd;read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k] 1 | |
| dd;__write;-;entry_SYSCALL_64_fastpath_[k];sys_write_[k];vfs_write_[k];rw_verify_area_[k];security_file_permission_[k];apparmor_file_permission_[k];common_file_perm_[k] 1 | |
| dd;__write 1 | |
| dd;[unknown];read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];__fget_light_[k] 1 | |
| dd;read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];rw_verify_area_[k];security_file_permission_[k] 1 | |
| dd;__write;-;entry_SYSCALL_64_fastpath_[k];sys_write_[k] 1 | |
| dd;[unknown] 1 | |
| dd;[unknown];read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];__fget_light_[k] 1 | |
| dd;read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];__vfs_read_[k];read_iter_zero_[k] 1 | |
| dd;[unknown];[unknown] 1 | |
| dd;__write;-;entry_SYSCALL_64_fastpath_[k];sys_write_[k];__fdget_pos_[k] 1 | |
| dd;read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];__vfs_read_[k];read_iter_zero_[k] 1 | |
| dd;[unknown];read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];__vfs_read_[k];read_iter_zero_[k] 1 | |
| dd;read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];__vfs_read_[k];_cond_resched_[k] 1 | |
| dd;[unknown];read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];iov_iter_init_[k] 1 | |
| dd;read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];rw_verify_area_[k];security_file_permission_[k];__fsnotify_parent_[k] 1 | |
| dd;[unknown];read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];__vfs_read_[k] 1 | |
| dd;__write;-;entry_SYSCALL_64_fastpath_[k];sys_write_[k];rw_verify_area_[k] 1 | |
| dd;[unknown];read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];rw_verify_area_[k];apparmor_file_permission_[k] 1 | |
| dd;[unknown];read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];__vfs_read_[k];read_iter_zero_[k];iov_iter_zero_[k] 1 | |
| dd;read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k] 1 | |
| dd;[unknown] 1 | |
| dd;[unknown];read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];rw_verify_area_[k];security_file_permission_[k];fsnotify_[k] 1 | |
| dd;read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];__vfs_read_[k] 1 | |
| dd;__write;-;entry_SYSCALL_64_fastpath_[k];sys_write_[k];__fdget_pos_[k] 1 | |
| dd;[unknown];read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];rw_verify_area_[k];security_file_permission_[k] 1 | |
| dd;__write;-;entry_SYSCALL_64_fastpath_[k];sys_write_[k];vfs_write_[k] 1 | |
| dd;[unknown];__write;-;entry_SYSCALL_64_fastpath_[k];sys_write_[k];vfs_write_[k];__vfs_write_[k] 1 | |
| dd;__write;-;entry_SYSCALL_64_fastpath_[k];sys_write_[k];vfs_write_[k];rw_verify_area_[k];apparmor_file_permission_[k] 1 | |
| dd;read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];rw_verify_area_[k];security_file_permission_[k] 1 | |
| dd;__write;-;entry_SYSCALL_64_fastpath_[k];sys_write_[k];vfs_write_[k];rw_verify_area_[k];security_file_permission_[k];apparmor_file_permission_[k];common_file_perm_[k] 1 | |
| dd;__write;-;entry_SYSCALL_64_fastpath_[k];sys_write_[k];vfs_write_[k] 1 | |
| dd;__write;-;entry_SYSCALL_64_fastpath_[k];sys_write_[k];vfs_write_[k] 1 | |
| dd;read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];__fget_light_[k] 1 | |
| dd;[unknown] 1 | |
| dd;[unknown];__write;-;entry_SYSCALL_64_fastpath_[k];sys_write_[k];vfs_write_[k];fsnotify_[k] 1 | |
| dd;[unknown];__write;-;entry_SYSCALL_64_fastpath_[k];sys_write_[k];vfs_write_[k];rw_verify_area_[k];security_file_permission_[k];apparmor_file_permission_[k];common_file_perm_[k] 1 | |
| dd;__write;-;entry_SYSCALL_64_fastpath_[k];sys_write_[k];vfs_write_[k];rw_verify_area_[k] 1 | |
| dd;[unknown];read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];__vfs_read_[k];read_iter_zero_[k];iov_iter_zero_[k] 1 | |
| dd;read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];fsnotify_[k] 1 | |
| dd;[unknown];__write;-;entry_SYSCALL_64_fastpath_[k];sys_write_[k];vfs_write_[k] 1 | |
| dd;__write;-;entry_SYSCALL_64_fastpath_[k];vfs_write_[k] 1 | |
| dd;[unknown];read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];rw_verify_area_[k] 1 | |
| dd;read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];__vfs_read_[k] 1 | |
| dd;read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];rw_verify_area_[k];security_file_permission_[k];apparmor_file_permission_[k];common_file_perm_[k] 1 | |
| dd;[unknown];read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];__vfs_read_[k];read_iter_zero_[k] 1 | |
| dd;[unknown];read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];rw_verify_area_[k];fsnotify_[k] 1 | |
| dd;[unknown];__write;-;entry_SYSCALL_64_fastpath_[k];sys_write_[k];vfs_write_[k];rw_verify_area_[k];apparmor_file_permission_[k] 2 | |
| dd;read;-;entry_SYSCALL_64_fastpath_[k];__fdget_pos_[k] 2 | |
| dd;[unknown];[unknown] 2 | |
| dd;read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];__fdget_pos_[k] 2 | |
| dd;[unknown];read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];rw_verify_area_[k];security_file_permission_[k];apparmor_file_permission_[k];common_file_perm_[k] 2 | |
| dd;[unknown];[unknown] 2 | |
| dd;[unknown];[unknown] 2 | |
| dd;[unknown];[unknown] 2 | |
| dd;[unknown];read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k] 2 | |
| dd;[unknown];[unknown] 2 | |
| dd;read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];__vfs_read_[k];read_iter_zero_[k];__clear_user_[k] 2 | |
| dd;__write;-;entry_SYSCALL_64_fastpath_[k];__fdget_pos_[k] 2 | |
| dd;[unknown];read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];__vfs_read_[k] 2 | |
| dd;[unknown];read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];__vfs_read_[k] 2 | |
| dd;[unknown];read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];__vfs_read_[k];read_iter_zero_[k];iov_iter_zero_[k];__clear_user_[k] 2 | |
| dd;[unknown];read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];__vfs_read_[k];read_iter_zero_[k];iov_iter_zero_[k];__clear_user_[k] 2 | |
| dd;[unknown];[unknown] 2 | |
| dd;__write;-;entry_SYSCALL_64_fastpath_[k];sys_write_[k];__fget_light_[k] 2 | |
| dd;read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];rw_verify_area_[k];security_file_permission_[k];fsnotify_[k] 2 | |
| dd;__write;-;sys_write_[k] 2 | |
| dd;[unknown];read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];fsnotify_[k] 2 | |
| dd;[unknown];[unknown] 2 | |
| dd;[unknown];read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];__vfs_read_[k] 2 | |
| dd;read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];__vfs_read_[k];read_iter_zero_[k];iov_iter_zero_[k] 2 | |
| dd;read;-;SyS_read_[k] 2 | |
| dd;[unknown] 2 | |
| dd;__write;-;entry_SYSCALL_64_fastpath_[k];sys_write_[k];vfs_write_[k];rw_verify_area_[k] 2 | |
| dd;[unknown];__write;-;entry_SYSCALL_64_fastpath_[k];sys_write_[k];__fget_light_[k] 2 | |
| dd;read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];__vfs_read_[k] 2 | |
| dd;__write;-;entry_SYSCALL_64_fastpath_[k];sys_write_[k];vfs_write_[k];rw_verify_area_[k];security_file_permission_[k];apparmor_file_permission_[k] 2 | |
| dd;read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];__vfs_read_[k];read_iter_zero_[k];__clear_user_[k] 2 | |
| dd;[unknown];__write;-;entry_SYSCALL_64_fastpath_[k];sys_write_[k];rw_verify_area_[k] 2 | |
| dd;[unknown];[unknown] 3 | |
| dd;[unknown];read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];rw_verify_area_[k] 3 | |
| dd;[unknown];[unknown] 3 | |
| dd;read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];__vfs_read_[k];read_iter_zero_[k];iov_iter_zero_[k];__clear_user_[k] 3 | |
| dd;read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];__vfs_read_[k];read_iter_zero_[k];iov_iter_zero_[k] 3 | |
| dd;[unknown];[unknown] 3 | |
| dd;read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];__vfs_read_[k];read_iter_zero_[k];iov_iter_zero_[k] 3 | |
| dd;[unknown];[unknown] 3 | |
| dd;[unknown];[unknown] 3 | |
| dd;__write;-;entry_SYSCALL_64_fastpath_[k];sys_write_[k];vfs_write_[k] 3 | |
| dd;[unknown];read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k] 3 | |
| dd;[unknown];read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];__vfs_read_[k];read_iter_zero_[k];iov_iter_zero_[k];__clear_user_[k] 3 | |
| dd;[unknown] 4 | |
| dd;[unknown];read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k] 4 | |
| dd;[unknown];[unknown] 4 | |
| dd;read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k] 4 | |
| dd;[unknown] 4 | |
| dd;[unknown];[unknown] 4 | |
| dd;[unknown];read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k] 4 | |
| dd;read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];__vfs_read_[k];read_iter_zero_[k];iov_iter_zero_[k];__clear_user_[k] 5 | |
| dd;[unknown];__write;-;entry_SYSCALL_64_fastpath_[k];sys_write_[k];vfs_write_[k] 5 | |
| dd;[unknown];[unknown] 5 | |
| dd;[unknown];[unknown] 5 | |
| dd;[unknown];read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];__vfs_read_[k];read_iter_zero_[k];iov_iter_zero_[k] 6 | |
| dd;read 15 | |
| dd;[unknown];read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];__vfs_read_[k];read_iter_zero_[k];iov_iter_zero_[k];__clear_user_[k] 19 | |
| dd;[unknown];__write;-;entry_SYSCALL_64_fastpath_[k] 20 | |
| dd;read;-;entry_SYSCALL_64_fastpath_[k] 23 | |
| dd;read;-;entry_SYSCALL_64_fastpath_[k];SyS_read_[k];vfs_read_[k];__vfs_read_[k];read_iter_zero_[k];iov_iter_zero_[k];__clear_user_[k] 24 | |
| dd;__write;-;entry_SYSCALL_64_fastpath_[k] 25 | |
| dd;__write 29 | |
| dd;[unknown];read;-;entry_SYSCALL_64_fastpath_[k] 31 | |
| This can be made into a flamegraph. Eg: | |
| # ./profile -adf -p `pgrep -n func_ab` 10 > out.profile | |
| # git clone https://github.com/brendangregg/FlameGraph | |
| # ./FlameGraph/flamegraph.pl --color=java < out.profile > out.svg | |
| It will highlight the kernel frames in orange, and user-level in red (and Java | |
| in green, and C++ in yellow). If you copy-n-paste the above output into a | |
| out.profile file, you can try it out. | |
| You can increase or decrease the sample frequency. Eg, sampling at 9 Hertz: | |
| # ./profile -F 9 | |
| Sampling at 9 Hertz of all threads by user + kernel stack... Hit Ctrl-C to end. | |
| ^C | |
| func_b | |
| main | |
| __libc_start_main | |
| [unknown] | |
| - func_ab (2930) | |
| 1 | |
| [...] | |
| native_safe_halt | |
| default_idle | |
| arch_cpu_idle | |
| default_idle_call | |
| cpu_startup_entry | |
| start_secondary | |
| - swapper/3 (0) | |
| 8 | |
| native_safe_halt | |
| default_idle | |
| arch_cpu_idle | |
| default_idle_call | |
| cpu_startup_entry | |
| rest_init | |
| start_kernel | |
| x86_64_start_reservations | |
| x86_64_start_kernel | |
| - swapper/0 (0) | |
| 8 | |
| You can also restrict profiling to just kernel stacks (-K) or user stacks (-U). | |
| For example, just user stacks: | |
| # ./profile -U | |
| Sampling at 49 Hertz of all threads by user stack... Hit Ctrl-C to end. | |
| ^C | |
| [unknown] | |
| [unknown] | |
| - dd (2931) | |
| 1 | |
| [unknown] | |
| [unknown] | |
| - dd (2931) | |
| 1 | |
| [unknown] | |
| [unknown] | |
| - dd (2931) | |
| 1 | |
| [unknown] | |
| [unknown] | |
| - dd (2931) | |
| 1 | |
| [unknown] | |
| [unknown] | |
| - dd (2931) | |
| 1 | |
| func_b | |
| main | |
| __libc_start_main | |
| [unknown] | |
| - func_ab (2930) | |
| 1 | |
| [unknown] | |
| - dd (2931) | |
| 1 | |
| [unknown] | |
| - dd (2931) | |
| 1 | |
| func_a | |
| main | |
| __libc_start_main | |
| [unknown] | |
| - func_ab (2930) | |
| 3 | |
| __write | |
| [unknown] | |
| - dd (2931) | |
| 3 | |
| func_a | |
| main | |
| __libc_start_main | |
| [unknown] | |
| - func_ab (2930) | |
| 4 | |
| func_b | |
| main | |
| __libc_start_main | |
| [unknown] | |
| - func_ab (2930) | |
| 7 | |
| - swapper/6 (0) | |
| 10 | |
| func_b | |
| main | |
| __libc_start_main | |
| [unknown] | |
| - func_ab (2930) | |
| 10 | |
| __write | |
| - dd (2931) | |
| 10 | |
| func_a | |
| main | |
| __libc_start_main | |
| [unknown] | |
| - func_ab (2930) | |
| 11 | |
| read | |
| - dd (2931) | |
| 12 | |
| read | |
| [unknown] | |
| - dd (2931) | |
| 14 | |
| - swapper/7 (0) | |
| 46 | |
| - swapper/0 (0) | |
| 46 | |
| - swapper/2 (0) | |
| 46 | |
| - swapper/1 (0) | |
| 46 | |
| - swapper/3 (0) | |
| 46 | |
| - swapper/4 (0) | |
| 46 | |
| If there are too many unique stack traces for the kernel to save, a warning | |
| will be printed. Eg: | |
| # ./profile | |
| [...] | |
| WARNING: 8 stack traces could not be displayed. Consider increasing --stack-storage-size. | |
| Run ./profile -h to see the default. | |
| USAGE message: | |
| # ./profile -h | |
| usage: profile [-h] [-p PID] [-U | -K] [-F FREQUENCY] [-d] [-a] [-f] | |
| [--stack-storage-size STACK_STORAGE_SIZE] | |
| [duration] | |
| Profile CPU stack traces at a timed interval | |
| positional arguments: | |
| duration duration of trace, in seconds | |
| optional arguments: | |
| -h, --help show this help message and exit | |
| -p PID, --pid PID profile this PID only | |
| -U, --user-stacks-only | |
| show stacks from user space only (no kernel space | |
| stacks) | |
| -K, --kernel-stacks-only | |
| show stacks from kernel space only (no user space | |
| stacks) | |
| -F FREQUENCY, --frequency FREQUENCY | |
| sample frequency, Hertz (default 49) | |
| -d, --delimited insert delimiter between kernel/user stacks | |
| -a, --annotations add _[k] annotations to kernel frames | |
| -f, --folded output folded format, one line per stack (for flame | |
| graphs) | |
| --stack-storage-size STACK_STORAGE_SIZE | |
| the number of unique stack traces that can be stored | |
| and displayed (default 2048) | |
| examples: | |
| ./profile # profile stack traces at 49 Hertz until Ctrl-C | |
| ./profile -F 99 # profile stack traces at 99 Hertz | |
| ./profile 5 # profile at 49 Hertz for 5 seconds only | |
| ./profile -f 5 # output in folded format for flame graphs | |
| ./profile -p 185 # only profile threads for PID 185 | |
| ./profile -U # only show user space stacks (no kernel) | |
| ./profile -K # only show kernel space stacks (no user) |