-
Notifications
You must be signed in to change notification settings - Fork 268
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Cleanup and update to gdb 7.8 #2
Cleanup and update to gdb 7.8 #2
Conversation
This fixes a compilation failure on ppc64. (ptesarik@suse.cz)
The last release of gdb 5.x was in 2002. We can remove support for it, cleaning up the code. Since setup_gdb_defaults is using its own strings, we can leave the block for this version empty to avoid messing with the version #defines that are used as indexes. We'll clean it up in a later patch. (jeffm@suse.com)
gdb 6.1 was released in August 2011. We can remove support for it, cleaning up the code. Since setup_gdb_defaults is using its own strings, we can leave the block for this version empty to avoid messing with the version #defines that are used as indexes. We'll clean it up in a later patch. (jeffm@suse.com)
gdb 7.3.1 was released in September 2011. We can remove support for it, cleaning up the code. Since setup_gdb_defaults is using its own strings, we can leave the block for this version empty to avoid messing with the version #defines that are used as indexes. We'll clean it up in a later patch. (jeffm@suse.com)
This patch cleans up the searching for a supported gdb, removing the empty blocks and replacing the index with a simple pointer. (jeffm@suse.com)
With the goal of significantly shrinking the size of the gdb patch, we rename crash's read_string to mem_read_string and remove the renaming of gdb's read_string. (jeffm@suse.com)
patch_kernel_symbol always deals with minimal symbols. Just pass the pointer in and back out to a callback that uses the right macros. This isn't entirely necessary for use with gdb 7.6 but will be for gdb 7.8 since it doesn't provide a single value that can be used to generate a pointer. (jeffm@suse.com)
The hook assignment method of providing an alternative interface has been deprecated from gdb for some time. Gdb 7.7 or 7.8 removed the deprecated_command_loop_hook callback. It's time to just use the interpreter interface. Since all we really want to do is provide a different event loop, we can copy the cli interpreter and override the command_loop_proc there. In doing so, we can also let gdb initialize it for us, get rid of the separate newline-and-exit callback for version queries, and drop the update_gdb_hooks callback. (jeffm@suse.com)
same_file doesn't modify either argument, so constify them. This avoids a warning when building against gdb 7.8, which must use objfile_name() instead of objfile->name. (jeffm@suse.com)
We don't change the module parameter but will pass it a const char *, causing build warnings. (jeffm@suse.com)
gdb 7.7 renamed prettyprint to prettyformat, so let's request that option first. Failing that, fall back to the old prettyprint names. (jeffm@suse.com)
We don't change the filename but can pass it a const char *, causing a build warning. (jeffm@suse.com)
There are a number of locations in which we do a callsite prototype declaration and there's no need to do that. Just put them in defs.h -- in the right place, since the prototypes are the same whether GDB_COMMON is set or not. (jeffm@suse.com)
gdb_kernel_objfile performs the same function as symfile_objfile, introduced sometime in the 7.x release stream. Let's just use that. (jeffm@suse.com)
We maintain a global variable that is set and cleared across a single function call. It's much cleaner to just pass it as a parameter through to the actual consumer. (jeffm@suse.com)
The crash-specific gdb interface can be implemented in a separate C file rather than patching gdb code. This is much easier to maintain in the long run. A later patch can integrate crash.c and gdb_interface.c for further simplification. (jeffm@suse.com)
This patch adds support for gdb 7.8. (jeffm@suse.com)
There seems to be an integration issue. 'dmesg' works but printing of the variables it uses for structured logging does not. |
----- Original Message -----
Let me get this straight -- you've had to make additional changes to gdb-7.8 for How exactly does this Python functionality get used in crash? I don't speak http://people.redhat.com/anderson/extensions.html#PYTHON
That's debatable on this end. Historically the only reason to update the embedded
Why? Removing support for earlier versions is something that I don't feel comfortable
I grant you that it is somewhat confusing, but the #ifdef lines w/respect to
Similar to our Red Hat "KABI" for changing kernel interfaces, Red Hat is also Given that there are only 2 (count 'em) callers of the read_string() function
Damn -- I see they've removed the deprecated_command_hook, which I was hoping
It's not clear to me how this works -- it looks like you've kept gdb-7.6 Again, I would like to keep things working in a traditional manner. When a
I'm currently not using the pull functionality at github. Changes of this But as noted above, I'd rather see a "traditional" bumping of the embedded Thanks,
|
Without the patch, the backtrace displays the "cannot resolve stack trace" warning, dumps the backtrace, and then the text symbols: crash> bt PID: 0 TASK: f0962180 CPU: 6 COMMAND: "swapper/6" bt: cannot resolve stack trace: #0 [f095ff1c] __schedule at c0b6ef8d #1 [f095ff58] schedule at c0b6f4a9 #2 [f095ff64] schedule_preempt_disabled at c0b6f728 #3 [f095ff6c] cpu_startup_entry at c04b0310 #4 [f095ff94] start_secondary at c04468c0 bt: text symbols on stack: [f095ff1c] __schedule at c0b6ef8d [f095ff58] schedule at c0b6f4ae [f095ff64] schedule_preempt_disabled at c0b6f72d [f095ff6c] cpu_startup_entry at c04b0315 [f095ff94] start_secondary at c04468c5 crash> The backtrace shown is actually correct. (anderson@redhat.com)
Signed-off-by: Leo Yan <leo.yan@linaro.org>
Previously we can only view the stack unwinding for the tasks which are running on each CPUs. This patch will enable the ability to view arbitrary tasks stack unwinding. After crash get initialized, "info threads" will output like the following: crash> info threads Id Target Id Frame 1 CPU 0 native_safe_halt () at arch/x86/include/asm/irqflags.h:54 ... * 8 CPU 7 blk_mq_rq_timed_out (req=0xffff880fdb246000, reserved=reserved@entry=false) at block/blk-mq.c:640 ... 13 CPU 12 <unavailable> in ?? () 14 CPU 13 native_safe_halt () at arch/x86/include/asm/irqflags.h:54 ... crash> ps PID PPID CPU TASK ST %MEM VSZ RSS COMM > 0 0 0 ffffffff819f9480 RU 0.0 0 0 [swapper/0] > 0 0 1 ffff880169411fa0 RU 0.0 0 0 [swapper/1] ... 0 0 23 ffff8801694e0000 RU 0.0 0 0 [swapper/23] 1 0 13 ffff880169b30000 IN 0.0 193052 4180 systemd "info threads" show the tasks which are currently running on each CPU. If we'd like to view systemd task's stack unwinding, which is inactive status, we do the following: crash> set 1 or crash> set ffff880169b30000 Then the register cache of systemd will be swapped into CPU 13: crash> info threads Id Target Id Frame 1 CPU 0 native_safe_halt () at arch/x86/include/asm/irqflags.h:54 ... 8 CPU 7 blk_mq_rq_timed_out (req=0xffff880fdb246000, reserved=reserved@entry=false) at block/blk-mq.c:640 ... 13 CPU 12 <unavailable> in ?? () * 14 CPU 13 0xffffffff816a8f65 in context_switch (rq=0x0, next=0x0, prev=0xffff880169b30000) at kernel/sched/core.c:2527 ... And we can view the stack unwinding of systemd: crash> bt PID: 1 TASK: ffff880169b30000 CPU: 13 COMMAND: "systemd" #0 [ffff880169b3bd58] __schedule at ffffffff816a8f65 crash-utility#1 [ffff880169b3bdc0] schedule at ffffffff816a94e9 crash-utility#2 [ffff880169b3bdd0] schedule_hrtimeout_range_clock at ffffffff816a86fd crash-utility#3 [ffff880169b3be68] schedule_hrtimeout_range at ffffffff816a8733 crash-utility#4 [ffff880169b3be78] ep_poll at ffffffff8124bb7e crash-utility#5 [ffff880169b3bf30] sys_epoll_wait at ffffffff8124d00d crash-utility#6 [ffff880169b3bf80] system_call_fastpath at ffffffff816b5009 RIP: 00007f0449407923 RSP: 00007ffc35a3c378 RFLAGS: 00010246 RAX: 00000000000000e8 RBX: ffffffff816b5009 RCX: 0000000000000071 RDX: 000000000000001d RSI: 00007ffc35a3d5a0 RDI: 0000000000000004 RBP: 00007ffc35a3d810 R8: 0000000000000000 R9: 0000000000000000 R10: 00000000ffffffff R11: 0000000000000293 R12: 0000563ca2ebe980 R13: 0000000000000003 R14: ffffffffffffffff R15: 0000000000000001 ORIG_RAX: 00000000000000e8 CS: 0033 SS: 002b crash> gdb bt #0 0xffffffff816a8f65 in context_switch (rq=0x0, next=0x0, prev=0xffff880169b30000) at kernel/sched/core.c:2527 crash-utility#1 __schedule () at kernel/sched/core.c:3540 crash-utility#2 0xffffffff816a94e9 in schedule () at kernel/sched/core.c:3577 crash-utility#3 0xffffffff816a86fd in schedule_hrtimeout_range_clock (expires=expires@entry=0x0, delta=delta@entry=0, mode=mode@entry=HRTIMER_MODE_ABS, clock=clock@entry=1) at kernel/hrtimer.c:1724 crash-utility#4 0xffffffff816a8733 in schedule_hrtimeout_range (expires=expires@entry=0x0, delta=delta@entry=0, mode=mode@entry=HRTIMER_MODE_ABS) at kernel/hrtimer.c:1778 crash-utility#5 0xffffffff8124bb7e in ep_poll (ep=0xffff880fd861f8c0, events=events@entry=0x7ffc35a3d5a0, maxevents=maxevents@entry=29, timeout=timeout@entry=-1) at fs/eventpoll.c:1669 crash-utility#6 0xffffffff8124d00d in SYSC_epoll_wait (timeout=<optimized out>, maxevents=29, events=<optimized out>, epfd=<optimized out>) at fs/eventpoll.c:2043 crash-utility#7 SyS_epoll_wait (epfd=<optimized out>, events=140721208415648, maxevents=29, timeout=4294967295) at fs/eventpoll.c:2008 crash-utility#8 <signal handler called> crash-utility#9 0x00007f0449407923 in ?? () Signed-off-by: Tao Liu <ltao@redhat.com>
The stack unwinding is for kernel addresses only. If non-kernel address encountered, it is usually a user space address, or non-address value like a function call parameter. So stopping stack unwinding at non-kernel address will decrease the invalid unwind results. Before: crash> gdb bt #0 0xffffffff816a8f65 in context_switch ... crash-utility#1 __schedule () ... crash-utility#2 0xffffffff816a94e9 in schedule ... crash-utility#3 0xffffffff816a86fd in schedule_hrtimeout_range_clock ... crash-utility#4 0xffffffff816a8733 in schedule_hrtimeout_range ... crash-utility#5 0xffffffff8124bb7e in ep_poll ... crash-utility#6 0xffffffff8124d00d in SYSC_epoll_wait ... crash-utility#7 SyS_epoll_wait ... crash-utility#8 <signal handler called> crash-utility#9 0x00007f0449407923 in ?? () crash-utility#10 0xffff880100000001 in ?? () crash-utility#11 0xffff880169b3c010 in ?? () crash-utility#12 0x0000000000000040 in irq_stack_union () crash-utility#13 0xffff880169b3c058 in ?? () crash-utility#14 0xffff880169b3c048 in ?? () crash-utility#15 0xffff880169b3c050 in ?? () crash-utility#16 0x0000000000000000 in ?? () After: crash> gdb bt #0 0xffffffff816a8f65 in context_switch ... crash-utility#1 __schedule () ... crash-utility#2 0xffffffff816a94e9 in schedule () ... crash-utility#3 0xffffffff816a86fd in schedule_hrtimeout_range_clock ... crash-utility#4 0xffffffff816a8733 in schedule_hrtimeout_range ... crash-utility#5 0xffffffff8124bb7e in ep_poll ... crash-utility#6 0xffffffff8124d00d in SYSC_epoll_wait ... crash-utility#7 SyS_epoll_wait ... crash-utility#8 <signal handler called> crash-utility#9 0x00007f0449407923 in ?? () Signed-off-by: Tao Liu <ltao@redhat.com>
The live debug can be enabled for ppc64. For inactive tasks, it can show the stack unwinding results: crash> sys KERNEL: /usr/lib/debug/lib/modules/5.14.0-425.el9.ppc64le/vmlinux DUMPFILE: /proc/kcore ... crash> set 1 crash> bt PID: 1 TASK: c0000000035fc900 CPU: 1 COMMAND: "systemd" #0 [c00000000369fa60] __schedule at c000000000fc3c58 crash-utility#1 [c00000000369fb20] schedule at c000000000fc411c crash-utility#2 [c00000000369fb50] schedule_hrtimeout_range_clock at c000000000fcd2a4 crash-utility#3 [c00000000369fc00] ep_poll at c00000000063640c crash-utility#4 [c00000000369fcf0] do_epoll_wait at c000000000636584 crash-utility#5 [c00000000369fd40] sys_epoll_wait at c000000000636608 crash-utility#6 [c00000000369fdb0] system_call_exception at c00000000002e994 crash-utility#7 [c00000000369fe10] system_call_vectored_common at c00000000000bfe8 crash> gdb bt #0 0xc000000000fc3c58 in context_switch ... crash-utility#1 __schedule ... crash-utility#2 0xc000000000fc411c in schedule_loop ... crash-utility#3 schedule ... crash-utility#4 0xc000000000fcd2a4 in schedule_hrtimeout_range_clock ... crash-utility#5 0xc00000000063640c in ep_poll ... crash-utility#6 0xc000000000636584 in do_epoll_wait ... crash-utility#7 0xc000000000636608 in __do_sys_epoll_wait ... crash-utility#8 __se_sys_epoll_wait ... crash-utility#9 0xc00000000002e994 in system_call_exception ... crash-utility#10 0xc00000000000bfe8 in system_call_vectored_common ... However for active tasks in live mode, stack unwind will fail. The behaviour is similar for "bt" and "gdb bt": crash> ps PID PPID CPU TASK ST %MEM VSZ RSS COMM 0 0 0 c000000002af6380 RU 0.0 0 0 [swapper/0] > 0 0 1 c0000000035f9000 RU 0.0 0 0 [swapper/1] > 0 0 2 c0000000035f0180 RU 0.0 0 0 [swapper/2] ... crash> set c0000000035f0180 crash> bt PID: 0 TASK: c0000000035f0180 CPU: 2 COMMAND: "swapper/2" (active) crash> gdb bt #0 0xc000000003847d50 in ?? () crash-utility#1 0x0000000000000000 in ?? () Signed-off-by: Tao Liu <ltao@redhat.com>
Previously we can only view the stack unwinding for the tasks which are running on each CPUs. This patch will enable the ability to view arbitrary tasks stack unwinding. After crash get initialized, "info threads" will output like the following: crash> info threads Id Target Id Frame 1 CPU 0 native_safe_halt () at arch/x86/include/asm/irqflags.h:54 ... * 8 CPU 7 blk_mq_rq_timed_out (req=0xffff880fdb246000, reserved=reserved@entry=false) at block/blk-mq.c:640 ... 13 CPU 12 <unavailable> in ?? () 14 CPU 13 native_safe_halt () at arch/x86/include/asm/irqflags.h:54 ... crash> ps PID PPID CPU TASK ST %MEM VSZ RSS COMM > 0 0 0 ffffffff819f9480 RU 0.0 0 0 [swapper/0] > 0 0 1 ffff880169411fa0 RU 0.0 0 0 [swapper/1] ... 0 0 23 ffff8801694e0000 RU 0.0 0 0 [swapper/23] 1 0 13 ffff880169b30000 IN 0.0 193052 4180 systemd "info threads" show the tasks which are currently running on each CPU. If we'd like to view systemd task's stack unwinding, which is inactive status, we do the following: crash> set 1 or crash> set ffff880169b30000 Then the register cache of systemd will be swapped into CPU 13: crash> info threads Id Target Id Frame 1 CPU 0 native_safe_halt () at arch/x86/include/asm/irqflags.h:54 ... 8 CPU 7 blk_mq_rq_timed_out (req=0xffff880fdb246000, reserved=reserved@entry=false) at block/blk-mq.c:640 ... 13 CPU 12 <unavailable> in ?? () * 14 CPU 13 0xffffffff816a8f65 in context_switch (rq=0x0, next=0x0, prev=0xffff880169b30000) at kernel/sched/core.c:2527 ... And we can view the stack unwinding of systemd: crash> bt PID: 1 TASK: ffff880169b30000 CPU: 13 COMMAND: "systemd" #0 [ffff880169b3bd58] __schedule at ffffffff816a8f65 crash-utility#1 [ffff880169b3bdc0] schedule at ffffffff816a94e9 crash-utility#2 [ffff880169b3bdd0] schedule_hrtimeout_range_clock at ffffffff816a86fd crash-utility#3 [ffff880169b3be68] schedule_hrtimeout_range at ffffffff816a8733 crash-utility#4 [ffff880169b3be78] ep_poll at ffffffff8124bb7e crash-utility#5 [ffff880169b3bf30] sys_epoll_wait at ffffffff8124d00d crash-utility#6 [ffff880169b3bf80] system_call_fastpath at ffffffff816b5009 RIP: 00007f0449407923 RSP: 00007ffc35a3c378 RFLAGS: 00010246 RAX: 00000000000000e8 RBX: ffffffff816b5009 RCX: 0000000000000071 RDX: 000000000000001d RSI: 00007ffc35a3d5a0 RDI: 0000000000000004 RBP: 00007ffc35a3d810 R8: 0000000000000000 R9: 0000000000000000 R10: 00000000ffffffff R11: 0000000000000293 R12: 0000563ca2ebe980 R13: 0000000000000003 R14: ffffffffffffffff R15: 0000000000000001 ORIG_RAX: 00000000000000e8 CS: 0033 SS: 002b crash> gdb bt #0 0xffffffff816a8f65 in context_switch (rq=0x0, next=0x0, prev=0xffff880169b30000) at kernel/sched/core.c:2527 crash-utility#1 __schedule () at kernel/sched/core.c:3540 crash-utility#2 0xffffffff816a94e9 in schedule () at kernel/sched/core.c:3577 crash-utility#3 0xffffffff816a86fd in schedule_hrtimeout_range_clock (expires=expires@entry=0x0, delta=delta@entry=0, mode=mode@entry=HRTIMER_MODE_ABS, clock=clock@entry=1) at kernel/hrtimer.c:1724 crash-utility#4 0xffffffff816a8733 in schedule_hrtimeout_range (expires=expires@entry=0x0, delta=delta@entry=0, mode=mode@entry=HRTIMER_MODE_ABS) at kernel/hrtimer.c:1778 crash-utility#5 0xffffffff8124bb7e in ep_poll (ep=0xffff880fd861f8c0, events=events@entry=0x7ffc35a3d5a0, maxevents=maxevents@entry=29, timeout=timeout@entry=-1) at fs/eventpoll.c:1669 crash-utility#6 0xffffffff8124d00d in SYSC_epoll_wait (timeout=<optimized out>, maxevents=29, events=<optimized out>, epfd=<optimized out>) at fs/eventpoll.c:2043 crash-utility#7 SyS_epoll_wait (epfd=<optimized out>, events=140721208415648, maxevents=29, timeout=4294967295) at fs/eventpoll.c:2008 crash-utility#8 <signal handler called> crash-utility#9 0x00007f0449407923 in ?? () Signed-off-by: Tao Liu <ltao@redhat.com>
The stack unwinding is for kernel addresses only. If non-kernel address encountered, it is usually a user space address, or non-address value like a function call parameter. So stopping stack unwinding at non-kernel address will decrease the invalid unwind results. Before: crash> gdb bt #0 0xffffffff816a8f65 in context_switch ... crash-utility#1 __schedule () ... crash-utility#2 0xffffffff816a94e9 in schedule ... crash-utility#3 0xffffffff816a86fd in schedule_hrtimeout_range_clock ... crash-utility#4 0xffffffff816a8733 in schedule_hrtimeout_range ... crash-utility#5 0xffffffff8124bb7e in ep_poll ... crash-utility#6 0xffffffff8124d00d in SYSC_epoll_wait ... crash-utility#7 SyS_epoll_wait ... crash-utility#8 <signal handler called> crash-utility#9 0x00007f0449407923 in ?? () crash-utility#10 0xffff880100000001 in ?? () crash-utility#11 0xffff880169b3c010 in ?? () crash-utility#12 0x0000000000000040 in irq_stack_union () crash-utility#13 0xffff880169b3c058 in ?? () crash-utility#14 0xffff880169b3c048 in ?? () crash-utility#15 0xffff880169b3c050 in ?? () crash-utility#16 0x0000000000000000 in ?? () After: crash> gdb bt #0 0xffffffff816a8f65 in context_switch ... crash-utility#1 __schedule () ... crash-utility#2 0xffffffff816a94e9 in schedule () ... crash-utility#3 0xffffffff816a86fd in schedule_hrtimeout_range_clock ... crash-utility#4 0xffffffff816a8733 in schedule_hrtimeout_range ... crash-utility#5 0xffffffff8124bb7e in ep_poll ... crash-utility#6 0xffffffff8124d00d in SYSC_epoll_wait ... crash-utility#7 SyS_epoll_wait ... crash-utility#8 <signal handler called> crash-utility#9 0x00007f0449407923 in ?? () Signed-off-by: Tao Liu <ltao@redhat.com>
The live debug can be enabled for ppc64. For inactive tasks, it can show the stack unwinding results: crash> sys KERNEL: /usr/lib/debug/lib/modules/5.14.0-425.el9.ppc64le/vmlinux DUMPFILE: /proc/kcore ... crash> set 1 crash> bt PID: 1 TASK: c0000000035fc900 CPU: 1 COMMAND: "systemd" #0 [c00000000369fa60] __schedule at c000000000fc3c58 crash-utility#1 [c00000000369fb20] schedule at c000000000fc411c crash-utility#2 [c00000000369fb50] schedule_hrtimeout_range_clock at c000000000fcd2a4 crash-utility#3 [c00000000369fc00] ep_poll at c00000000063640c crash-utility#4 [c00000000369fcf0] do_epoll_wait at c000000000636584 crash-utility#5 [c00000000369fd40] sys_epoll_wait at c000000000636608 crash-utility#6 [c00000000369fdb0] system_call_exception at c00000000002e994 crash-utility#7 [c00000000369fe10] system_call_vectored_common at c00000000000bfe8 crash> gdb bt #0 0xc000000000fc3c58 in context_switch ... crash-utility#1 __schedule ... crash-utility#2 0xc000000000fc411c in schedule_loop ... crash-utility#3 schedule ... crash-utility#4 0xc000000000fcd2a4 in schedule_hrtimeout_range_clock ... crash-utility#5 0xc00000000063640c in ep_poll ... crash-utility#6 0xc000000000636584 in do_epoll_wait ... crash-utility#7 0xc000000000636608 in __do_sys_epoll_wait ... crash-utility#8 __se_sys_epoll_wait ... crash-utility#9 0xc00000000002e994 in system_call_exception ... crash-utility#10 0xc00000000000bfe8 in system_call_vectored_common ... However for active tasks in live mode, stack unwind will fail. The behaviour is similar for "bt" and "gdb bt": crash> ps PID PPID CPU TASK ST %MEM VSZ RSS COMM 0 0 0 c000000002af6380 RU 0.0 0 0 [swapper/0] > 0 0 1 c0000000035f9000 RU 0.0 0 0 [swapper/1] > 0 0 2 c0000000035f0180 RU 0.0 0 0 [swapper/2] ... crash> set c0000000035f0180 crash> bt PID: 0 TASK: c0000000035f0180 CPU: 2 COMMAND: "swapper/2" (active) crash> gdb bt #0 0xc000000003847d50 in ?? () crash-utility#1 0x0000000000000000 in ?? () Signed-off-by: Tao Liu <ltao@redhat.com>
Currently, gdb passthroughs of 'bt', 'frame', 'up', 'down', 'info locals' don't work. This is due to gdb not knowing the register values to unwind the stack frames Every gdb passthrough goes through `gdb_interface`. And then, gdb expects `crash_target::fetch_registers` to give it the register values, which is dependent on `machdep->get_cpu_reg` to read the register values for specific architecture. ---------------------------- gdb passthrough (eg. "bt") | | crash -------------------------> | | | gdb_interface | | | | | | ---------------------- | fetch_registers | | | | crash_target<-------------------------+--| gdb | | --------------------------+->| | | Registers (SP,NIP, etc.)| | | | | | | | | ---------------------- | ---------------------------- Implement `machdep->get_cpu_reg` on PPC64, so that crash provides the register values to gdb to unwind stack frames properly With these changes, on powerpc, 'bt' command output in gdb mode, will look like this: gdb> bt #0 0xc0000000002a53e8 in crash_setup_regs (oldregs=<optimized out>, newregs=0xc00000000486f8d8) at ./arch/powerpc/include/asm/kexec.h:69 crash-utility#1 __crash_kexec (regs=<optimized out>) at kernel/kexec_core.c:974 crash-utility#2 0xc000000000168918 in panic (fmt=<optimized out>) at kernel/panic.c:358 crash-utility#3 0xc000000000b735f8 in sysrq_handle_crash (key=<optimized out>) at drivers/tty/sysrq.c:155 crash-utility#4 0xc000000000b742cc in __handle_sysrq (key=key@entry=99, check_mask=check_mask@entry=false) at drivers/tty/sysrq.c:602 crash-utility#5 0xc000000000b7506c in write_sysrq_trigger (file=<optimized out>, buf=<optimized out>, count=2, ppos=<optimized out>) at drivers/tty/sysrq.c:1163 crash-utility#6 0xc00000000069a7bc in pde_write (ppos=<optimized out>, count=<optimized out>, buf=<optimized out>, file=<optimized out>, pde=0xc000000009ed3a80) at fs/proc/inode.c:340 crash-utility#7 proc_reg_write (file=<optimized out>, buf=<optimized out>, count=<optimized out>, ppos=<optimized out>) at fs/proc/inode.c:352 crash-utility#8 0xc0000000005b3bbc in vfs_write (file=file@entry=0xc00000009dda7d00, buf=buf@entry=0xebcfc7c6040 <error: Cannot access memory at address 0xebcfc7c6040>, count=count@entry=2, pos=pos@entry=0xc00000000486fda0) at fs/read_write.c:582 instead of earlier output without this patch: gdb> bt #0 <unavailable> in ?? () Backtrace stopped: previous frame identical to this frame (corrupt stack?) Also, 'get_dumpfile_regs' has been introduced to get registers from multiple supported vmcore formats. Correspondingly a flag 'BT_NO_PRINT_REGS' has been introduced to tell helper functions to get registers, to not print registers with every call to backtrace in gdb. Note: This feature to support GDB unwinding doesn't support live debugging Cc: Sourabh Jain <sourabhjain@linux.ibm.com> Cc: Hari Bathini <hbathini@linux.ibm.com> Cc: Mahesh J Salgaonkar <mahesh@linux.ibm.com> Cc: Naveen N. Rao <naveen.n.rao@linux.vnet.ibm.com> Cc: Lianbo Jiang <lijiang@redhat.com> Cc: HAGIO KAZUHITO(萩尾 一仁) <k-hagio-ab@nec.com> Improved-by: Tao Liu <ltao@redhat.com> Signed-off-by: Aditya Gupta <adityag@linux.ibm.com>
Previously we can only view the stack unwinding for the tasks which are running on each CPUs. This patch will enable the ability to view arbitrary tasks stack unwinding. After crash get initialized, "info threads" will output like the following: crash> info threads Id Target Id Frame 1 CPU 0 native_safe_halt () at arch/x86/include/asm/irqflags.h:54 ... * 8 CPU 7 blk_mq_rq_timed_out (req=0xffff880fdb246000, reserved=reserved@entry=false) at block/blk-mq.c:640 ... 13 CPU 12 <unavailable> in ?? () 14 CPU 13 native_safe_halt () at arch/x86/include/asm/irqflags.h:54 ... crash> ps PID PPID CPU TASK ST %MEM VSZ RSS COMM > 0 0 0 ffffffff819f9480 RU 0.0 0 0 [swapper/0] > 0 0 1 ffff880169411fa0 RU 0.0 0 0 [swapper/1] ... 0 0 23 ffff8801694e0000 RU 0.0 0 0 [swapper/23] 1 0 13 ffff880169b30000 IN 0.0 193052 4180 systemd "info threads" show the tasks which are currently running on each CPU. If we'd like to view systemd task's stack unwinding, which is inactive status, we do the following: crash> set 1 or crash> set ffff880169b30000 Then the register cache of systemd will be swapped into CPU 13: crash> info threads Id Target Id Frame 1 CPU 0 native_safe_halt () at arch/x86/include/asm/irqflags.h:54 ... 8 CPU 7 blk_mq_rq_timed_out (req=0xffff880fdb246000, reserved=reserved@entry=false) at block/blk-mq.c:640 ... 13 CPU 12 <unavailable> in ?? () * 14 CPU 13 0xffffffff816a8f65 in context_switch (rq=0x0, next=0x0, prev=0xffff880169b30000) at kernel/sched/core.c:2527 ... And we can view the stack unwinding of systemd: crash> bt PID: 1 TASK: ffff880169b30000 CPU: 13 COMMAND: "systemd" #0 [ffff880169b3bd58] __schedule at ffffffff816a8f65 crash-utility#1 [ffff880169b3bdc0] schedule at ffffffff816a94e9 crash-utility#2 [ffff880169b3bdd0] schedule_hrtimeout_range_clock at ffffffff816a86fd crash-utility#3 [ffff880169b3be68] schedule_hrtimeout_range at ffffffff816a8733 crash-utility#4 [ffff880169b3be78] ep_poll at ffffffff8124bb7e crash-utility#5 [ffff880169b3bf30] sys_epoll_wait at ffffffff8124d00d crash-utility#6 [ffff880169b3bf80] system_call_fastpath at ffffffff816b5009 RIP: 00007f0449407923 RSP: 00007ffc35a3c378 RFLAGS: 00010246 RAX: 00000000000000e8 RBX: ffffffff816b5009 RCX: 0000000000000071 RDX: 000000000000001d RSI: 00007ffc35a3d5a0 RDI: 0000000000000004 RBP: 00007ffc35a3d810 R8: 0000000000000000 R9: 0000000000000000 R10: 00000000ffffffff R11: 0000000000000293 R12: 0000563ca2ebe980 R13: 0000000000000003 R14: ffffffffffffffff R15: 0000000000000001 ORIG_RAX: 00000000000000e8 CS: 0033 SS: 002b crash> gdb bt #0 0xffffffff816a8f65 in context_switch (rq=0x0, next=0x0, prev=0xffff880169b30000) at kernel/sched/core.c:2527 crash-utility#1 __schedule () at kernel/sched/core.c:3540 crash-utility#2 0xffffffff816a94e9 in schedule () at kernel/sched/core.c:3577 crash-utility#3 0xffffffff816a86fd in schedule_hrtimeout_range_clock (expires=expires@entry=0x0, delta=delta@entry=0, mode=mode@entry=HRTIMER_MODE_ABS, clock=clock@entry=1) at kernel/hrtimer.c:1724 crash-utility#4 0xffffffff816a8733 in schedule_hrtimeout_range (expires=expires@entry=0x0, delta=delta@entry=0, mode=mode@entry=HRTIMER_MODE_ABS) at kernel/hrtimer.c:1778 crash-utility#5 0xffffffff8124bb7e in ep_poll (ep=0xffff880fd861f8c0, events=events@entry=0x7ffc35a3d5a0, maxevents=maxevents@entry=29, timeout=timeout@entry=-1) at fs/eventpoll.c:1669 crash-utility#6 0xffffffff8124d00d in SYSC_epoll_wait (timeout=<optimized out>, maxevents=29, events=<optimized out>, epfd=<optimized out>) at fs/eventpoll.c:2043 crash-utility#7 SyS_epoll_wait (epfd=<optimized out>, events=140721208415648, maxevents=29, timeout=4294967295) at fs/eventpoll.c:2008 crash-utility#8 <signal handler called> crash-utility#9 0x00007f0449407923 in ?? () Signed-off-by: Tao Liu <ltao@redhat.com> Signed-off-by: Aditya Gupta <adityag@linux.ibm.com>
The stack unwinding is for kernel addresses only. If non-kernel address encountered, it is usually a user space address, or non-address value like a function call parameter. So stopping stack unwinding at non-kernel address will decrease the invalid unwind results. Before: crash> gdb bt #0 0xffffffff816a8f65 in context_switch ... crash-utility#1 __schedule () ... crash-utility#2 0xffffffff816a94e9 in schedule ... crash-utility#3 0xffffffff816a86fd in schedule_hrtimeout_range_clock ... crash-utility#4 0xffffffff816a8733 in schedule_hrtimeout_range ... crash-utility#5 0xffffffff8124bb7e in ep_poll ... crash-utility#6 0xffffffff8124d00d in SYSC_epoll_wait ... crash-utility#7 SyS_epoll_wait ... crash-utility#8 <signal handler called> crash-utility#9 0x00007f0449407923 in ?? () crash-utility#10 0xffff880100000001 in ?? () crash-utility#11 0xffff880169b3c010 in ?? () crash-utility#12 0x0000000000000040 in irq_stack_union () crash-utility#13 0xffff880169b3c058 in ?? () crash-utility#14 0xffff880169b3c048 in ?? () crash-utility#15 0xffff880169b3c050 in ?? () crash-utility#16 0x0000000000000000 in ?? () After: crash> gdb bt #0 0xffffffff816a8f65 in context_switch ... crash-utility#1 __schedule () ... crash-utility#2 0xffffffff816a94e9 in schedule () ... crash-utility#3 0xffffffff816a86fd in schedule_hrtimeout_range_clock ... crash-utility#4 0xffffffff816a8733 in schedule_hrtimeout_range ... crash-utility#5 0xffffffff8124bb7e in ep_poll ... crash-utility#6 0xffffffff8124d00d in SYSC_epoll_wait ... crash-utility#7 SyS_epoll_wait ... crash-utility#8 <signal handler called> crash-utility#9 0x00007f0449407923 in ?? () Signed-off-by: Tao Liu <ltao@redhat.com>
…ame" warning The "bogus exception frame" warning was observed again on a specific vmcore, and the remaining frame was truncated on x86_64 machine, when executing the "bt" command as below: crash> bt 0 -c 8 PID: 0 TASK: ffff9948c08f5640 CPU: 8 COMMAND: "swapper/8" #0 [fffffe1788788e58] crash_nmi_callback at ffffffff972672bb #1 [fffffe1788788e68] nmi_handle at ffffffff9722eb8e #2 [fffffe1788788eb0] default_do_nmi at ffffffff97e51cd0 #3 [fffffe1788788ed0] exc_nmi at ffffffff97e51ee1 #4 [fffffe1788788ef0] end_repeat_nmi at ffffffff980015f9 [exception RIP: __update_load_avg_se+13] RIP: ffffffff9736b16d RSP: ffffbec3c08acc78 RFLAGS: 00000046 RAX: 0000000000000000 RBX: ffff994c2f2b1a40 RCX: ffffbec3c08acdc0 RDX: ffff9948e4fe1d80 RSI: ffff994c2f2b1a40 RDI: 0000001d7ad7d55d RBP: ffffbec3c08acc88 R8: 0000001d921fca6f R9: ffff994c2f2b1328 R10: 00000000fffd0010 R11: ffffffff98e060c0 R12: 0000001d7ad7d55d R13: 0000000000000005 R14: ffff994c2f2b19c0 R15: 0000000000000001 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 --- <NMI exception stack> --- #5 [ffffbec3c08acc78] __update_load_avg_se at ffffffff9736b16d #6 [ffffbec3c08acce0] enqueue_entity at ffffffff9735c9ab #7 [ffffbec3c08acd28] enqueue_task_fair at ffffffff9735cef8 ... #18 [ffffbec3c08acf90] blk_complete_reqs at ffffffff977978d0 #19 [ffffbec3c08acfa0] __do_softirq at ffffffff97e66f7a #20 [ffffbec3c08acff0] do_softirq at ffffffff9730f6ef --- <IRQ stack> --- #21 [ffffbec3c022ff18] do_idle at ffffffff97368288 [exception RIP: unknown or invalid address] RIP: 0000000000000000 RSP: 0000000000000000 RFLAGS: 00000000 RAX: 0000000000000000 RBX: 000000089726a2d0 RCX: 0000000000000000 RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000 RBP: ffffffff9726a3dd R8: 0000000000000000 R9: 0000000000000000 R10: ffffffff9720015a R11: e48885e126bc1600 R12: 0000000000000000 R13: ffffffff973684a9 R14: 0000000000000094 R15: 0000000040000000 ORIG_RAX: 0000000000000000 CS: 0000 SS: 0000 bt: WARNING: possibly bogus exception frame crash> Actually there is no exception frame, when called from do_softirq(). With the patch: crash> bt 0 -c 8 ... #18 [ffffbec3c08acf90] blk_complete_reqs at ffffffff977978d0 #19 [ffffbec3c08acfa0] __do_softirq at ffffffff97e66f7a #20 [ffffbec3c08acff0] do_softirq at ffffffff9730f6ef --- <IRQ stack> --- #21 [ffffbec3c022ff28] cpu_startup_entry at ffffffff973684a9 #22 [ffffbec3c022ff38] start_secondary at ffffffff9726a3dd #23 [ffffbec3c022ff50] secondary_startup_64_no_verify at ffffffff9720015a crash> Reported-by: Jie Li <jieli@redhat.com> Signed-off-by: Lianbo Jiang <lijiang@redhat.com>
Previously we can only view the stack unwinding for the tasks which are running on each CPUs. This patch will enable the ability to view arbitrary tasks stack unwinding. After crash get initialized, "info threads" will output like the following: crash> info threads Id Target Id Frame 1 CPU 0 native_safe_halt () at arch/x86/include/asm/irqflags.h:54 ... * 8 CPU 7 blk_mq_rq_timed_out (req=0xffff880fdb246000, reserved=reserved@entry=false) at block/blk-mq.c:640 ... 13 CPU 12 <unavailable> in ?? () 14 CPU 13 native_safe_halt () at arch/x86/include/asm/irqflags.h:54 ... crash> ps PID PPID CPU TASK ST %MEM VSZ RSS COMM > 0 0 0 ffffffff819f9480 RU 0.0 0 0 [swapper/0] > 0 0 1 ffff880169411fa0 RU 0.0 0 0 [swapper/1] ... 0 0 23 ffff8801694e0000 RU 0.0 0 0 [swapper/23] 1 0 13 ffff880169b30000 IN 0.0 193052 4180 systemd "info threads" show the tasks which are currently running on each CPU. If we'd like to view systemd task's stack unwinding, which is inactive status, we do the following: crash> set 1 or crash> set ffff880169b30000 Then the register cache of systemd will be swapped into CPU 13: crash> info threads Id Target Id Frame 1 CPU 0 native_safe_halt () at arch/x86/include/asm/irqflags.h:54 ... 8 CPU 7 blk_mq_rq_timed_out (req=0xffff880fdb246000, reserved=reserved@entry=false) at block/blk-mq.c:640 ... 13 CPU 12 <unavailable> in ?? () * 14 CPU 13 0xffffffff816a8f65 in context_switch (rq=0x0, next=0x0, prev=0xffff880169b30000) at kernel/sched/core.c:2527 ... And we can view the stack unwinding of systemd: crash> bt PID: 1 TASK: ffff880169b30000 CPU: 13 COMMAND: "systemd" #0 [ffff880169b3bd58] __schedule at ffffffff816a8f65 crash-utility#1 [ffff880169b3bdc0] schedule at ffffffff816a94e9 crash-utility#2 [ffff880169b3bdd0] schedule_hrtimeout_range_clock at ffffffff816a86fd crash-utility#3 [ffff880169b3be68] schedule_hrtimeout_range at ffffffff816a8733 crash-utility#4 [ffff880169b3be78] ep_poll at ffffffff8124bb7e crash-utility#5 [ffff880169b3bf30] sys_epoll_wait at ffffffff8124d00d crash-utility#6 [ffff880169b3bf80] system_call_fastpath at ffffffff816b5009 RIP: 00007f0449407923 RSP: 00007ffc35a3c378 RFLAGS: 00010246 RAX: 00000000000000e8 RBX: ffffffff816b5009 RCX: 0000000000000071 RDX: 000000000000001d RSI: 00007ffc35a3d5a0 RDI: 0000000000000004 RBP: 00007ffc35a3d810 R8: 0000000000000000 R9: 0000000000000000 R10: 00000000ffffffff R11: 0000000000000293 R12: 0000563ca2ebe980 R13: 0000000000000003 R14: ffffffffffffffff R15: 0000000000000001 ORIG_RAX: 00000000000000e8 CS: 0033 SS: 002b crash> gdb bt #0 0xffffffff816a8f65 in context_switch (rq=0x0, next=0x0, prev=0xffff880169b30000) at kernel/sched/core.c:2527 crash-utility#1 __schedule () at kernel/sched/core.c:3540 crash-utility#2 0xffffffff816a94e9 in schedule () at kernel/sched/core.c:3577 crash-utility#3 0xffffffff816a86fd in schedule_hrtimeout_range_clock (expires=expires@entry=0x0, delta=delta@entry=0, mode=mode@entry=HRTIMER_MODE_ABS, clock=clock@entry=1) at kernel/hrtimer.c:1724 crash-utility#4 0xffffffff816a8733 in schedule_hrtimeout_range (expires=expires@entry=0x0, delta=delta@entry=0, mode=mode@entry=HRTIMER_MODE_ABS) at kernel/hrtimer.c:1778 crash-utility#5 0xffffffff8124bb7e in ep_poll (ep=0xffff880fd861f8c0, events=events@entry=0x7ffc35a3d5a0, maxevents=maxevents@entry=29, timeout=timeout@entry=-1) at fs/eventpoll.c:1669 crash-utility#6 0xffffffff8124d00d in SYSC_epoll_wait (timeout=<optimized out>, maxevents=29, events=<optimized out>, epfd=<optimized out>) at fs/eventpoll.c:2043 crash-utility#7 SyS_epoll_wait (epfd=<optimized out>, events=140721208415648, maxevents=29, timeout=4294967295) at fs/eventpoll.c:2008 crash-utility#8 <signal handler called> crash-utility#9 0x00007f0449407923 in ?? () Signed-off-by: Tao Liu <ltao@redhat.com> Signed-off-by: Aditya Gupta <adityag@linux.ibm.com>
The stack unwinding is for kernel addresses only. If non-kernel address encountered, it is usually a user space address, or non-address value like a function call parameter. So stopping stack unwinding at non-kernel address will decrease the invalid unwind results. Before: crash> gdb bt #0 0xffffffff816a8f65 in context_switch ... crash-utility#1 __schedule () ... crash-utility#2 0xffffffff816a94e9 in schedule ... crash-utility#3 0xffffffff816a86fd in schedule_hrtimeout_range_clock ... crash-utility#4 0xffffffff816a8733 in schedule_hrtimeout_range ... crash-utility#5 0xffffffff8124bb7e in ep_poll ... crash-utility#6 0xffffffff8124d00d in SYSC_epoll_wait ... crash-utility#7 SyS_epoll_wait ... crash-utility#8 <signal handler called> crash-utility#9 0x00007f0449407923 in ?? () crash-utility#10 0xffff880100000001 in ?? () crash-utility#11 0xffff880169b3c010 in ?? () crash-utility#12 0x0000000000000040 in irq_stack_union () crash-utility#13 0xffff880169b3c058 in ?? () crash-utility#14 0xffff880169b3c048 in ?? () crash-utility#15 0xffff880169b3c050 in ?? () crash-utility#16 0x0000000000000000 in ?? () After: crash> gdb bt #0 0xffffffff816a8f65 in context_switch ... crash-utility#1 __schedule () ... crash-utility#2 0xffffffff816a94e9 in schedule () ... crash-utility#3 0xffffffff816a86fd in schedule_hrtimeout_range_clock ... crash-utility#4 0xffffffff816a8733 in schedule_hrtimeout_range ... crash-utility#5 0xffffffff8124bb7e in ep_poll ... crash-utility#6 0xffffffff8124d00d in SYSC_epoll_wait ... crash-utility#7 SyS_epoll_wait ... crash-utility#8 <signal handler called> crash-utility#9 0x00007f0449407923 in ?? () Signed-off-by: Tao Liu <ltao@redhat.com>
Currently, gdb passthroughs of 'bt', 'frame', 'up', 'down', 'info locals' don't work. This is due to gdb not knowing the register values to unwind the stack frames Every gdb passthrough goes through `gdb_interface`. And then, gdb expects `crash_target::fetch_registers` to give it the register values, which is dependent on `machdep->get_cpu_reg` to read the register values for specific architecture. ---------------------------- gdb passthrough (eg. "bt") | | crash -------------------------> | | | gdb_interface | | | | | | ---------------------- | fetch_registers | | | | crash_target<-------------------------+--| gdb | | --------------------------+->| | | Registers (SP,NIP, etc.)| | | | | | | | | ---------------------- | ---------------------------- Implement `machdep->get_cpu_reg` on PPC64, so that crash provides the register values to gdb to unwind stack frames properly With these changes, on powerpc, 'bt' command output in gdb mode, will look like this: gdb> bt #0 0xc0000000002a53e8 in crash_setup_regs (oldregs=<optimized out>, newregs=0xc00000000486f8d8) at ./arch/powerpc/include/asm/kexec.h:69 crash-utility#1 __crash_kexec (regs=<optimized out>) at kernel/kexec_core.c:974 crash-utility#2 0xc000000000168918 in panic (fmt=<optimized out>) at kernel/panic.c:358 crash-utility#3 0xc000000000b735f8 in sysrq_handle_crash (key=<optimized out>) at drivers/tty/sysrq.c:155 crash-utility#4 0xc000000000b742cc in __handle_sysrq (key=key@entry=99, check_mask=check_mask@entry=false) at drivers/tty/sysrq.c:602 crash-utility#5 0xc000000000b7506c in write_sysrq_trigger (file=<optimized out>, buf=<optimized out>, count=2, ppos=<optimized out>) at drivers/tty/sysrq.c:1163 crash-utility#6 0xc00000000069a7bc in pde_write (ppos=<optimized out>, count=<optimized out>, buf=<optimized out>, file=<optimized out>, pde=0xc000000009ed3a80) at fs/proc/inode.c:340 crash-utility#7 proc_reg_write (file=<optimized out>, buf=<optimized out>, count=<optimized out>, ppos=<optimized out>) at fs/proc/inode.c:352 crash-utility#8 0xc0000000005b3bbc in vfs_write (file=file@entry=0xc00000009dda7d00, buf=buf@entry=0xebcfc7c6040 <error: Cannot access memory at address 0xebcfc7c6040>, count=count@entry=2, pos=pos@entry=0xc00000000486fda0) at fs/read_write.c:582 instead of earlier output without this patch: gdb> bt #0 <unavailable> in ?? () Backtrace stopped: previous frame identical to this frame (corrupt stack?) Also, 'get_dumpfile_regs' has been introduced to get registers from multiple supported vmcore formats. Correspondingly a flag 'BT_NO_PRINT_REGS' has been introduced to tell helper functions to get registers, to not print registers with every call to backtrace in gdb. Note: This feature to support GDB unwinding doesn't support live debugging Cc: Sourabh Jain <sourabhjain@linux.ibm.com> Cc: Hari Bathini <hbathini@linux.ibm.com> Cc: Mahesh J Salgaonkar <mahesh@linux.ibm.com> Cc: Naveen N. Rao <naveen.n.rao@linux.vnet.ibm.com> Cc: Lianbo Jiang <lijiang@redhat.com> Cc: HAGIO KAZUHITO(萩尾 一仁) <k-hagio-ab@nec.com> Improved-by: Tao Liu <ltao@redhat.com> Signed-off-by: Aditya Gupta <adityag@linux.ibm.com>
Previously we can only view the stack unwinding for the tasks which are running on each CPUs. This patch will enable the ability to view arbitrary tasks stack unwinding. After crash get initialized, "info threads" will output like the following: crash> info threads Id Target Id Frame 1 CPU 0 native_safe_halt () at arch/x86/include/asm/irqflags.h:54 ... * 8 CPU 7 blk_mq_rq_timed_out (req=0xffff880fdb246000, reserved=reserved@entry=false) at block/blk-mq.c:640 ... 13 CPU 12 <unavailable> in ?? () 14 CPU 13 native_safe_halt () at arch/x86/include/asm/irqflags.h:54 ... crash> ps PID PPID CPU TASK ST %MEM VSZ RSS COMM > 0 0 0 ffffffff819f9480 RU 0.0 0 0 [swapper/0] > 0 0 1 ffff880169411fa0 RU 0.0 0 0 [swapper/1] ... 0 0 23 ffff8801694e0000 RU 0.0 0 0 [swapper/23] 1 0 13 ffff880169b30000 IN 0.0 193052 4180 systemd "info threads" show the tasks which are currently running on each CPU. If we'd like to view systemd task's stack unwinding, which is inactive status, we do the following: crash> set 1 or crash> set ffff880169b30000 Then the register cache of systemd will be swapped into CPU 13: crash> info threads Id Target Id Frame 1 CPU 0 native_safe_halt () at arch/x86/include/asm/irqflags.h:54 ... 8 CPU 7 blk_mq_rq_timed_out (req=0xffff880fdb246000, reserved=reserved@entry=false) at block/blk-mq.c:640 ... 13 CPU 12 <unavailable> in ?? () * 14 CPU 13 0xffffffff816a8f65 in context_switch (rq=0x0, next=0x0, prev=0xffff880169b30000) at kernel/sched/core.c:2527 ... And we can view the stack unwinding of systemd: crash> bt PID: 1 TASK: ffff880169b30000 CPU: 13 COMMAND: "systemd" #0 [ffff880169b3bd58] __schedule at ffffffff816a8f65 crash-utility#1 [ffff880169b3bdc0] schedule at ffffffff816a94e9 crash-utility#2 [ffff880169b3bdd0] schedule_hrtimeout_range_clock at ffffffff816a86fd crash-utility#3 [ffff880169b3be68] schedule_hrtimeout_range at ffffffff816a8733 crash-utility#4 [ffff880169b3be78] ep_poll at ffffffff8124bb7e crash-utility#5 [ffff880169b3bf30] sys_epoll_wait at ffffffff8124d00d crash-utility#6 [ffff880169b3bf80] system_call_fastpath at ffffffff816b5009 RIP: 00007f0449407923 RSP: 00007ffc35a3c378 RFLAGS: 00010246 RAX: 00000000000000e8 RBX: ffffffff816b5009 RCX: 0000000000000071 RDX: 000000000000001d RSI: 00007ffc35a3d5a0 RDI: 0000000000000004 RBP: 00007ffc35a3d810 R8: 0000000000000000 R9: 0000000000000000 R10: 00000000ffffffff R11: 0000000000000293 R12: 0000563ca2ebe980 R13: 0000000000000003 R14: ffffffffffffffff R15: 0000000000000001 ORIG_RAX: 00000000000000e8 CS: 0033 SS: 002b crash> gdb bt #0 0xffffffff816a8f65 in context_switch (rq=0x0, next=0x0, prev=0xffff880169b30000) at kernel/sched/core.c:2527 crash-utility#1 __schedule () at kernel/sched/core.c:3540 crash-utility#2 0xffffffff816a94e9 in schedule () at kernel/sched/core.c:3577 crash-utility#3 0xffffffff816a86fd in schedule_hrtimeout_range_clock (expires=expires@entry=0x0, delta=delta@entry=0, mode=mode@entry=HRTIMER_MODE_ABS, clock=clock@entry=1) at kernel/hrtimer.c:1724 crash-utility#4 0xffffffff816a8733 in schedule_hrtimeout_range (expires=expires@entry=0x0, delta=delta@entry=0, mode=mode@entry=HRTIMER_MODE_ABS) at kernel/hrtimer.c:1778 crash-utility#5 0xffffffff8124bb7e in ep_poll (ep=0xffff880fd861f8c0, events=events@entry=0x7ffc35a3d5a0, maxevents=maxevents@entry=29, timeout=timeout@entry=-1) at fs/eventpoll.c:1669 crash-utility#6 0xffffffff8124d00d in SYSC_epoll_wait (timeout=<optimized out>, maxevents=29, events=<optimized out>, epfd=<optimized out>) at fs/eventpoll.c:2043 crash-utility#7 SyS_epoll_wait (epfd=<optimized out>, events=140721208415648, maxevents=29, timeout=4294967295) at fs/eventpoll.c:2008 crash-utility#8 <signal handler called> crash-utility#9 0x00007f0449407923 in ?? () Signed-off-by: Tao Liu <ltao@redhat.com> Signed-off-by: Aditya Gupta <adityag@linux.ibm.com>
The stack unwinding is for kernel addresses only. If non-kernel address encountered, it is usually a user space address, or non-address value like a function call parameter. So stopping stack unwinding at non-kernel address will decrease the invalid unwind results. Before: crash> gdb bt #0 0xffffffff816a8f65 in context_switch ... crash-utility#1 __schedule () ... crash-utility#2 0xffffffff816a94e9 in schedule ... crash-utility#3 0xffffffff816a86fd in schedule_hrtimeout_range_clock ... crash-utility#4 0xffffffff816a8733 in schedule_hrtimeout_range ... crash-utility#5 0xffffffff8124bb7e in ep_poll ... crash-utility#6 0xffffffff8124d00d in SYSC_epoll_wait ... crash-utility#7 SyS_epoll_wait ... crash-utility#8 <signal handler called> crash-utility#9 0x00007f0449407923 in ?? () crash-utility#10 0xffff880100000001 in ?? () crash-utility#11 0xffff880169b3c010 in ?? () crash-utility#12 0x0000000000000040 in irq_stack_union () crash-utility#13 0xffff880169b3c058 in ?? () crash-utility#14 0xffff880169b3c048 in ?? () crash-utility#15 0xffff880169b3c050 in ?? () crash-utility#16 0x0000000000000000 in ?? () After: crash> gdb bt #0 0xffffffff816a8f65 in context_switch ... crash-utility#1 __schedule () ... crash-utility#2 0xffffffff816a94e9 in schedule () ... crash-utility#3 0xffffffff816a86fd in schedule_hrtimeout_range_clock ... crash-utility#4 0xffffffff816a8733 in schedule_hrtimeout_range ... crash-utility#5 0xffffffff8124bb7e in ep_poll ... crash-utility#6 0xffffffff8124d00d in SYSC_epoll_wait ... crash-utility#7 SyS_epoll_wait ... crash-utility#8 <signal handler called> crash-utility#9 0x00007f0449407923 in ?? () Signed-off-by: Tao Liu <ltao@redhat.com>
Currently, gdb passthroughs of 'bt', 'frame', 'up', 'down', 'info locals' don't work. This is due to gdb not knowing the register values to unwind the stack frames Every gdb passthrough goes through `gdb_interface`. And then, gdb expects `crash_target::fetch_registers` to give it the register values, which is dependent on `machdep->get_cpu_reg` to read the register values for specific architecture. ---------------------------- gdb passthrough (eg. "bt") | | crash -------------------------> | | | gdb_interface | | | | | | ---------------------- | fetch_registers | | | | crash_target<-------------------------+--| gdb | | --------------------------+->| | | Registers (SP,NIP, etc.)| | | | | | | | | ---------------------- | ---------------------------- Implement `machdep->get_cpu_reg` on PPC64, so that crash provides the register values to gdb to unwind stack frames properly With these changes, on powerpc, 'bt' command output in gdb mode, will look like this: gdb> bt #0 0xc0000000002a53e8 in crash_setup_regs (oldregs=<optimized out>, newregs=0xc00000000486f8d8) at ./arch/powerpc/include/asm/kexec.h:69 crash-utility#1 __crash_kexec (regs=<optimized out>) at kernel/kexec_core.c:974 crash-utility#2 0xc000000000168918 in panic (fmt=<optimized out>) at kernel/panic.c:358 crash-utility#3 0xc000000000b735f8 in sysrq_handle_crash (key=<optimized out>) at drivers/tty/sysrq.c:155 crash-utility#4 0xc000000000b742cc in __handle_sysrq (key=key@entry=99, check_mask=check_mask@entry=false) at drivers/tty/sysrq.c:602 crash-utility#5 0xc000000000b7506c in write_sysrq_trigger (file=<optimized out>, buf=<optimized out>, count=2, ppos=<optimized out>) at drivers/tty/sysrq.c:1163 crash-utility#6 0xc00000000069a7bc in pde_write (ppos=<optimized out>, count=<optimized out>, buf=<optimized out>, file=<optimized out>, pde=0xc000000009ed3a80) at fs/proc/inode.c:340 crash-utility#7 proc_reg_write (file=<optimized out>, buf=<optimized out>, count=<optimized out>, ppos=<optimized out>) at fs/proc/inode.c:352 crash-utility#8 0xc0000000005b3bbc in vfs_write (file=file@entry=0xc00000009dda7d00, buf=buf@entry=0xebcfc7c6040 <error: Cannot access memory at address 0xebcfc7c6040>, count=count@entry=2, pos=pos@entry=0xc00000000486fda0) at fs/read_write.c:582 instead of earlier output without this patch: gdb> bt #0 <unavailable> in ?? () Backtrace stopped: previous frame identical to this frame (corrupt stack?) Also, 'get_dumpfile_regs' has been introduced to get registers from multiple supported vmcore formats. Correspondingly a flag 'BT_NO_PRINT_REGS' has been introduced to tell helper functions to get registers, to not print registers with every call to backtrace in gdb. Note: This feature to support GDB unwinding doesn't support live debugging Cc: Sourabh Jain <sourabhjain@linux.ibm.com> Cc: Hari Bathini <hbathini@linux.ibm.com> Cc: Mahesh J Salgaonkar <mahesh@linux.ibm.com> Cc: Naveen N. Rao <naveen.n.rao@linux.vnet.ibm.com> Cc: Lianbo Jiang <lijiang@redhat.com> Cc: HAGIO KAZUHITO(萩尾 一仁) <k-hagio-ab@nec.com> Improved-by: Tao Liu <ltao@redhat.com> Signed-off-by: Aditya Gupta <adityag@linux.ibm.com>
The stack unwinding is for kernel addresses only. If non-kernel address encountered, it is usually a user space address, or non-address value like a function call parameter. So stopping stack unwinding at non-kernel address will decrease the invalid unwind results. Before: crash> gdb bt #0 0xffffffff816a8f65 in context_switch ... crash-utility#1 __schedule () ... crash-utility#2 0xffffffff816a94e9 in schedule ... crash-utility#3 0xffffffff816a86fd in schedule_hrtimeout_range_clock ... crash-utility#4 0xffffffff816a8733 in schedule_hrtimeout_range ... crash-utility#5 0xffffffff8124bb7e in ep_poll ... crash-utility#6 0xffffffff8124d00d in SYSC_epoll_wait ... crash-utility#7 SyS_epoll_wait ... crash-utility#8 <signal handler called> crash-utility#9 0x00007f0449407923 in ?? () crash-utility#10 0xffff880100000001 in ?? () crash-utility#11 0xffff880169b3c010 in ?? () crash-utility#12 0x0000000000000040 in irq_stack_union () crash-utility#13 0xffff880169b3c058 in ?? () crash-utility#14 0xffff880169b3c048 in ?? () crash-utility#15 0xffff880169b3c050 in ?? () crash-utility#16 0x0000000000000000 in ?? () After: crash> gdb bt #0 0xffffffff816a8f65 in context_switch ... crash-utility#1 __schedule () ... crash-utility#2 0xffffffff816a94e9 in schedule () ... crash-utility#3 0xffffffff816a86fd in schedule_hrtimeout_range_clock ... crash-utility#4 0xffffffff816a8733 in schedule_hrtimeout_range ... crash-utility#5 0xffffffff8124bb7e in ep_poll ... crash-utility#6 0xffffffff8124d00d in SYSC_epoll_wait ... crash-utility#7 SyS_epoll_wait ... crash-utility#8 <signal handler called> crash-utility#9 0x00007f0449407923 in ?? () Signed-off-by: Tao Liu <ltao@redhat.com>
Currently, gdb passthroughs of 'bt', 'frame', 'up', 'down', 'info locals' don't work. This is due to gdb not knowing the register values to unwind the stack frames Every gdb passthrough goes through `gdb_interface`. And then, gdb expects `crash_target::fetch_registers` to give it the register values, which is dependent on `machdep->get_cpu_reg` to read the register values for specific architecture. ---------------------------- gdb passthrough (eg. "bt") | | crash -------------------------> | | | gdb_interface | | | | | | ---------------------- | fetch_registers | | | | crash_target<-------------------------+--| gdb | | --------------------------+->| | | Registers (SP,NIP, etc.)| | | | | | | | | ---------------------- | ---------------------------- Implement `machdep->get_cpu_reg` on PPC64, so that crash provides the register values to gdb to unwind stack frames properly With these changes, on powerpc, 'bt' command output in gdb mode, will look like this: gdb> bt #0 0xc0000000002a53e8 in crash_setup_regs (oldregs=<optimized out>, newregs=0xc00000000486f8d8) at ./arch/powerpc/include/asm/kexec.h:69 crash-utility#1 __crash_kexec (regs=<optimized out>) at kernel/kexec_core.c:974 crash-utility#2 0xc000000000168918 in panic (fmt=<optimized out>) at kernel/panic.c:358 crash-utility#3 0xc000000000b735f8 in sysrq_handle_crash (key=<optimized out>) at drivers/tty/sysrq.c:155 crash-utility#4 0xc000000000b742cc in __handle_sysrq (key=key@entry=99, check_mask=check_mask@entry=false) at drivers/tty/sysrq.c:602 crash-utility#5 0xc000000000b7506c in write_sysrq_trigger (file=<optimized out>, buf=<optimized out>, count=2, ppos=<optimized out>) at drivers/tty/sysrq.c:1163 crash-utility#6 0xc00000000069a7bc in pde_write (ppos=<optimized out>, count=<optimized out>, buf=<optimized out>, file=<optimized out>, pde=0xc000000009ed3a80) at fs/proc/inode.c:340 crash-utility#7 proc_reg_write (file=<optimized out>, buf=<optimized out>, count=<optimized out>, ppos=<optimized out>) at fs/proc/inode.c:352 crash-utility#8 0xc0000000005b3bbc in vfs_write (file=file@entry=0xc00000009dda7d00, buf=buf@entry=0xebcfc7c6040 <error: Cannot access memory at address 0xebcfc7c6040>, count=count@entry=2, pos=pos@entry=0xc00000000486fda0) at fs/read_write.c:582 instead of earlier output without this patch: gdb> bt #0 <unavailable> in ?? () Backtrace stopped: previous frame identical to this frame (corrupt stack?) Also, 'get_dumpfile_regs' has been introduced to get registers from multiple supported vmcore formats. Correspondingly a flag 'BT_NO_PRINT_REGS' has been introduced to tell helper functions to get registers, to not print registers with every call to backtrace in gdb. Note: This feature to support GDB unwinding doesn't support live debugging Cc: Sourabh Jain <sourabhjain@linux.ibm.com> Cc: Hari Bathini <hbathini@linux.ibm.com> Cc: Mahesh J Salgaonkar <mahesh@linux.ibm.com> Cc: Naveen N. Rao <naveen.n.rao@linux.vnet.ibm.com> Cc: Lianbo Jiang <lijiang@redhat.com> Cc: HAGIO KAZUHITO(萩尾 一仁) <k-hagio-ab@nec.com> Improved-by: Tao Liu <ltao@redhat.com> Signed-off-by: Aditya Gupta <adityag@linux.ibm.com>
The following segmentation fault occurred during session initialization: $ crash vmlinx vmcore ... please wait... (determining panic task)Segmentation fault Here is the backtrace of the crash-utility: (gdb) bt #0 value_search_module_6_4 (value=18446603338276298752, offset=0x7ffffffface0) at symbols.c:5564 #1 0x0000555555812bd0 in value_to_symstr (value=18446603338276298752, buf=buf@entry=0x7fffffffb9c0 "", radix=10, radix@entry=0) at symbols.c:5872 #2 0x00005555557694a2 in display_memory (addr=<optimized out>, count=2048, flag=208, memtype=memtype@entry=1, opt=opt@entry=0x0) at memory.c:1740 #3 0x0000555555769e1f in raw_stack_dump (stackbase=<optimized out>, size=<optimized out>) at memory.c:2194 #4 0x00005555557923ff in get_active_set_panic_task () at task.c:8639 #5 0x00005555557930d2 in get_dumpfile_panic_task () at task.c:7628 #6 0x00005555557a89d3 in panic_search () at task.c:7380 #7 get_panic_context () at task.c:6267 #8 task_init () at task.c:687 #9 0x00005555557305b3 in main_loop () at main.c:787 ... This is due to lack of existence check on module symbol table. Not all mod_mem_type will be existent for a module, e.g. in the following module case: (gdb) p lm->symtable[0] $1 = (struct syment *) 0x4dcbad0 (gdb) p lm->symtable[1] $2 = (struct syment *) 0x4dcbb70 (gdb) p lm->symtable[2] $3 = (struct syment *) 0x4dcbc10 (gdb) p lm->symtable[3] $4 = (struct syment *) 0x0 (gdb) p lm->symtable[4] $5 = (struct syment *) 0x4dcbcb0 (gdb) p lm->symtable[5] $6 = (struct syment *) 0x4dcbd00 (gdb) p lm->symtable[6] $7 = (struct syment *) 0x0 MOD_RO_AFTER_INIT(3) and MOD_INIT_RODATA(6) do not exist, which should be skipped, otherwise the segmentation fault will happen. Fixes: 7750e61 ("Support module memory layout change on Linux 6.4") Closes: #176 Reported-by: Naveen Chaudhary <naveenchaudhary2010@hotmail.com> Signed-off-by: Tao Liu <ltao@redhat.com>
The stack unwinding is for kernel addresses only. If non-kernel address encountered, it is usually a user space address, or non-address value like a function call parameter. So stopping stack unwinding at non-kernel address will decrease the invalid unwind results. Before: crash> gdb bt #0 0xffffffff816a8f65 in context_switch ... crash-utility#1 __schedule () ... crash-utility#2 0xffffffff816a94e9 in schedule ... crash-utility#3 0xffffffff816a86fd in schedule_hrtimeout_range_clock ... crash-utility#4 0xffffffff816a8733 in schedule_hrtimeout_range ... crash-utility#5 0xffffffff8124bb7e in ep_poll ... crash-utility#6 0xffffffff8124d00d in SYSC_epoll_wait ... crash-utility#7 SyS_epoll_wait ... crash-utility#8 <signal handler called> crash-utility#9 0x00007f0449407923 in ?? () crash-utility#10 0xffff880100000001 in ?? () crash-utility#11 0xffff880169b3c010 in ?? () crash-utility#12 0x0000000000000040 in irq_stack_union () crash-utility#13 0xffff880169b3c058 in ?? () crash-utility#14 0xffff880169b3c048 in ?? () crash-utility#15 0xffff880169b3c050 in ?? () crash-utility#16 0x0000000000000000 in ?? () After: crash> gdb bt #0 0xffffffff816a8f65 in context_switch ... crash-utility#1 __schedule () ... crash-utility#2 0xffffffff816a94e9 in schedule () ... crash-utility#3 0xffffffff816a86fd in schedule_hrtimeout_range_clock ... crash-utility#4 0xffffffff816a8733 in schedule_hrtimeout_range ... crash-utility#5 0xffffffff8124bb7e in ep_poll ... crash-utility#6 0xffffffff8124d00d in SYSC_epoll_wait ... crash-utility#7 SyS_epoll_wait ... crash-utility#8 <signal handler called> crash-utility#9 0x00007f0449407923 in ?? () Cc: Sourabh Jain <sourabhjain@linux.ibm.com> Cc: Hari Bathini <hbathini@linux.ibm.com> Cc: Mahesh J Salgaonkar <mahesh@linux.ibm.com> Cc: Naveen N. Rao <naveen.n.rao@linux.vnet.ibm.com> Cc: Lianbo Jiang <lijiang@redhat.com> Cc: HAGIO KAZUHITO(萩尾 一仁) <k-hagio-ab@nec.com> Cc: Tao Liu <ltao@redhat.com> Cc: Alexey Makhalov <alexey.makhalov@broadcom.com> Signed-off-by: Tao Liu <ltao@redhat.com>
Currently, gdb passthroughs of 'bt', 'frame', 'up', 'down', 'info locals' don't work. This is due to gdb not knowing the register values to unwind the stack frames Every gdb passthrough goes through `gdb_interface`. And then, gdb expects `crash_target::fetch_registers` to give it the register values, which is dependent on `machdep->get_cpu_reg` to read the register values for specific architecture. ---------------------------- gdb passthrough (eg. "bt") | | crash -------------------------> | | | gdb_interface | | | | | | ---------------------- | fetch_registers | | | | crash_target<-------------------------+--| gdb | | --------------------------+->| | | Registers (SP,NIP, etc.)| | | | | | | | | ---------------------- | ---------------------------- Implement `machdep->get_cpu_reg` on PPC64, so that crash provides the register values to gdb to unwind stack frames properly With these changes, on powerpc, 'bt' command output in gdb mode, will look like this: gdb> bt #0 0xc0000000002a53e8 in crash_setup_regs (oldregs=<optimized out>, newregs=0xc00000000486f8d8) at ./arch/powerpc/include/asm/kexec.h:69 crash-utility#1 __crash_kexec (regs=<optimized out>) at kernel/kexec_core.c:974 crash-utility#2 0xc000000000168918 in panic (fmt=<optimized out>) at kernel/panic.c:358 crash-utility#3 0xc000000000b735f8 in sysrq_handle_crash (key=<optimized out>) at drivers/tty/sysrq.c:155 crash-utility#4 0xc000000000b742cc in __handle_sysrq (key=key@entry=99, check_mask=check_mask@entry=false) at drivers/tty/sysrq.c:602 crash-utility#5 0xc000000000b7506c in write_sysrq_trigger (file=<optimized out>, buf=<optimized out>, count=2, ppos=<optimized out>) at drivers/tty/sysrq.c:1163 crash-utility#6 0xc00000000069a7bc in pde_write (ppos=<optimized out>, count=<optimized out>, buf=<optimized out>, file=<optimized out>, pde=0xc000000009ed3a80) at fs/proc/inode.c:340 crash-utility#7 proc_reg_write (file=<optimized out>, buf=<optimized out>, count=<optimized out>, ppos=<optimized out>) at fs/proc/inode.c:352 crash-utility#8 0xc0000000005b3bbc in vfs_write (file=file@entry=0xc00000009dda7d00, buf=buf@entry=0xebcfc7c6040 <error: Cannot access memory at address 0xebcfc7c6040>, count=count@entry=2, pos=pos@entry=0xc00000000486fda0) at fs/read_write.c:582 instead of earlier output without this patch: gdb> bt #0 <unavailable> in ?? () Backtrace stopped: previous frame identical to this frame (corrupt stack?) Also, 'get_dumpfile_regs' has been introduced to get registers from multiple supported vmcore formats. Correspondingly a flag 'BT_NO_PRINT_REGS' has been introduced to tell helper functions to get registers, to not print registers with every call to backtrace in gdb. Note: This feature to support GDB unwinding doesn't support live debugging Cc: Sourabh Jain <sourabhjain@linux.ibm.com> Cc: Hari Bathini <hbathini@linux.ibm.com> Cc: Mahesh J Salgaonkar <mahesh@linux.ibm.com> Cc: Naveen N. Rao <naveen.n.rao@linux.vnet.ibm.com> Cc: Lianbo Jiang <lijiang@redhat.com> Cc: HAGIO KAZUHITO(萩尾 一仁) <k-hagio-ab@nec.com> Cc: Tao Liu <ltao@redhat.com> Cc: Alexey Makhalov <alexey.makhalov@broadcom.com> Improved-by: Tao Liu <ltao@redhat.com> Signed-off-by: Aditya Gupta <adityag@linux.ibm.com>
With Kernel commit 65c9cc9e2c14 ("x86/fred: Reserve space for the FRED stack frame") in Linux 6.9-rc1 and later, x86_64 will add extra padding ('TOP_OF_KERNEL_STACK_PADDING (2 * 8)', see: arch/x86/include/asm\ /thread_info.h,) for kernel stack when the CONFIG_X86_FRED is enabled. As a result, the pt_regs will be moved downwards due to the offset of padding, and the values of registers read from pt_regs will be incorrect as below. Without the patch: crash> bt PID: 2040 TASK: ffff969136fc4180 CPU: 16 COMMAND: "bash" #0 [ffffa996409aba38] machine_kexec at ffffffff9f881eb7 #1 [ffffa996409aba90] __crash_kexec at ffffffff9fa1e49e #2 [ffffa996409abb48] panic at ffffffff9f91a6cd #3 [ffffa996409abbc8] sysrq_handle_crash at ffffffffa0015076 #4 [ffffa996409abbd0] __handle_sysrq at ffffffffa0015640 #5 [ffffa996409abc00] write_sysrq_trigger at ffffffffa0015ce5 #6 [ffffa996409abc28] proc_reg_write at ffffffff9fd35bf5 #7 [ffffa996409abc40] vfs_write at ffffffff9fc8d462 #8 [ffffa996409abcd0] ksys_write at ffffffff9fc8dadf #9 [ffffa996409abd08] do_syscall_64 at ffffffffa0517429 #10 [ffffa996409abf40] entry_SYSCALL_64_after_hwframe at ffffffffa060012b [exception RIP: unknown or invalid address] RIP: 0000000000000246 RSP: 0000000000000000 RFLAGS: 0000002b RAX: 0000000000000002 RBX: 00007f9b9f5b13e0 RCX: 000055cee7486fb0 RDX: 0000000000000001 RSI: 0000000000000001 RDI: 00007f9b9f4fda57 RBP: 0000000000000246 R8: 00007f9b9f4fda57 R9: ffffffffffffffda R10: 0000000000000000 R11: 00007f9b9f5b14e0 R12: 0000000000000002 R13: 000055cee7486fb0 R14: 0000000000000002 R15: 00007f9b9f5fb780 ORIG_RAX: 0000000000000033 CS: 7ffe65327978 SS: 0000 bt: WARNING: possibly bogus exception frame crash> With the patch: crash> bt PID: 2040 TASK: ffff969136fc4180 CPU: 16 COMMAND: "bash" #0 [ffffa996409aba38] machine_kexec at ffffffff9f881eb7 #1 [ffffa996409aba90] __crash_kexec at ffffffff9fa1e49e #2 [ffffa996409abb48] panic at ffffffff9f91a6cd #3 [ffffa996409abbc8] sysrq_handle_crash at ffffffffa0015076 #4 [ffffa996409abbd0] __handle_sysrq at ffffffffa0015640 #5 [ffffa996409abc00] write_sysrq_trigger at ffffffffa0015ce5 #6 [ffffa996409abc28] proc_reg_write at ffffffff9fd35bf5 #7 [ffffa996409abc40] vfs_write at ffffffff9fc8d462 #8 [ffffa996409abcd0] ksys_write at ffffffff9fc8dadf #9 [ffffa996409abd08] do_syscall_64 at ffffffffa0517429 #10 [ffffa996409abf40] entry_SYSCALL_64_after_hwframe at ffffffffa060012b RIP: 00007f9b9f4fda57 RSP: 00007ffe65327978 RFLAGS: 00000246 RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007f9b9f4fda57 RDX: 0000000000000002 RSI: 000055cee7486fb0 RDI: 0000000000000001 RBP: 000055cee7486fb0 R8: 0000000000000000 R9: 00007f9b9f5b14e0 R10: 00007f9b9f5b13e0 R11: 0000000000000246 R12: 0000000000000002 R13: 00007f9b9f5fb780 R14: 0000000000000002 R15: 00007f9b9f5f69e0 ORIG_RAX: 0000000000000001 CS: 0033 SS: 002b crash> Link: https://www.mail-archive.com/devel@lists.crash-utility.osci.io/msg00754.html Signed-off-by: Lianbo Jiang <lijiang@redhat.com> Signed-off-by: Tao Liu <ltao@redhat.com>
The commit 48764a1 may cause a regression issue when the CONFIG_X86_FRED is not enabled, this is because the SIZE(fred_frame) will call the SIZE_verify() to determine if the fred_frame is valid, otherwise it will emit an error: crash> bt 1 bt: invalid structure size: fred_frame FILE: x86_64.c LINE: 4089 FUNCTION: x86_64_low_budget_back_trace_cmd() [/home/k-hagio/bin/crash] error trace: 588df3 => 5cbc72 => 5eb3e1 => 5eb366 PID: 1 TASK: ffff9f94c024b980 CPU: 2 COMMAND: "systemd" #0 [ffffade44001bca8] __schedule at ffffffffb948ebbb #1 [ffffade44001bd10] schedule at ffffffffb948f04d #2 [ffffade44001bd20] schedule_hrtimeout_range_clock at ffffffffb9494fef #3 [ffffade44001bda8] ep_poll at ffffffffb8c91be8 #4 [ffffade44001be48] do_epoll_wait at ffffffffb8c91d11 #5 [ffffade44001be80] __x64_sys_epoll_wait at ffffffffb8c92590 #6 [ffffade44001bed0] do_syscall_64 at ffffffffb947f459 #7 [ffffade44001bf50] entry_SYSCALL_64_after_hwframe at ffffffffb96000ea 5eb366: SIZE_verify.part.42+70 5eb3e1: SIZE_verify+49 5cbc72: x86_64_low_budget_back_trace_cmd+3010 588df3: back_trace+1523 bt: invalid structure size: fred_frame FILE: x86_64.c LINE: 4089 FUNCTION: x86_64_low_budget_back_trace_cmd() Let's replace the SIZE(fred_frame) with the VALID_SIZE(fred_frame) to fix it. Fixes: 48764a1 ("x86_64: fix for adding top_of_kernel_stack_padding for kernel stack") Reported-by: Kazuhito Hagio <k-hagio-ab@nec.com> Signed-off-by: Lianbo Jiang <lijiang@redhat.com>
If we use crash to parse ramdump(Qcom phone device) rathen than vmcore. Start command should be like: crash vmlinux --kaslr=xxx DDRCS0_0.BIN@0x0000000080000000,... --machdep vabits_actual=39 Then We will see bt command show misleading backtrace information below: crash> bt 16930 PID: 16930 TASK: ffffff89b3eada00 CPU: 2 COMMAND: "Firebase Backgr" #0 [ffffffc034c437f0] __switch_to at ffffffe0036832d4 crash-utility#1 [ffffffc034c43850] __kvm_nvhe_$d.2314 at 6be732e004cf05a0 crash-utility#2 [ffffffc034c438b0] __kvm_nvhe_$d.2314 at 86c54c6004ceff80 crash-utility#3 [ffffffc034c43950] __kvm_nvhe_$d.2314 at 55d6f96003a7b120 crash-utility#4 [ffffffc034c439f0] __kvm_nvhe_$d.2314 at 9ccec46003a80a64 crash-utility#5 [ffffffc034c43ac0] __kvm_nvhe_$d.2314 at 8cf41e6003a945c4 crash-utility#6 [ffffffc034c43b10] __kvm_nvhe_$d.2314 at a8f181e00372c818 crash-utility#7 [ffffffc034c43b40] __kvm_nvhe_$d.2314 at 6dedde600372c0d0 crash-utility#8 [ffffffc034c43b90] __kvm_nvhe_$d.2314 at 62cc07e00373d0ac crash-utility#9 [ffffffc034c43c00] __kvm_nvhe_$d.2314 at 72fb1de00373bedc ... PC: 00000073f5294840 LR: 00000070d8f39ba4 SP: 00000070d4afd5d0 X29: 00000070d4afd600 X28: b4000071efcda7f0 X27: 00000070d4afe000 X26: 0000000000000000 X25: 00000070d9616000 X24: 0000000000000000 X23: 0000000000000000 X22: 0000000000000000 X21: 0000000000000000 X20: b40000728fd27520 X19: b40000728fd27550 X18: 000000702daba000 X17: 00000073f5294820 X16: 00000070d940f9d8 X15: 00000000000000bf X14: 0000000000000000 X13: 00000070d8ad2fac X12: b40000718fce5040 X11: 0000000000000000 X10: 0000000000000070 X9: 0000000000000001 X8: 0000000000000062 X7: 0000000000000020 X6: 0000000000000000 X5: 0000000000000000 X4: 0000000000000000 X3: 0000000000000000 X2: 0000000000000002 X1: 0000000000000080 X0: b40000728fd27550 ORIG_X0: b40000728fd27550 SYSCALLNO: ffffffff PSTATE: 40001000 By checking the raw data below, will see the lr (fp+8) data show the pointer which already been replaced by PAC prefix. crash> bt -f PID: 16930 TASK: ffffff89b3eada00 CPU: 2 COMMAND: "Firebase Backgr" #0 [ffffffc034c437f0] __switch_to at ffffffe0036832d4 ffffffc034c437f0: ffffffc034c43850 6be732e004cf05a4 ffffffc034c43800: ffffffe006186108 a0ed07e004cf09c4 ffffffc034c43810: ffffff8a1a340000 ffffff8a8d343c00 ffffffc034c43820: ffffff89b3eada00 ffffff8b780db540 ffffffc034c43830: ffffff89b3eada00 0000000000000000 ffffffc034c43840: 0000000000000004 712b828118484a00 crash-utility#1 [ffffffc034c43850] __kvm_nvhe_$d.2314 at 6be732e004cf05a0 ffffffc034c43850: ffffffc034c438b0 86c54c6004ceff84 ffffffc034c43860: 000000708070f000 ffffffc034c43938 ffffffc034c43870: ffffff88bd822878 ffffff89b3eada00 ... So we check the CONFIG_ARM64_PTR_AUTH and CONFIG_ARM64_PTR_AUTH_KERNEL to double check if pac mechanism been enabled on this ramdump. Then we use vabits to figure it out. Fix then show the right backtrace below: crash> bt 16930 PID: 16930 TASK: ffffff89b3eada00 CPU: 2 COMMAND: "Firebase Backgr" #0 [ffffffc034c437f0] __switch_to at ffffffe0036832d4 crash-utility#1 [ffffffc034c43850] __schedule at ffffffe004cf05a0 crash-utility#2 [ffffffc034c438b0] preempt_schedule_common at ffffffe004ceff80 crash-utility#3 [ffffffc034c43950] unmap_page_range at ffffffe003a7b120 crash-utility#4 [ffffffc034c439f0] unmap_vmas at ffffffe003a80a64 crash-utility#5 [ffffffc034c43ac0] exit_mmap at ffffffe003a945c4 crash-utility#6 [ffffffc034c43b10] __mmput at ffffffe00372c818 crash-utility#7 [ffffffc034c43b40] mmput at ffffffe00372c0d0 crash-utility#8 [ffffffc034c43b90] exit_mm at ffffffe00373d0ac crash-utility#9 [ffffffc034c43c00] do_exit at ffffffe00373bedc PC: 00000073f5294840 LR: 00000070d8f39ba4 SP: 00000070d4afd5d0 X29: 00000070d4afd600 X28: b4000071efcda7f0 X27: 00000070d4afe000 X26: 0000000000000000 X25: 00000070d9616000 X24: 0000000000000000 X23: 0000000000000000 X22: 0000000000000000 X21: 0000000000000000 X20: b40000728fd27520 X19: b40000728fd27550 X18: 000000702daba000 X17: 00000073f5294820 X16: 00000070d940f9d8 X15: 00000000000000bf X14: 0000000000000000 X13: 00000070d8ad2fac X12: b40000718fce5040 X11: 0000000000000000 X10: 0000000000000070 X9: 0000000000000001 X8: 0000000000000062 X7: 0000000000000020 X6: 0000000000000000 X5: 0000000000000000 X4: 0000000000000000 X3: 0000000000000000 X2: 0000000000000002 X1: 0000000000000080 X0: b40000728fd27550 ORIG_X0: b40000728fd27550 SYSCALLNO: ffffffff PSTATE: 40001000 Let's use GENMASK to replace the pac pointer to fix it. gki related commit url here: https://lore.kernel.org/all/20230412160134.306148-4-mark.rutland@arm.com/
If we use crash to parse ramdump(Qcom phone device) rathen than vmcore. Start command should be like: crash vmlinux --kaslr=xxx DDRCS0_0.BIN@0x0000000080000000,... --machdep vabits_actual=39 Then We will see bt command show misleading backtrace information below: crash> bt 16930 PID: 16930 TASK: ffffff89b3eada00 CPU: 2 COMMAND: "Firebase Backgr" #0 [ffffffc034c437f0] __switch_to at ffffffe0036832d4 crash-utility#1 [ffffffc034c43850] __kvm_nvhe_$d.2314 at 6be732e004cf05a0 crash-utility#2 [ffffffc034c438b0] __kvm_nvhe_$d.2314 at 86c54c6004ceff80 crash-utility#3 [ffffffc034c43950] __kvm_nvhe_$d.2314 at 55d6f96003a7b120 crash-utility#4 [ffffffc034c439f0] __kvm_nvhe_$d.2314 at 9ccec46003a80a64 crash-utility#5 [ffffffc034c43ac0] __kvm_nvhe_$d.2314 at 8cf41e6003a945c4 crash-utility#6 [ffffffc034c43b10] __kvm_nvhe_$d.2314 at a8f181e00372c818 crash-utility#7 [ffffffc034c43b40] __kvm_nvhe_$d.2314 at 6dedde600372c0d0 crash-utility#8 [ffffffc034c43b90] __kvm_nvhe_$d.2314 at 62cc07e00373d0ac crash-utility#9 [ffffffc034c43c00] __kvm_nvhe_$d.2314 at 72fb1de00373bedc ... PC: 00000073f5294840 LR: 00000070d8f39ba4 SP: 00000070d4afd5d0 X29: 00000070d4afd600 X28: b4000071efcda7f0 X27: 00000070d4afe000 X26: 0000000000000000 X25: 00000070d9616000 X24: 0000000000000000 X23: 0000000000000000 X22: 0000000000000000 X21: 0000000000000000 X20: b40000728fd27520 X19: b40000728fd27550 X18: 000000702daba000 X17: 00000073f5294820 X16: 00000070d940f9d8 X15: 00000000000000bf X14: 0000000000000000 X13: 00000070d8ad2fac X12: b40000718fce5040 X11: 0000000000000000 X10: 0000000000000070 X9: 0000000000000001 X8: 0000000000000062 X7: 0000000000000020 X6: 0000000000000000 X5: 0000000000000000 X4: 0000000000000000 X3: 0000000000000000 X2: 0000000000000002 X1: 0000000000000080 X0: b40000728fd27550 ORIG_X0: b40000728fd27550 SYSCALLNO: ffffffff PSTATE: 40001000 By checking the raw data below, will see the lr (fp+8) data show the pointer which already been replaced by PAC prefix. crash> bt -f PID: 16930 TASK: ffffff89b3eada00 CPU: 2 COMMAND: "Firebase Backgr" #0 [ffffffc034c437f0] __switch_to at ffffffe0036832d4 ffffffc034c437f0: ffffffc034c43850 6be732e004cf05a4 ffffffc034c43800: ffffffe006186108 a0ed07e004cf09c4 ffffffc034c43810: ffffff8a1a340000 ffffff8a8d343c00 ffffffc034c43820: ffffff89b3eada00 ffffff8b780db540 ffffffc034c43830: ffffff89b3eada00 0000000000000000 ffffffc034c43840: 0000000000000004 712b828118484a00 crash-utility#1 [ffffffc034c43850] __kvm_nvhe_$d.2314 at 6be732e004cf05a0 ffffffc034c43850: ffffffc034c438b0 86c54c6004ceff84 ffffffc034c43860: 000000708070f000 ffffffc034c43938 ffffffc034c43870: ffffff88bd822878 ffffff89b3eada00 ... So we check the CONFIG_ARM64_PTR_AUTH and CONFIG_ARM64_PTR_AUTH_KERNEL to double check if pac mechanism been enabled on this ramdump. Then we use vabits to figure it out. Fix then show the right backtrace below: crash> bt 16930 PID: 16930 TASK: ffffff89b3eada00 CPU: 2 COMMAND: "Firebase Backgr" #0 [ffffffc034c437f0] __switch_to at ffffffe0036832d4 crash-utility#1 [ffffffc034c43850] __schedule at ffffffe004cf05a0 crash-utility#2 [ffffffc034c438b0] preempt_schedule_common at ffffffe004ceff80 crash-utility#3 [ffffffc034c43950] unmap_page_range at ffffffe003a7b120 crash-utility#4 [ffffffc034c439f0] unmap_vmas at ffffffe003a80a64 crash-utility#5 [ffffffc034c43ac0] exit_mmap at ffffffe003a945c4 crash-utility#6 [ffffffc034c43b10] __mmput at ffffffe00372c818 crash-utility#7 [ffffffc034c43b40] mmput at ffffffe00372c0d0 crash-utility#8 [ffffffc034c43b90] exit_mm at ffffffe00373d0ac crash-utility#9 [ffffffc034c43c00] do_exit at ffffffe00373bedc PC: 00000073f5294840 LR: 00000070d8f39ba4 SP: 00000070d4afd5d0 X29: 00000070d4afd600 X28: b4000071efcda7f0 X27: 00000070d4afe000 X26: 0000000000000000 X25: 00000070d9616000 X24: 0000000000000000 X23: 0000000000000000 X22: 0000000000000000 X21: 0000000000000000 X20: b40000728fd27520 X19: b40000728fd27550 X18: 000000702daba000 X17: 00000073f5294820 X16: 00000070d940f9d8 X15: 00000000000000bf X14: 0000000000000000 X13: 00000070d8ad2fac X12: b40000718fce5040 X11: 0000000000000000 X10: 0000000000000070 X9: 0000000000000001 X8: 0000000000000062 X7: 0000000000000020 X6: 0000000000000000 X5: 0000000000000000 X4: 0000000000000000 X3: 0000000000000000 X2: 0000000000000002 X1: 0000000000000080 X0: b40000728fd27550 ORIG_X0: b40000728fd27550 SYSCALLNO: ffffffff PSTATE: 40001000 Let's use GENMASK to replace the pac pointer to fix it. gki related commit url here: https://lore.kernel.org/all/20230412160134.306148-4-mark.rutland@arm.com/ Signed-off-by: bevis_chen <bevis_chen@asus.com>
If we use crash to parse ramdump(Qcom phone device) rathen than vmcore. Start command should be like: crash vmlinux --kaslr=xxx DDRCS0_0.BIN@0x0000000080000000,... --machdep vabits_actual=39 Then We will see bt command show misleading backtrace information below: crash> bt 16930 PID: 16930 TASK: ffffff89b3eada00 CPU: 2 COMMAND: "Firebase Backgr" #0 [ffffffc034c437f0] __switch_to at ffffffe0036832d4 crash-utility#1 [ffffffc034c43850] __kvm_nvhe_$d.2314 at 6be732e004cf05a0 crash-utility#2 [ffffffc034c438b0] __kvm_nvhe_$d.2314 at 86c54c6004ceff80 crash-utility#3 [ffffffc034c43950] __kvm_nvhe_$d.2314 at 55d6f96003a7b120 crash-utility#4 [ffffffc034c439f0] __kvm_nvhe_$d.2314 at 9ccec46003a80a64 crash-utility#5 [ffffffc034c43ac0] __kvm_nvhe_$d.2314 at 8cf41e6003a945c4 crash-utility#6 [ffffffc034c43b10] __kvm_nvhe_$d.2314 at a8f181e00372c818 crash-utility#7 [ffffffc034c43b40] __kvm_nvhe_$d.2314 at 6dedde600372c0d0 crash-utility#8 [ffffffc034c43b90] __kvm_nvhe_$d.2314 at 62cc07e00373d0ac crash-utility#9 [ffffffc034c43c00] __kvm_nvhe_$d.2314 at 72fb1de00373bedc ... PC: 00000073f5294840 LR: 00000070d8f39ba4 SP: 00000070d4afd5d0 X29: 00000070d4afd600 X28: b4000071efcda7f0 X27: 00000070d4afe000 X26: 0000000000000000 X25: 00000070d9616000 X24: 0000000000000000 X23: 0000000000000000 X22: 0000000000000000 X21: 0000000000000000 X20: b40000728fd27520 X19: b40000728fd27550 X18: 000000702daba000 X17: 00000073f5294820 X16: 00000070d940f9d8 X15: 00000000000000bf X14: 0000000000000000 X13: 00000070d8ad2fac X12: b40000718fce5040 X11: 0000000000000000 X10: 0000000000000070 X9: 0000000000000001 X8: 0000000000000062 X7: 0000000000000020 X6: 0000000000000000 X5: 0000000000000000 X4: 0000000000000000 X3: 0000000000000000 X2: 0000000000000002 X1: 0000000000000080 X0: b40000728fd27550 ORIG_X0: b40000728fd27550 SYSCALLNO: ffffffff PSTATE: 40001000 By checking the raw data below, will see the lr (fp+8) data show the pointer which already been replaced by PAC prefix. crash> bt -f PID: 16930 TASK: ffffff89b3eada00 CPU: 2 COMMAND: "Firebase Backgr" #0 [ffffffc034c437f0] __switch_to at ffffffe0036832d4 ffffffc034c437f0: ffffffc034c43850 6be732e004cf05a4 ffffffc034c43800: ffffffe006186108 a0ed07e004cf09c4 ffffffc034c43810: ffffff8a1a340000 ffffff8a8d343c00 ffffffc034c43820: ffffff89b3eada00 ffffff8b780db540 ffffffc034c43830: ffffff89b3eada00 0000000000000000 ffffffc034c43840: 0000000000000004 712b828118484a00 crash-utility#1 [ffffffc034c43850] __kvm_nvhe_$d.2314 at 6be732e004cf05a0 ffffffc034c43850: ffffffc034c438b0 86c54c6004ceff84 ffffffc034c43860: 000000708070f000 ffffffc034c43938 ffffffc034c43870: ffffff88bd822878 ffffff89b3eada00 ... So we check the CONFIG_ARM64_PTR_AUTH and CONFIG_ARM64_PTR_AUTH_KERNEL to double check if pac mechanism been enabled on this ramdump. Then we use vabits to figure it out. Fix then show the right backtrace below: crash> bt 16930 PID: 16930 TASK: ffffff89b3eada00 CPU: 2 COMMAND: "Firebase Backgr" #0 [ffffffc034c437f0] __switch_to at ffffffe0036832d4 crash-utility#1 [ffffffc034c43850] __schedule at ffffffe004cf05a0 crash-utility#2 [ffffffc034c438b0] preempt_schedule_common at ffffffe004ceff80 crash-utility#3 [ffffffc034c43950] unmap_page_range at ffffffe003a7b120 crash-utility#4 [ffffffc034c439f0] unmap_vmas at ffffffe003a80a64 crash-utility#5 [ffffffc034c43ac0] exit_mmap at ffffffe003a945c4 crash-utility#6 [ffffffc034c43b10] __mmput at ffffffe00372c818 crash-utility#7 [ffffffc034c43b40] mmput at ffffffe00372c0d0 crash-utility#8 [ffffffc034c43b90] exit_mm at ffffffe00373d0ac crash-utility#9 [ffffffc034c43c00] do_exit at ffffffe00373bedc PC: 00000073f5294840 LR: 00000070d8f39ba4 SP: 00000070d4afd5d0 X29: 00000070d4afd600 X28: b4000071efcda7f0 X27: 00000070d4afe000 X26: 0000000000000000 X25: 00000070d9616000 X24: 0000000000000000 X23: 0000000000000000 X22: 0000000000000000 X21: 0000000000000000 X20: b40000728fd27520 X19: b40000728fd27550 X18: 000000702daba000 X17: 00000073f5294820 X16: 00000070d940f9d8 X15: 00000000000000bf X14: 0000000000000000 X13: 00000070d8ad2fac X12: b40000718fce5040 X11: 0000000000000000 X10: 0000000000000070 X9: 0000000000000001 X8: 0000000000000062 X7: 0000000000000020 X6: 0000000000000000 X5: 0000000000000000 X4: 0000000000000000 X3: 0000000000000000 X2: 0000000000000002 X1: 0000000000000080 X0: b40000728fd27550 ORIG_X0: b40000728fd27550 SYSCALLNO: ffffffff PSTATE: 40001000 Let's use GENMASK to replace the pac pointer to fix it. gki related commit url here: https://lore.kernel.org/all/20230412160134.306148-4-mark.rutland@arm.com/ Signed-off-by: bevis_chen <bevis_chen@asus.com>
Hi Dave -
Our Hack Week at SUSE is next week, and I'm planning on continuing a project I've been working on to extend the Python functionality exported by GDB to be useful through crash. I'm aware of several other projects with similar goals. I've already gotten the needed bits integrated into our gdb, which is at version 7.8. When I updated the patches from an earlier version to 7.8, there were substantial differences. I didn't really want to maintain two sets of patches: one against gdb and the other against the gdb that crash uses. Since there's no real reason not to update crash's gdb version to 7.8, I went ahead and integrated it with crash.
This pull request covers a fair amount of ground.
I've done some light testing with the command loop, but if you have a test system, I'd be happy to drive it a bit more.