Starting kernel ... hang #21
Comments
|
Hi, 请使用Rockchip提供的echo SDK for RK3229 |
|
我们之前在debian上面做了大量应用，目前我们急需在rk3229运行debian，快速验证软件和测试性能。而且我们团队对echo SDK的buildroot开发环境不熟悉，学习成本太高。 |
|
请通过业务正式提出需求 |
|
kernel使用的是什么配置？ |
|
@keveryang 启动的时候还是卡在这里： 下面是我用的https://github.com/rockchip-linux/build 脚本的配置信息，kernel用的是rockchip_linux_defconfig. |
|
kernel下面这几个配置尝试更新一下，原来是STRONG |
|
@keveryang 但是还是卡在Starting kernel ... |
|
ttyfiq |
|
是调试串口不对吗？？具体的是啥，这个串口是uart2 |
|
console=ttyFIQ0,1500000n8 |
|
我现在换回了ttyFIQ0: 还是一样，卡在Starting kernel .. |
|
试了下 可以起到一半 uart没输出主要是那个rk322x-android.dtsi里同时enable了FIQ和uart2 你花下功夫配dts |
|
但这个是没正式支持的, 如果你只是验证cpu性能, 你还得确保下cpufreq和thermal在正常工作 |
|
@wzyy2 |
|
3229没有正式的linux支持。需要正式的linux支持的话请联系业务部提出正式需求 |
rkchrome
pushed a commit
that referenced
this issue
Jul 19, 2018
[ Upstream commit 2bbea6e ] when mounting an ISO filesystem sometimes (very rarely) the system hangs because of a race condition between two tasks. PID: 6766 TASK: ffff88007b2a6dd0 CPU: 0 COMMAND: "mount" #0 [ffff880078447ae0] __schedule at ffffffff8168d605 #1 [ffff880078447b48] schedule_preempt_disabled at ffffffff8168ed49 #2 [ffff880078447b58] __mutex_lock_slowpath at ffffffff8168c995 #3 [ffff880078447bb8] mutex_lock at ffffffff8168bdef #4 [ffff880078447bd0] sr_block_ioctl at ffffffffa00b6818 [sr_mod] #5 [ffff880078447c10] blkdev_ioctl at ffffffff812fea50 #6 [ffff880078447c70] ioctl_by_bdev at ffffffff8123a8b3 #7 [ffff880078447c90] isofs_fill_super at ffffffffa04fb1e1 [isofs] #8 [ffff880078447da8] mount_bdev at ffffffff81202570 #9 [ffff880078447e18] isofs_mount at ffffffffa04f9828 [isofs] #10 [ffff880078447e28] mount_fs at ffffffff81202d09 #11 [ffff880078447e70] vfs_kern_mount at ffffffff8121ea8f #12 [ffff880078447ea8] do_mount at ffffffff81220fee #13 [ffff880078447f28] sys_mount at ffffffff812218d6 #14 [ffff880078447f80] system_call_fastpath at ffffffff81698c49 RIP: 00007fd9ea914e9a RSP: 00007ffd5d9bf648 RFLAGS: 00010246 RAX: 00000000000000a5 RBX: ffffffff81698c49 RCX: 0000000000000010 RDX: 00007fd9ec2bc210 RSI: 00007fd9ec2bc290 RDI: 00007fd9ec2bcf30 RBP: 0000000000000000 R8: 0000000000000000 R9: 0000000000000010 R10: 00000000c0ed0001 R11: 0000000000000206 R12: 00007fd9ec2bc040 R13: 00007fd9eb6b2380 R14: 00007fd9ec2bc210 R15: 00007fd9ec2bcf30 ORIG_RAX: 00000000000000a5 CS: 0033 SS: 002b This task was trying to mount the cdrom. It allocated and configured a super_block struct and owned the write-lock for the super_block->s_umount rwsem. While exclusively owning the s_umount lock, it called sr_block_ioctl and waited to acquire the global sr_mutex lock. PID: 6785 TASK: ffff880078720fb0 CPU: 0 COMMAND: "systemd-udevd" #0 [ffff880078417898] __schedule at ffffffff8168d605 #1 [ffff880078417900] schedule at ffffffff8168dc59 #2 [ffff880078417910] rwsem_down_read_failed at ffffffff8168f605 #3 [ffff880078417980] call_rwsem_down_read_failed at ffffffff81328838 #4 [ffff8800784179d0] down_read at ffffffff8168cde0 #5 [ffff8800784179e8] get_super at ffffffff81201cc7 #6 [ffff880078417a10] __invalidate_device at ffffffff8123a8de #7 [ffff880078417a40] flush_disk at ffffffff8123a94b #8 [ffff880078417a88] check_disk_change at ffffffff8123ab50 #9 [ffff880078417ab0] cdrom_open at ffffffffa00a29e1 [cdrom] #10 [ffff880078417b68] sr_block_open at ffffffffa00b6f9b [sr_mod] #11 [ffff880078417b98] __blkdev_get at ffffffff8123ba86 #12 [ffff880078417bf0] blkdev_get at ffffffff8123bd65 #13 [ffff880078417c78] blkdev_open at ffffffff8123bf9b #14 [ffff880078417c90] do_dentry_open at ffffffff811fc7f7 #15 [ffff880078417cd8] vfs_open at ffffffff811fc9cf #16 [ffff880078417d00] do_last at ffffffff8120d53d #17 [ffff880078417db0] path_openat at ffffffff8120e6b2 #18 [ffff880078417e48] do_filp_open at ffffffff8121082b #19 [ffff880078417f18] do_sys_open at ffffffff811fdd33 #20 [ffff880078417f70] sys_open at ffffffff811fde4e #21 [ffff880078417f80] system_call_fastpath at ffffffff81698c49 RIP: 00007f29438b0c20 RSP: 00007ffc76624b78 RFLAGS: 00010246 RAX: 0000000000000002 RBX: ffffffff81698c49 RCX: 0000000000000000 RDX: 00007f2944a5fa70 RSI: 00000000000a0800 RDI: 00007f2944a5fa70 RBP: 00007f2944a5f540 R8: 0000000000000000 R9: 0000000000000020 R10: 00007f2943614c40 R11: 0000000000000246 R12: ffffffff811fde4e R13: ffff880078417f78 R14: 000000000000000c R15: 00007f2944a4b010 ORIG_RAX: 0000000000000002 CS: 0033 SS: 002b This task tried to open the cdrom device, the sr_block_open function acquired the global sr_mutex lock. The call to check_disk_change() then saw an event flag indicating a possible media change and tried to flush any cached data for the device. As part of the flush, it tried to acquire the super_block->s_umount lock associated with the cdrom device. This was the same super_block as created and locked by the previous task. The first task acquires the s_umount lock and then the sr_mutex_lock; the second task acquires the sr_mutex_lock and then the s_umount lock. This patch fixes the issue by moving check_disk_change() out of cdrom_open() and let the caller take care of it. Signed-off-by: Maurizio Lombardi <mlombard@redhat.com> Signed-off-by: Jens Axboe <axboe@kernel.dk> Signed-off-by: Sasha Levin <alexander.levin@microsoft.com> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
rkchrome
pushed a commit
that referenced
this issue
Sep 26, 2018
[ Upstream commit 2bbea6e ] when mounting an ISO filesystem sometimes (very rarely) the system hangs because of a race condition between two tasks. PID: 6766 TASK: ffff88007b2a6dd0 CPU: 0 COMMAND: "mount" #0 [ffff880078447ae0] __schedule at ffffffff8168d605 #1 [ffff880078447b48] schedule_preempt_disabled at ffffffff8168ed49 #2 [ffff880078447b58] __mutex_lock_slowpath at ffffffff8168c995 #3 [ffff880078447bb8] mutex_lock at ffffffff8168bdef #4 [ffff880078447bd0] sr_block_ioctl at ffffffffa00b6818 [sr_mod] #5 [ffff880078447c10] blkdev_ioctl at ffffffff812fea50 #6 [ffff880078447c70] ioctl_by_bdev at ffffffff8123a8b3 #7 [ffff880078447c90] isofs_fill_super at ffffffffa04fb1e1 [isofs] #8 [ffff880078447da8] mount_bdev at ffffffff81202570 #9 [ffff880078447e18] isofs_mount at ffffffffa04f9828 [isofs] #10 [ffff880078447e28] mount_fs at ffffffff81202d09 #11 [ffff880078447e70] vfs_kern_mount at ffffffff8121ea8f #12 [ffff880078447ea8] do_mount at ffffffff81220fee #13 [ffff880078447f28] sys_mount at ffffffff812218d6 #14 [ffff880078447f80] system_call_fastpath at ffffffff81698c49 RIP: 00007fd9ea914e9a RSP: 00007ffd5d9bf648 RFLAGS: 00010246 RAX: 00000000000000a5 RBX: ffffffff81698c49 RCX: 0000000000000010 RDX: 00007fd9ec2bc210 RSI: 00007fd9ec2bc290 RDI: 00007fd9ec2bcf30 RBP: 0000000000000000 R8: 0000000000000000 R9: 0000000000000010 R10: 00000000c0ed0001 R11: 0000000000000206 R12: 00007fd9ec2bc040 R13: 00007fd9eb6b2380 R14: 00007fd9ec2bc210 R15: 00007fd9ec2bcf30 ORIG_RAX: 00000000000000a5 CS: 0033 SS: 002b This task was trying to mount the cdrom. It allocated and configured a super_block struct and owned the write-lock for the super_block->s_umount rwsem. While exclusively owning the s_umount lock, it called sr_block_ioctl and waited to acquire the global sr_mutex lock. PID: 6785 TASK: ffff880078720fb0 CPU: 0 COMMAND: "systemd-udevd" #0 [ffff880078417898] __schedule at ffffffff8168d605 #1 [ffff880078417900] schedule at ffffffff8168dc59 #2 [ffff880078417910] rwsem_down_read_failed at ffffffff8168f605 #3 [ffff880078417980] call_rwsem_down_read_failed at ffffffff81328838 #4 [ffff8800784179d0] down_read at ffffffff8168cde0 #5 [ffff8800784179e8] get_super at ffffffff81201cc7 #6 [ffff880078417a10] __invalidate_device at ffffffff8123a8de #7 [ffff880078417a40] flush_disk at ffffffff8123a94b #8 [ffff880078417a88] check_disk_change at ffffffff8123ab50 #9 [ffff880078417ab0] cdrom_open at ffffffffa00a29e1 [cdrom] #10 [ffff880078417b68] sr_block_open at ffffffffa00b6f9b [sr_mod] #11 [ffff880078417b98] __blkdev_get at ffffffff8123ba86 #12 [ffff880078417bf0] blkdev_get at ffffffff8123bd65 #13 [ffff880078417c78] blkdev_open at ffffffff8123bf9b #14 [ffff880078417c90] do_dentry_open at ffffffff811fc7f7 #15 [ffff880078417cd8] vfs_open at ffffffff811fc9cf #16 [ffff880078417d00] do_last at ffffffff8120d53d #17 [ffff880078417db0] path_openat at ffffffff8120e6b2 #18 [ffff880078417e48] do_filp_open at ffffffff8121082b #19 [ffff880078417f18] do_sys_open at ffffffff811fdd33 #20 [ffff880078417f70] sys_open at ffffffff811fde4e #21 [ffff880078417f80] system_call_fastpath at ffffffff81698c49 RIP: 00007f29438b0c20 RSP: 00007ffc76624b78 RFLAGS: 00010246 RAX: 0000000000000002 RBX: ffffffff81698c49 RCX: 0000000000000000 RDX: 00007f2944a5fa70 RSI: 00000000000a0800 RDI: 00007f2944a5fa70 RBP: 00007f2944a5f540 R8: 0000000000000000 R9: 0000000000000020 R10: 00007f2943614c40 R11: 0000000000000246 R12: ffffffff811fde4e R13: ffff880078417f78 R14: 000000000000000c R15: 00007f2944a4b010 ORIG_RAX: 0000000000000002 CS: 0033 SS: 002b This task tried to open the cdrom device, the sr_block_open function acquired the global sr_mutex lock. The call to check_disk_change() then saw an event flag indicating a possible media change and tried to flush any cached data for the device. As part of the flush, it tried to acquire the super_block->s_umount lock associated with the cdrom device. This was the same super_block as created and locked by the previous task. The first task acquires the s_umount lock and then the sr_mutex_lock; the second task acquires the sr_mutex_lock and then the s_umount lock. This patch fixes the issue by moving check_disk_change() out of cdrom_open() and let the caller take care of it. Signed-off-by: Maurizio Lombardi <mlombard@redhat.com> Signed-off-by: Jens Axboe <axboe@kernel.dk> Signed-off-by: Sasha Levin <alexander.levin@microsoft.com> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
0lvin
pushed a commit
to free-z4u/roc-rk3328-cc-official
that referenced
this issue
Apr 14, 2019
tests/generic/251 of fstest suit complains us with below message: ------------[ cut here ]------------ invalid opcode: 0000 [FireflyTeam#1] PREEMPT SMP CPU: 2 PID: 7698 Comm: fstrim Tainted: G O 4.7.0+ rockchip-linux#21 task: e9f4e000 task.stack: e7262000 EIP: 0060:[<f89fcefe>] EFLAGS: 00010202 CPU: 2 EIP is at write_checkpoint+0xfde/0x1020 [f2fs] EAX: f33eb300 EBX: eecac310 ECX: 00000001 EDX: ffff0001 ESI: eecac000 EDI: eecac5f0 EBP: e7263dec ESP: e7263d18 DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068 CR0: 80050033 CR2: b76ab01c CR3: 2eb89de0 CR4: 000406f0 Stack: 00000001 a220fb7b e9f4e000 00000002 419ff2d3 b3a05151 00000002 e9f4e5d8 e9f4e000 419ff2d3 b3a05151 eecac310 c10b8154 b3a05151 419ff2d3 c10b78bd e9f4e000 e9f4e000 e9f4e5d8 00000001 e9f4e000 ec409000 eecac2cc eecac288 Call Trace: [<c10b8154>] ? __lock_acquire+0x3c4/0x760 [<c10b78bd>] ? mark_held_locks+0x5d/0x80 [<f8a10632>] f2fs_trim_fs+0x1c2/0x2e0 [f2fs] [<f89e9f56>] f2fs_ioctl+0x6b6/0x10b0 [f2fs] [<c13d51df>] ? __this_cpu_preempt_check+0xf/0x20 [<c10b4281>] ? trace_hardirqs_off_caller+0x91/0x120 [<f89e98a0>] ? __exchange_data_block+0xd30/0xd30 [f2fs] [<c120b2e1>] do_vfs_ioctl+0x81/0x7f0 [<c11d57c5>] ? kmem_cache_free+0x245/0x2e0 [<c1217840>] ? get_unused_fd_flags+0x40/0x40 [<c1206eec>] ? putname+0x4c/0x50 [<c11f631e>] ? do_sys_open+0x16e/0x1d0 [<c1001990>] ? do_fast_syscall_32+0x30/0x1c0 [<c13d51df>] ? __this_cpu_preempt_check+0xf/0x20 [<c120baa8>] SyS_ioctl+0x58/0x80 [<c1001a01>] do_fast_syscall_32+0xa1/0x1c0 [<c178cc54>] sysenter_past_esp+0x45/0x74 EIP: [<f89fcefe>] write_checkpoint+0xfde/0x1020 [f2fs] SS:ESP 0068:e7263d18 ---[ end trace 4de95d7e6b3aa7c6 ]--- The reason is: with below call stack, we will encounter BUG_ON during doing fstrim. Thread A Thread B - write_checkpoint - do_checkpoint - f2fs_write_inode - update_inode_page - update_inode - set_page_dirty - f2fs_set_node_page_dirty - inc_page_count - percpu_counter_inc - set_sbi_flag(SBI_IS_DIRTY) - clear_sbi_flag(SBI_IS_DIRTY) Thread C Thread D - f2fs_write_node_page - set_node_addr - __set_nat_cache_dirty - nm_i->dirty_nat_cnt++ - do_vfs_ioctl - f2fs_ioctl - f2fs_trim_fs - write_checkpoint - f2fs_bug_on(nm_i->dirty_nat_cnt) Fix it by setting superblock dirty correctly in do_checkpoint and f2fs_write_node_page. Signed-off-by: Chao Yu <yuchao0@huawei.com> Signed-off-by: Jaegeuk Kim <jaegeuk@kernel.org>
0lvin
pushed a commit
to free-z4u/roc-rk3328-cc-official
that referenced
this issue
Apr 27, 2019
Function ib_create_qp() was failing to return an error when rdma_rw_init_mrs() fails, causing a crash further down in ib_create_qp() when trying to dereferece the qp pointer which was actually a negative errno. The crash: crash> log|grep BUG [ 136.458121] BUG: unable to handle kernel NULL pointer dereference at 0000000000000098 crash> bt PID: 3736 TASK: ffff8808543215c0 CPU: 2 COMMAND: "kworker/u64:2" #0 [ffff88084d323340] machine_kexec at ffffffff8105fbb0 FireflyTeam#1 [ffff88084d3233b0] __crash_kexec at ffffffff81116758 FireflyTeam#2 [ffff88084d323480] crash_kexec at ffffffff8111682d FireflyTeam#3 [ffff88084d3234b0] oops_end at ffffffff81032bd6 FireflyTeam#4 [ffff88084d3234e0] no_context at ffffffff8106e431 FireflyTeam#5 [ffff88084d323530] __bad_area_nosemaphore at ffffffff8106e610 FireflyTeam#6 [ffff88084d323590] bad_area_nosemaphore at ffffffff8106e6f4 FireflyTeam#7 [ffff88084d3235a0] __do_page_fault at ffffffff8106ebdc FireflyTeam#8 [ffff88084d323620] do_page_fault at ffffffff8106f057 FireflyTeam#9 [ffff88084d323660] page_fault at ffffffff816e3148 [exception RIP: ib_create_qp+427] RIP: ffffffffa02554fb RSP: ffff88084d323718 RFLAGS: 00010246 RAX: 0000000000000004 RBX: fffffffffffffff4 RCX: 000000018020001f RDX: ffff880830997fc0 RSI: 0000000000000001 RDI: ffff88085f407200 RBP: ffff88084d323778 R8: 0000000000000001 R9: ffffea0020bae210 R10: ffffea0020bae218 R11: 0000000000000001 R12: ffff88084d3237c8 R13: 00000000fffffff4 R14: ffff880859fa5000 R15: ffff88082eb89800 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 FireflyTeam#10 [ffff88084d323780] rdma_create_qp at ffffffffa0782681 [rdma_cm] FireflyTeam#11 [ffff88084d3237b0] nvmet_rdma_create_queue_ib at ffffffffa07c43f3 [nvmet_rdma] FireflyTeam#12 [ffff88084d323860] nvmet_rdma_alloc_queue at ffffffffa07c5ba9 [nvmet_rdma] FireflyTeam#13 [ffff88084d323900] nvmet_rdma_queue_connect at ffffffffa07c5c96 [nvmet_rdma] FireflyTeam#14 [ffff88084d323980] nvmet_rdma_cm_handler at ffffffffa07c6450 [nvmet_rdma] FireflyTeam#15 [ffff88084d3239b0] iw_conn_req_handler at ffffffffa0787480 [rdma_cm] FireflyTeam#16 [ffff88084d323a60] cm_conn_req_handler at ffffffffa0775f06 [iw_cm] rockchip-linux#17 [ffff88084d323ab0] process_event at ffffffffa0776019 [iw_cm] rockchip-linux#18 [ffff88084d323af0] cm_work_handler at ffffffffa0776170 [iw_cm] rockchip-linux#19 [ffff88084d323cb0] process_one_work at ffffffff810a1483 rockchip-linux#20 [ffff88084d323d90] worker_thread at ffffffff810a211d rockchip-linux#21 [ffff88084d323ec0] kthread at ffffffff810a6c5c rockchip-linux#22 [ffff88084d323f50] ret_from_fork at ffffffff816e1ebf Fixes: 632bc3f ("IB/core, RDMA RW API: Do not exceed QP SGE send limit") Signed-off-by: Steve Wise <swise@opengridcomputing.com> Cc: stable@vger.kernel.org Reviewed-by: Bart Van Assche <bart.vanassche@sandisk.com> Signed-off-by: Doug Ledford <dledford@redhat.com>
0lvin
pushed a commit
to free-z4u/roc-rk3328-cc-official
that referenced
this issue
May 1, 2019
Commit afc3f83 ("scsi: ipr: Add asynchronous error notification") introduced the warn on shown below. To fix this, rather than attempting to send the KOBJ_CHANGE uevent from interrupt context, which is what is causing the WARN_ON, just wake the ipr worker thread which will send a KOBJ_CHANGE uevent. [ 142.278120] WARNING: CPU: 15 PID: 0 at kernel/softirq.c:161 __local_bh_enable_ip+0x7c/0xd0 [ 142.278124] Modules linked in: ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ebtable_filter ebtables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw ip6table_filter ip6_tables iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw iptable_filter ses enclosure scsi_transport_sas sg pseries_rng nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod sd_mod cdrom ipr libata ibmvscsi scsi_transport_srp ibmveth dm_mirror dm_region_hash dm_log dm_mod [ 142.278208] CPU: 15 PID: 0 Comm: swapper/15 Not tainted 4.8.0.ipr+ rockchip-linux#21 [ 142.278213] task: c00000010cf24480 task.stack: c00000010cfec000 [ 142.278217] NIP: c0000000000c0c7c LR: c000000000881778 CTR: c0000000003c5bf0 [ 142.278221] REGS: c00000010cfef080 TRAP: 0700 Not tainted (4.8.0.ipr+) [ 142.278224] MSR: 8000000000029033 <SF,EE,ME,IR,DR,RI,LE> CR: 28008022 XER: 2000000f [ 142.278236] CFAR: c0000000000c0c20 SOFTE: 0 GPR00: c000000000706c78 c00000010cfef300 c000000000f91d00 c000000000706c78 GPR04: 0000000000000200 c000000000f7bc80 0000000000000000 00000000024000c0 GPR08: 0000000000000000 0000000000000001 c000000000ee1d00 c000000000a9bdd0 GPR12: c0000000003c5bf0 c00000000eb22d00 c000000100ca3880 c00000020ed38400 GPR16: 0000000000000000 0000000000000000 c000000100940508 0000000000000000 GPR20: 0000000000000000 0000000000000000 0000000000000000 00000000024000c0 GPR24: c0000000004588e0 c00000010863bd00 c00000010863bd00 c0000000013773f8 GPR28: c000000000f7bc80 0000000000000000 ffffffffffffffff c000000000f7bcd8 [ 142.278290] NIP [c0000000000c0c7c] __local_bh_enable_ip+0x7c/0xd0 [ 142.278296] LR [c000000000881778] _raw_spin_unlock_bh+0x38/0x60 [ 142.278299] Call Trace: [ 142.278303] [c00000010cfef300] [c000000000f7bc80] init_net+0x0/0x1900 (unreliable) [ 142.278310] [c00000010cfef320] [c000000000706c78] peernet2id+0x58/0x80 [ 142.278316] [c00000010cfef370] [c00000000075caec] netlink_broadcast_filtered+0x30c/0x550 [ 142.278323] [c00000010cfef430] [c000000000459078] kobject_uevent_env+0x588/0x780 [ 142.278331] [c00000010cfef510] [d000000003163a6c] ipr_process_error+0x11c/0x240 [ipr] [ 142.278337] [c00000010cfef5c0] [d000000003152298] ipr_fail_all_ops+0x108/0x220 [ipr] [ 142.278343] [c00000010cfef670] [d0000000031643f8] ipr_reset_restore_cfg_space+0xa8/0x240 [ipr] [ 142.278350] [c00000010cfef6f0] [d000000003158a00] ipr_reset_ioa_job+0x80/0xe0 [ipr] [ 142.278356] [c00000010cfef720] [d000000003153f78] ipr_reset_timer_done+0xa8/0xe0 [ipr] [ 142.278363] [c00000010cfef770] [c000000000149c88] call_timer_fn+0x58/0x1c0 [ 142.278368] [c00000010cfef800] [c000000000149f60] expire_timers+0x140/0x200 [ 142.278373] [c00000010cfef870] [c00000000014a0e8] run_timer_softirq+0xc8/0x230 [ 142.278379] [c00000010cfef900] [c0000000000c0844] __do_softirq+0x164/0x3c0 [ 142.278384] [c00000010cfef9f0] [c0000000000c0f18] irq_exit+0x1a8/0x1c0 [ 142.278389] [c00000010cfefa20] [c000000000020b54] timer_interrupt+0xa4/0xe0 [ 142.278394] [c00000010cfefa50] [c000000000002414] decrementer_common+0x114/0x180 Signed-off-by: Brian King <brking@linux.vnet.ibm.com> Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
0lvin
pushed a commit
to free-z4u/roc-rk3328-cc-official
that referenced
this issue
May 3, 2019
When streaming a lot of data and the RZ/A1 can't keep up, some status bits will get set that are not being checked or cleared which cause the following messages and the Ethernet driver to stop working. This patch fixes that issue. irq 21: nobody cared (try booting with the "irqpoll" option) handlers: [<c036b71c>] sh_eth_interrupt Disabling IRQ rockchip-linux#21 Fixes: db89347 ("sh_eth: Add support for r7s72100") Signed-off-by: Chris Brandt <chris.brandt@renesas.com> Acked-by: Sergei Shtylyov <sergei.shtylyov@cogentembedded.com> Signed-off-by: David S. Miller <davem@davemloft.net>
0lvin
pushed a commit
to free-z4u/roc-rk3328-cc-official
that referenced
this issue
Jun 2, 2019
There have been several reports over the years of NULL pointer
dereferences in xfs_trans_log_inode during xfs_fsr processes,
when the process is doing an fput and tearing down extents
on the temporary inode, something like:
BUG: unable to handle kernel NULL pointer dereference at 0000000000000018
PID: 29439 TASK: ffff880550584fa0 CPU: 6 COMMAND: "xfs_fsr"
[exception RIP: xfs_trans_log_inode+0x10]
FireflyTeam#9 [ffff8800a57bbbe0] xfs_bunmapi at ffffffffa037398e [xfs]
FireflyTeam#10 [ffff8800a57bbce8] xfs_itruncate_extents at ffffffffa0391b29 [xfs]
FireflyTeam#11 [ffff8800a57bbd88] xfs_inactive_truncate at ffffffffa0391d0c [xfs]
FireflyTeam#12 [ffff8800a57bbdb8] xfs_inactive at ffffffffa0392508 [xfs]
FireflyTeam#13 [ffff8800a57bbdd8] xfs_fs_evict_inode at ffffffffa035907e [xfs]
FireflyTeam#14 [ffff8800a57bbe00] evict at ffffffff811e1b67
FireflyTeam#15 [ffff8800a57bbe28] iput at ffffffff811e23a5
FireflyTeam#16 [ffff8800a57bbe58] dentry_kill at ffffffff811dcfc8
rockchip-linux#17 [ffff8800a57bbe88] dput at ffffffff811dd06c
rockchip-linux#18 [ffff8800a57bbea8] __fput at ffffffff811c823b
rockchip-linux#19 [ffff8800a57bbef0] ____fput at ffffffff811c846e
rockchip-linux#20 [ffff8800a57bbf00] task_work_run at ffffffff81093b27
rockchip-linux#21 [ffff8800a57bbf30] do_notify_resume at ffffffff81013b0c
rockchip-linux#22 [ffff8800a57bbf50] int_signal at ffffffff8161405d
As it turns out, this is because the i_itemp pointer, along
with the d_ops pointer, has been overwritten with zeros
when we tear down the extents during truncate. When the in-core
inode fork on the temporary inode used by xfs_fsr was originally
set up during the extent swap, we mistakenly looked at di_nextents
to determine whether all extents fit inline, but this misses extents
generated by speculative preallocation; we should be using if_bytes
instead.
This mistake corrupts the in-memory inode, and code in
xfs_iext_remove_inline eventually gets bad inputs, causing
it to memmove and memset incorrect ranges; this became apparent
because the two values in ifp->if_u2.if_inline_ext[1] contained
what should have been in d_ops and i_itemp; they were memmoved due
to incorrect array indexing and then the original locations
were zeroed with memset, again due to an array overrun.
Fix this by properly using i_df.if_bytes to determine the number
of extents, not di_nextents.
Thanks to dchinner for looking at this with me and spotting the
root cause.
Cc: stable@vger.kernel.org
Signed-off-by: Eric Sandeen <sandeen@redhat.com>
Reviewed-by: Brian Foster <bfoster@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>
0lvin
pushed a commit
to free-z4u/roc-rk3328-cc-official
that referenced
this issue
Jun 8, 2019
That's unclear why lockdep shows the following warning but adding a lockdep class to struct pmac_i2c_bus solves it [ 20.507795] ====================================================== [ 20.507796] [ INFO: possible circular locking dependency detected ] [ 20.507800] 4.8.0-rc7-00037-gd2ffb01 rockchip-linux#21 Not tainted [ 20.507801] ------------------------------------------------------- [ 20.507803] swapper/0/1 is trying to acquire lock: [ 20.507818] (&bus->mutex){+.+.+.}, at: [<c000000000052830>] .pmac_i2c_open+0x30/0x100 [ 20.507819] [ 20.507819] but task is already holding lock: [ 20.507829] (&policy->rwsem){+.+.+.}, at: [<c00000000068adcc>] .cpufreq_online+0x1ac/0x9d0 [ 20.507830] [ 20.507830] which lock already depends on the new lock. [ 20.507830] [ 20.507832] [ 20.507832] the existing dependency chain (in reverse order) is: [ 20.507837] [ 20.507837] -> FireflyTeam#4 (&policy->rwsem){+.+.+.}: [ 20.507844] [<c00000000082385c>] .down_write+0x6c/0x110 [ 20.507849] [<c00000000068adcc>] .cpufreq_online+0x1ac/0x9d0 [ 20.507855] [<c0000000004d76d8>] .subsys_interface_register+0xb8/0x110 [ 20.507860] [<c000000000689bb0>] .cpufreq_register_driver+0x1d0/0x250 [ 20.507866] [<c000000000b4f8f4>] .g5_cpufreq_init+0x9cc/0xa28 [ 20.507872] [<c00000000000a98c>] .do_one_initcall+0x5c/0x1d0 [ 20.507878] [<c000000000b0f86c>] .kernel_init_freeable+0x1ac/0x28c [ 20.507883] [<c00000000000b3bc>] .kernel_init+0x1c/0x140 [ 20.507887] [<c0000000000098f4>] .ret_from_kernel_thread+0x58/0x64 [ 20.507894] [ 20.507894] -> FireflyTeam#3 (subsys mutex#2){+.+.+.}: [ 20.507899] [<c000000000820448>] .mutex_lock_nested+0xa8/0x590 [ 20.507903] [<c0000000004d7f24>] .bus_probe_device+0x44/0xe0 [ 20.507907] [<c0000000004d5208>] .device_add+0x508/0x730 [ 20.507911] [<c0000000004dd528>] .register_cpu+0x118/0x190 [ 20.507916] [<c000000000b14450>] .topology_init+0x148/0x248 [ 20.507921] [<c00000000000a98c>] .do_one_initcall+0x5c/0x1d0 [ 20.507925] [<c000000000b0f86c>] .kernel_init_freeable+0x1ac/0x28c [ 20.507929] [<c00000000000b3bc>] .kernel_init+0x1c/0x140 [ 20.507934] [<c0000000000098f4>] .ret_from_kernel_thread+0x58/0x64 [ 20.507939] [ 20.507939] -> FireflyTeam#2 (cpu_add_remove_lock){+.+.+.}: [ 20.507944] [<c000000000820448>] .mutex_lock_nested+0xa8/0x590 [ 20.507950] [<c000000000087a9c>] .register_cpu_notifier+0x2c/0x70 [ 20.507955] [<c000000000b267e0>] .spawn_ksoftirqd+0x18/0x4c [ 20.507959] [<c00000000000a98c>] .do_one_initcall+0x5c/0x1d0 [ 20.507964] [<c000000000b0f770>] .kernel_init_freeable+0xb0/0x28c [ 20.507968] [<c00000000000b3bc>] .kernel_init+0x1c/0x140 [ 20.507972] [<c0000000000098f4>] .ret_from_kernel_thread+0x58/0x64 [ 20.507978] [ 20.507978] -> FireflyTeam#1 (&host->mutex){+.+.+.}: [ 20.507982] [<c000000000820448>] .mutex_lock_nested+0xa8/0x590 [ 20.507987] [<c0000000000527e8>] .kw_i2c_open+0x18/0x30 [ 20.507991] [<c000000000052894>] .pmac_i2c_open+0x94/0x100 [ 20.507995] [<c000000000b220a0>] .smp_core99_probe+0x260/0x410 [ 20.507999] [<c000000000b185bc>] .smp_prepare_cpus+0x280/0x2ac [ 20.508003] [<c000000000b0f748>] .kernel_init_freeable+0x88/0x28c [ 20.508008] [<c00000000000b3bc>] .kernel_init+0x1c/0x140 [ 20.508012] [<c0000000000098f4>] .ret_from_kernel_thread+0x58/0x64 [ 20.508018] [ 20.508018] -> #0 (&bus->mutex){+.+.+.}: [ 20.508023] [<c0000000000ed5b4>] .lock_acquire+0x84/0x100 [ 20.508027] [<c000000000820448>] .mutex_lock_nested+0xa8/0x590 [ 20.508032] [<c000000000052830>] .pmac_i2c_open+0x30/0x100 [ 20.508037] [<c000000000052e14>] .pmac_i2c_do_begin+0x34/0x120 [ 20.508040] [<c000000000056bc0>] .pmf_call_one+0x50/0xd0 [ 20.508045] [<c00000000068ff1c>] .g5_pfunc_switch_volt+0x2c/0xc0 [ 20.508050] [<c00000000068fecc>] .g5_pfunc_switch_freq+0x1cc/0x1f0 [ 20.508054] [<c00000000068fc2c>] .g5_cpufreq_target+0x2c/0x40 [ 20.508058] [<c0000000006873ec>] .__cpufreq_driver_target+0x23c/0x840 [ 20.508062] [<c00000000068c798>] .cpufreq_gov_performance_limits+0x18/0x30 [ 20.508067] [<c00000000068915c>] .cpufreq_start_governor+0xac/0x100 [ 20.508071] [<c00000000068a788>] .cpufreq_set_policy+0x208/0x260 [ 20.508076] [<c00000000068abdc>] .cpufreq_init_policy+0x6c/0xb0 [ 20.508081] [<c00000000068ae70>] .cpufreq_online+0x250/0x9d0 [ 20.508085] [<c0000000004d76d8>] .subsys_interface_register+0xb8/0x110 [ 20.508090] [<c000000000689bb0>] .cpufreq_register_driver+0x1d0/0x250 [ 20.508094] [<c000000000b4f8f4>] .g5_cpufreq_init+0x9cc/0xa28 [ 20.508099] [<c00000000000a98c>] .do_one_initcall+0x5c/0x1d0 [ 20.508103] [<c000000000b0f86c>] .kernel_init_freeable+0x1ac/0x28c [ 20.508107] [<c00000000000b3bc>] .kernel_init+0x1c/0x140 [ 20.508112] [<c0000000000098f4>] .ret_from_kernel_thread+0x58/0x64 [ 20.508113] [ 20.508113] other info that might help us debug this: [ 20.508113] [ 20.508121] Chain exists of: [ 20.508121] &bus->mutex --> subsys mutex#2 --> &policy->rwsem [ 20.508121] [ 20.508123] Possible unsafe locking scenario: [ 20.508123] [ 20.508124] CPU0 CPU1 [ 20.508125] ---- ---- [ 20.508128] lock(&policy->rwsem); [ 20.508132] lock(subsys mutex#2); [ 20.508135] lock(&policy->rwsem); [ 20.508138] lock(&bus->mutex); [ 20.508139] [ 20.508139] *** DEADLOCK *** [ 20.508139] [ 20.508141] 3 locks held by swapper/0/1: [ 20.508150] #0: (cpu_hotplug.lock){++++++}, at: [<c000000000087838>] .get_online_cpus+0x48/0xc0 [ 20.508159] FireflyTeam#1: (subsys mutex#2){+.+.+.}, at: [<c0000000004d7670>] .subsys_interface_register+0x50/0x110 [ 20.508168] FireflyTeam#2: (&policy->rwsem){+.+.+.}, at: [<c00000000068adcc>] .cpufreq_online+0x1ac/0x9d0 [ 20.508169] [ 20.508169] stack backtrace: [ 20.508173] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.8.0-rc7-00037-gd2ffb01 rockchip-linux#21 [ 20.508175] Call Trace: [ 20.508180] [c0000000790c2b90] [c00000000082cc70] .dump_stack+0xe0/0x14c (unreliable) [ 20.508184] [c0000000790c2c20] [c000000000828c88] .print_circular_bug+0x350/0x388 [ 20.508188] [c0000000790c2cd0] [c0000000000ecb0c] .__lock_acquire+0x196c/0x1d30 [ 20.508192] [c0000000790c2e50] [c0000000000ed5b4] .lock_acquire+0x84/0x100 [ 20.508196] [c0000000790c2f20] [c000000000820448] .mutex_lock_nested+0xa8/0x590 [ 20.508201] [c0000000790c3030] [c000000000052830] .pmac_i2c_open+0x30/0x100 [ 20.508206] [c0000000790c30c0] [c000000000052e14] .pmac_i2c_do_begin+0x34/0x120 [ 20.508209] [c0000000790c3150] [c000000000056bc0] .pmf_call_one+0x50/0xd0 [ 20.508213] [c0000000790c31e0] [c00000000068ff1c] .g5_pfunc_switch_volt+0x2c/0xc0 [ 20.508217] [c0000000790c3250] [c00000000068fecc] .g5_pfunc_switch_freq+0x1cc/0x1f0 [ 20.508221] [c0000000790c3320] [c00000000068fc2c] .g5_cpufreq_target+0x2c/0x40 [ 20.508226] [c0000000790c3390] [c0000000006873ec] .__cpufreq_driver_target+0x23c/0x840 [ 20.508230] [c0000000790c3440] [c00000000068c798] .cpufreq_gov_performance_limits+0x18/0x30 [ 20.508235] [c0000000790c34b0] [c00000000068915c] .cpufreq_start_governor+0xac/0x100 [ 20.508239] [c0000000790c3530] [c00000000068a788] .cpufreq_set_policy+0x208/0x260 [ 20.508244] [c0000000790c35d0] [c00000000068abdc] .cpufreq_init_policy+0x6c/0xb0 [ 20.508249] [c0000000790c3940] [c00000000068ae70] .cpufreq_online+0x250/0x9d0 [ 20.508253] [c0000000790c3a30] [c0000000004d76d8] .subsys_interface_register+0xb8/0x110 [ 20.508258] [c0000000790c3ad0] [c000000000689bb0] .cpufreq_register_driver+0x1d0/0x250 [ 20.508262] [c0000000790c3b60] [c000000000b4f8f4] .g5_cpufreq_init+0x9cc/0xa28 [ 20.508267] [c0000000790c3c20] [c00000000000a98c] .do_one_initcall+0x5c/0x1d0 [ 20.508271] [c0000000790c3d00] [c000000000b0f86c] .kernel_init_freeable+0x1ac/0x28c [ 20.508276] [c0000000790c3db0] [c00000000000b3bc] .kernel_init+0x1c/0x140 [ 20.508280] [c0000000790c3e30] [c0000000000098f4] .ret_from_kernel_thread+0x58/0x64 Signed-off-by: Denis Kirjanov <kda@linux-powerpc.org> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>
0lvin
pushed a commit
to free-z4u/roc-rk3328-cc-official
that referenced
this issue
Jul 18, 2019
As Eric Dumazet pointed out this also needs to be fixed in IPv6. v2: Contains the IPv6 tcp/Ipv6 dccp patches as well. We have seen a few incidents lately where a dst_enty has been freed with a dangling TCP socket reference (sk->sk_dst_cache) pointing to that dst_entry. If the conditions/timings are right a crash then ensues when the freed dst_entry is referenced later on. A Common crashing back trace is: FireflyTeam#8 [] page_fault at ffffffff8163e648 [exception RIP: __tcp_ack_snd_check+74] . . FireflyTeam#9 [] tcp_rcv_established at ffffffff81580b64 FireflyTeam#10 [] tcp_v4_do_rcv at ffffffff8158b54a FireflyTeam#11 [] tcp_v4_rcv at ffffffff8158cd02 FireflyTeam#12 [] ip_local_deliver_finish at ffffffff815668f4 FireflyTeam#13 [] ip_local_deliver at ffffffff81566bd9 FireflyTeam#14 [] ip_rcv_finish at ffffffff8156656d FireflyTeam#15 [] ip_rcv at ffffffff81566f06 FireflyTeam#16 [] __netif_receive_skb_core at ffffffff8152b3a2 rockchip-linux#17 [] __netif_receive_skb at ffffffff8152b608 rockchip-linux#18 [] netif_receive_skb at ffffffff8152b690 rockchip-linux#19 [] vmxnet3_rq_rx_complete at ffffffffa015eeaf [vmxnet3] rockchip-linux#20 [] vmxnet3_poll_rx_only at ffffffffa015f32a [vmxnet3] rockchip-linux#21 [] net_rx_action at ffffffff8152bac2 rockchip-linux#22 [] __do_softirq at ffffffff81084b4f rockchip-linux#23 [] call_softirq at ffffffff8164845c rockchip-linux#24 [] do_softirq at ffffffff81016fc5 rockchip-linux#25 [] irq_exit at ffffffff81084ee5 rockchip-linux#26 [] do_IRQ at ffffffff81648ff8 Of course it may happen with other NIC drivers as well. It's found the freed dst_entry here: 224 static bool tcp_in_quickack_mode(struct sock *sk)↩ 225 {↩ 226 ▹ const struct inet_connection_sock *icsk = inet_csk(sk);↩ 227 ▹ const struct dst_entry *dst = __sk_dst_get(sk);↩ 228 ↩ 229 ▹ return (dst && dst_metric(dst, RTAX_QUICKACK)) ||↩ 230 ▹ ▹ (icsk->icsk_ack.quick && !icsk->icsk_ack.pingpong);↩ 231 }↩ But there are other backtraces attributed to the same freed dst_entry in netfilter code as well. All the vmcores showed 2 significant clues: - Remote hosts behind the default gateway had always been redirected to a different gateway. A rtable/dst_entry will be added for that host. Making more dst_entrys with lower reference counts. Making this more probable. - All vmcores showed a postitive LockDroppedIcmps value, e.g: LockDroppedIcmps 267 A closer look at the tcp_v4_err() handler revealed that do_redirect() will run regardless of whether user space has the socket locked. This can result in a race condition where the same dst_entry cached in sk->sk_dst_entry can be decremented twice for the same socket via: do_redirect()->__sk_dst_check()-> dst_release(). Which leads to the dst_entry being prematurely freed with another socket pointing to it via sk->sk_dst_cache and a subsequent crash. To fix this skip do_redirect() if usespace has the socket locked. Instead let the redirect take place later when user space does not have the socket locked. The dccp/IPv6 code is very similar in this respect, so fixing it there too. As Eric Garver pointed out the following commit now invalidates routes. Which can set the dst->obsolete flag so that ipv4_dst_check() returns null and triggers the dst_release(). Fixes: ceb3320 ("ipv4: Kill routes during PMTU/redirect updates.") Cc: Eric Garver <egarver@redhat.com> Cc: Hannes Sowa <hsowa@redhat.com> Signed-off-by: Jon Maxwell <jmaxwell37@gmail.com> Signed-off-by: David S. Miller <davem@davemloft.net>
0lvin
pushed a commit
to free-z4u/roc-rk3328-cc-official
that referenced
this issue
Jul 26, 2019
When the final cifsFileInfo_put() is called from cifsiod and an oplock break work is queued, lockdep complains loudly: ============================================= [ INFO: possible recursive locking detected ] 4.11.0+ rockchip-linux#21 Not tainted --------------------------------------------- kworker/0:2/78 is trying to acquire lock: ("cifsiod"){++++.+}, at: flush_work+0x215/0x350 but task is already holding lock: ("cifsiod"){++++.+}, at: process_one_work+0x255/0x8e0 other info that might help us debug this: Possible unsafe locking scenario: CPU0 ---- lock("cifsiod"); lock("cifsiod"); *** DEADLOCK *** May be due to missing lock nesting notation 2 locks held by kworker/0:2/78: #0: ("cifsiod"){++++.+}, at: process_one_work+0x255/0x8e0 FireflyTeam#1: ((&wdata->work)){+.+...}, at: process_one_work+0x255/0x8e0 stack backtrace: CPU: 0 PID: 78 Comm: kworker/0:2 Not tainted 4.11.0+ rockchip-linux#21 Workqueue: cifsiod cifs_writev_complete Call Trace: dump_stack+0x85/0xc2 __lock_acquire+0x17dd/0x2260 ? match_held_lock+0x20/0x2b0 ? trace_hardirqs_off_caller+0x86/0x130 ? mark_lock+0xa6/0x920 lock_acquire+0xcc/0x260 ? lock_acquire+0xcc/0x260 ? flush_work+0x215/0x350 flush_work+0x236/0x350 ? flush_work+0x215/0x350 ? destroy_worker+0x170/0x170 __cancel_work_timer+0x17d/0x210 ? ___preempt_schedule+0x16/0x18 cancel_work_sync+0x10/0x20 cifsFileInfo_put+0x338/0x7f0 cifs_writedata_release+0x2a/0x40 ? cifs_writedata_release+0x2a/0x40 cifs_writev_complete+0x29d/0x850 ? preempt_count_sub+0x18/0xd0 process_one_work+0x304/0x8e0 worker_thread+0x9b/0x6a0 kthread+0x1b2/0x200 ? process_one_work+0x8e0/0x8e0 ? kthread_create_on_node+0x40/0x40 ret_from_fork+0x31/0x40 This is a real warning. Since the oplock is queued on the same workqueue this can deadlock if there is only one worker thread active for the workqueue (which will be the case during memory pressure when the rescuer thread is handling it). Furthermore, there is at least one other kind of hang possible due to the oplock break handling if there is only worker. (This can be reproduced without introducing memory pressure by having passing 1 for the max_active parameter of cifsiod.) cifs_oplock_break() can wait indefintely in the filemap_fdatawait() while the cifs_writev_complete() work is blocked: sysrq: SysRq : Show Blocked State task PC stack pid father kworker/0:1 D 0 16 2 0x00000000 Workqueue: cifsiod cifs_oplock_break Call Trace: __schedule+0x562/0xf40 ? mark_held_locks+0x4a/0xb0 schedule+0x57/0xe0 io_schedule+0x21/0x50 wait_on_page_bit+0x143/0x190 ? add_to_page_cache_lru+0x150/0x150 __filemap_fdatawait_range+0x134/0x190 ? do_writepages+0x51/0x70 filemap_fdatawait_range+0x14/0x30 filemap_fdatawait+0x3b/0x40 cifs_oplock_break+0x651/0x710 ? preempt_count_sub+0x18/0xd0 process_one_work+0x304/0x8e0 worker_thread+0x9b/0x6a0 kthread+0x1b2/0x200 ? process_one_work+0x8e0/0x8e0 ? kthread_create_on_node+0x40/0x40 ret_from_fork+0x31/0x40 dd D 0 683 171 0x00000000 Call Trace: __schedule+0x562/0xf40 ? mark_held_locks+0x29/0xb0 schedule+0x57/0xe0 io_schedule+0x21/0x50 wait_on_page_bit+0x143/0x190 ? add_to_page_cache_lru+0x150/0x150 __filemap_fdatawait_range+0x134/0x190 ? do_writepages+0x51/0x70 filemap_fdatawait_range+0x14/0x30 filemap_fdatawait+0x3b/0x40 filemap_write_and_wait+0x4e/0x70 cifs_flush+0x6a/0xb0 filp_close+0x52/0xa0 __close_fd+0xdc/0x150 SyS_close+0x33/0x60 entry_SYSCALL_64_fastpath+0x1f/0xbe Showing all locks held in the system: 2 locks held by kworker/0:1/16: #0: ("cifsiod"){.+.+.+}, at: process_one_work+0x255/0x8e0 FireflyTeam#1: ((&cfile->oplock_break)){+.+.+.}, at: process_one_work+0x255/0x8e0 Showing busy workqueues and worker pools: workqueue cifsiod: flags=0xc pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/1 in-flight: 16:cifs_oplock_break delayed: cifs_writev_complete, cifs_echo_request pool 0: cpus=0 node=0 flags=0x0 nice=0 hung=0s workers=3 idle: 750 3 Fix these problems by creating a a new workqueue (with a rescuer) for the oplock break work. Signed-off-by: Rabin Vincent <rabinv@axis.com> Signed-off-by: Steve French <smfrench@gmail.com> CC: Stable <stable@vger.kernel.org>
0lvin
pushed a commit
to free-z4u/roc-rk3328-cc-official
that referenced
this issue
Sep 20, 2019
When RCU stall warning triggers, it can print out a lot of messages while holding spinlocks. If the console device is slow (e.g. an actual or IPMI serial console), it may end up triggering NMI hard lockup watchdog like the following. *** CPU printking while holding RCU spinlock PID: 4149739 TASK: ffff881a46baa880 CPU: 13 COMMAND: "CPUThreadPool8" #0 [ffff881fff945e48] crash_nmi_callback at ffffffff8103f7d0 FireflyTeam#1 [ffff881fff945e58] nmi_handle at ffffffff81020653 FireflyTeam#2 [ffff881fff945eb0] default_do_nmi at ffffffff81020c36 FireflyTeam#3 [ffff881fff945ed0] do_nmi at ffffffff81020d32 FireflyTeam#4 [ffff881fff945ef0] end_repeat_nmi at ffffffff81956a7e [exception RIP: io_serial_in+21] RIP: ffffffff81630e55 RSP: ffff881fff943b88 RFLAGS: 00000002 RAX: 000000000000ca00 RBX: ffffffff8230e188 RCX: 0000000000000000 RDX: 00000000000002fd RSI: 0000000000000005 RDI: ffffffff8230e188 RBP: ffff881fff943bb0 R8: 0000000000000000 R9: ffffffff820cb3c4 R10: 0000000000000019 R11: 0000000000002000 R12: 00000000000026e1 R13: 0000000000000020 R14: ffffffff820cd398 R15: 0000000000000035 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0000 --- <NMI exception stack> --- FireflyTeam#5 [ffff881fff943b88] io_serial_in at ffffffff81630e55 FireflyTeam#6 [ffff881fff943b90] wait_for_xmitr at ffffffff8163175c FireflyTeam#7 [ffff881fff943bb8] serial8250_console_putchar at ffffffff816317dc FireflyTeam#8 [ffff881fff943bd8] uart_console_write at ffffffff8162ac00 FireflyTeam#9 [ffff881fff943c08] serial8250_console_write at ffffffff81634691 FireflyTeam#10 [ffff881fff943c80] univ8250_console_write at ffffffff8162f7c2 FireflyTeam#11 [ffff881fff943c90] console_unlock at ffffffff810dfc55 FireflyTeam#12 [ffff881fff943cf0] vprintk_emit at ffffffff810dffb5 FireflyTeam#13 [ffff881fff943d50] vprintk_default at ffffffff810e01bf FireflyTeam#14 [ffff881fff943d60] vprintk_func at ffffffff810e1127 FireflyTeam#15 [ffff881fff943d70] printk at ffffffff8119a8a4 FireflyTeam#16 [ffff881fff943dd0] print_cpu_stall_info at ffffffff810eb78c rockchip-linux#17 [ffff881fff943e88] rcu_check_callbacks at ffffffff810ef133 rockchip-linux#18 [ffff881fff943ee8] update_process_times at ffffffff810f3497 rockchip-linux#19 [ffff881fff943f10] tick_sched_timer at ffffffff81103037 rockchip-linux#20 [ffff881fff943f38] __hrtimer_run_queues at ffffffff810f3f38 rockchip-linux#21 [ffff881fff943f88] hrtimer_interrupt at ffffffff810f442b *** CPU triggering the hardlockup watchdog PID: 4149709 TASK: ffff88010f88c380 CPU: 26 COMMAND: "CPUThreadPool35" #0 [ffff883fff1059d0] machine_kexec at ffffffff8104a874 FireflyTeam#1 [ffff883fff105a30] __crash_kexec at ffffffff811116cc FireflyTeam#2 [ffff883fff105af0] __crash_kexec at ffffffff81111795 FireflyTeam#3 [ffff883fff105b08] panic at ffffffff8119a6ae FireflyTeam#4 [ffff883fff105b98] watchdog_overflow_callback at ffffffff81135dbd FireflyTeam#5 [ffff883fff105bb0] __perf_event_overflow at ffffffff81186866 FireflyTeam#6 [ffff883fff105be8] perf_event_overflow at ffffffff81192bc4 FireflyTeam#7 [ffff883fff105bf8] intel_pmu_handle_irq at ffffffff8100b265 FireflyTeam#8 [ffff883fff105df8] perf_event_nmi_handler at ffffffff8100489f FireflyTeam#9 [ffff883fff105e58] nmi_handle at ffffffff81020653 FireflyTeam#10 [ffff883fff105eb0] default_do_nmi at ffffffff81020b94 FireflyTeam#11 [ffff883fff105ed0] do_nmi at ffffffff81020d32 FireflyTeam#12 [ffff883fff105ef0] end_repeat_nmi at ffffffff81956a7e [exception RIP: queued_spin_lock_slowpath+248] RIP: ffffffff810da958 RSP: ffff883fff103e68 RFLAGS: 00000046 RAX: 0000000000000000 RBX: 0000000000000046 RCX: 00000000006d0000 RDX: ffff883fff49a950 RSI: 0000000000d10101 RDI: ffffffff81e54300 RBP: ffff883fff103e80 R8: ffff883fff11a950 R9: 0000000000000000 R10: 000000000e5873ba R11: 000000000000010f R12: ffffffff81e54300 R13: 0000000000000000 R14: ffff88010f88c380 R15: ffffffff81e54300 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 --- <NMI exception stack> --- FireflyTeam#13 [ffff883fff103e68] queued_spin_lock_slowpath at ffffffff810da958 FireflyTeam#14 [ffff883fff103e70] _raw_spin_lock_irqsave at ffffffff8195550b FireflyTeam#15 [ffff883fff103e88] rcu_check_callbacks at ffffffff810eed18 FireflyTeam#16 [ffff883fff103ee8] update_process_times at ffffffff810f3497 rockchip-linux#17 [ffff883fff103f10] tick_sched_timer at ffffffff81103037 rockchip-linux#18 [ffff883fff103f38] __hrtimer_run_queues at ffffffff810f3f38 rockchip-linux#19 [ffff883fff103f88] hrtimer_interrupt at ffffffff810f442b --- <IRQ stack> --- Avoid spuriously triggering NMI hardlockup watchdog by touching it from the print functions. show_state_filter() shares the same problem and solution. v2: Relocate the comment to where it belongs. Signed-off-by: Tejun Heo <tj@kernel.org> Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
0lvin
pushed a commit
to free-z4u/roc-rk3328-cc-official
that referenced
this issue
Sep 21, 2019
Commit 9b61e30 (spi: Pick spi bus number from Linux idr or spi alias) ceased to unregister SPI buses with fixed bus numbers. Moreover this is visible only if CONFIG_SPI_DEBUG=y is set or when trying to re-register the same SPI controller. rmmod spi_pxa2xx_platform (with CONFIG_SPI_DEBUG=y): [ 26.788362] spi_master spi1: attempting to delete unregistered controller [spi1] modprobe spi_pxa2xx_platform: [ 37.883137] sysfs: cannot create duplicate filename '/devices/pci0000:00/0000:00:19.0/pxa2xx-spi.12/spi_master/spi1' [ 37.894984] CPU: 1 PID: 1467 Comm: modprobe Not tainted 4.16.0-rc4+ rockchip-linux#21 [ 37.902384] Call Trace: ... [ 38.122680] kobject_add_internal failed for spi1 with -EEXIST, don't try to register things with the same name in the same directory. [ 38.136154] WARNING: CPU: 1 PID: 1467 at lib/kobject.c:238 kobject_add_internal+0x2a5/0x2f0 ... [ 38.513817] pxa2xx-spi pxa2xx-spi.12: problem registering spi master [ 38.521036] pxa2xx-spi: probe of pxa2xx-spi.12 failed with error -17 Fix this by not returning immediately from spi_unregister_controller() if idr_find() doesn't find controller with given ID/bus number. It finds only those controllers that were registered with dynamic SPI bus numbers. Only conditional cleanup between dynamic and fixed bus numbers is to remove allocated IDR. Fixes: 9b61e30 (spi: Pick spi bus number from Linux idr or spi alias) Cc: stable@vger.kernel.org Signed-off-by: Jarkko Nikula <jarkko.nikula@linux.intel.com> Signed-off-by: Mark Brown <broonie@kernel.org>
0lvin
pushed a commit
to free-z4u/roc-rk3328-cc-official
that referenced
this issue
Sep 21, 2019
when mounting an ISO filesystem sometimes (very rarely) the system hangs because of a race condition between two tasks. PID: 6766 TASK: ffff88007b2a6dd0 CPU: 0 COMMAND: "mount" #0 [ffff880078447ae0] __schedule at ffffffff8168d605 FireflyTeam#1 [ffff880078447b48] schedule_preempt_disabled at ffffffff8168ed49 FireflyTeam#2 [ffff880078447b58] __mutex_lock_slowpath at ffffffff8168c995 FireflyTeam#3 [ffff880078447bb8] mutex_lock at ffffffff8168bdef FireflyTeam#4 [ffff880078447bd0] sr_block_ioctl at ffffffffa00b6818 [sr_mod] FireflyTeam#5 [ffff880078447c10] blkdev_ioctl at ffffffff812fea50 FireflyTeam#6 [ffff880078447c70] ioctl_by_bdev at ffffffff8123a8b3 FireflyTeam#7 [ffff880078447c90] isofs_fill_super at ffffffffa04fb1e1 [isofs] FireflyTeam#8 [ffff880078447da8] mount_bdev at ffffffff81202570 FireflyTeam#9 [ffff880078447e18] isofs_mount at ffffffffa04f9828 [isofs] FireflyTeam#10 [ffff880078447e28] mount_fs at ffffffff81202d09 FireflyTeam#11 [ffff880078447e70] vfs_kern_mount at ffffffff8121ea8f FireflyTeam#12 [ffff880078447ea8] do_mount at ffffffff81220fee FireflyTeam#13 [ffff880078447f28] sys_mount at ffffffff812218d6 FireflyTeam#14 [ffff880078447f80] system_call_fastpath at ffffffff81698c49 RIP: 00007fd9ea914e9a RSP: 00007ffd5d9bf648 RFLAGS: 00010246 RAX: 00000000000000a5 RBX: ffffffff81698c49 RCX: 0000000000000010 RDX: 00007fd9ec2bc210 RSI: 00007fd9ec2bc290 RDI: 00007fd9ec2bcf30 RBP: 0000000000000000 R8: 0000000000000000 R9: 0000000000000010 R10: 00000000c0ed0001 R11: 0000000000000206 R12: 00007fd9ec2bc040 R13: 00007fd9eb6b2380 R14: 00007fd9ec2bc210 R15: 00007fd9ec2bcf30 ORIG_RAX: 00000000000000a5 CS: 0033 SS: 002b This task was trying to mount the cdrom. It allocated and configured a super_block struct and owned the write-lock for the super_block->s_umount rwsem. While exclusively owning the s_umount lock, it called sr_block_ioctl and waited to acquire the global sr_mutex lock. PID: 6785 TASK: ffff880078720fb0 CPU: 0 COMMAND: "systemd-udevd" #0 [ffff880078417898] __schedule at ffffffff8168d605 FireflyTeam#1 [ffff880078417900] schedule at ffffffff8168dc59 FireflyTeam#2 [ffff880078417910] rwsem_down_read_failed at ffffffff8168f605 FireflyTeam#3 [ffff880078417980] call_rwsem_down_read_failed at ffffffff81328838 FireflyTeam#4 [ffff8800784179d0] down_read at ffffffff8168cde0 FireflyTeam#5 [ffff8800784179e8] get_super at ffffffff81201cc7 FireflyTeam#6 [ffff880078417a10] __invalidate_device at ffffffff8123a8de FireflyTeam#7 [ffff880078417a40] flush_disk at ffffffff8123a94b FireflyTeam#8 [ffff880078417a88] check_disk_change at ffffffff8123ab50 FireflyTeam#9 [ffff880078417ab0] cdrom_open at ffffffffa00a29e1 [cdrom] FireflyTeam#10 [ffff880078417b68] sr_block_open at ffffffffa00b6f9b [sr_mod] FireflyTeam#11 [ffff880078417b98] __blkdev_get at ffffffff8123ba86 FireflyTeam#12 [ffff880078417bf0] blkdev_get at ffffffff8123bd65 FireflyTeam#13 [ffff880078417c78] blkdev_open at ffffffff8123bf9b FireflyTeam#14 [ffff880078417c90] do_dentry_open at ffffffff811fc7f7 FireflyTeam#15 [ffff880078417cd8] vfs_open at ffffffff811fc9cf FireflyTeam#16 [ffff880078417d00] do_last at ffffffff8120d53d rockchip-linux#17 [ffff880078417db0] path_openat at ffffffff8120e6b2 rockchip-linux#18 [ffff880078417e48] do_filp_open at ffffffff8121082b rockchip-linux#19 [ffff880078417f18] do_sys_open at ffffffff811fdd33 rockchip-linux#20 [ffff880078417f70] sys_open at ffffffff811fde4e rockchip-linux#21 [ffff880078417f80] system_call_fastpath at ffffffff81698c49 RIP: 00007f29438b0c20 RSP: 00007ffc76624b78 RFLAGS: 00010246 RAX: 0000000000000002 RBX: ffffffff81698c49 RCX: 0000000000000000 RDX: 00007f2944a5fa70 RSI: 00000000000a0800 RDI: 00007f2944a5fa70 RBP: 00007f2944a5f540 R8: 0000000000000000 R9: 0000000000000020 R10: 00007f2943614c40 R11: 0000000000000246 R12: ffffffff811fde4e R13: ffff880078417f78 R14: 000000000000000c R15: 00007f2944a4b010 ORIG_RAX: 0000000000000002 CS: 0033 SS: 002b This task tried to open the cdrom device, the sr_block_open function acquired the global sr_mutex lock. The call to check_disk_change() then saw an event flag indicating a possible media change and tried to flush any cached data for the device. As part of the flush, it tried to acquire the super_block->s_umount lock associated with the cdrom device. This was the same super_block as created and locked by the previous task. The first task acquires the s_umount lock and then the sr_mutex_lock; the second task acquires the sr_mutex_lock and then the s_umount lock. This patch fixes the issue by moving check_disk_change() out of cdrom_open() and let the caller take care of it. Signed-off-by: Maurizio Lombardi <mlombard@redhat.com> Signed-off-by: Jens Axboe <axboe@kernel.dk>
0lvin
pushed a commit
to free-z4u/roc-rk3328-cc-official
that referenced
this issue
Sep 29, 2019
[ Upstream commit 2542604 ] When a matchall classifier is added, there is a small time interval in which tp->root is NULL. If we receive a packet in this small time slice a NULL pointer dereference will happen, leading to a kernel panic: # tc qdisc replace dev eth0 ingress # tc filter add dev eth0 parent ffff: matchall action gact drop Unable to handle kernel NULL pointer dereference at virtual address 0000000000000034 Mem abort info: ESR = 0x96000005 Exception class = DABT (current EL), IL = 32 bits SET = 0, FnV = 0 EA = 0, S1PTW = 0 Data abort info: ISV = 0, ISS = 0x00000005 CM = 0, WnR = 0 user pgtable: 4k pages, 39-bit VAs, pgdp = 00000000a623d530 [0000000000000034] pgd=0000000000000000, pud=0000000000000000 Internal error: Oops: 96000005 [FireflyTeam#1] SMP Modules linked in: cls_matchall sch_ingress nls_iso8859_1 nls_cp437 vfat fat m25p80 spi_nor mtd xhci_plat_hcd xhci_hcd phy_generic sfp mdio_i2c usbcore i2c_mv64xxx marvell10g mvpp2 usb_common spi_orion mvmdio i2c_core sbsa_gwdt phylink ip_tables x_tables autofs4 Process ksoftirqd/0 (pid: 9, stack limit = 0x0000000009de7d62) CPU: 0 PID: 9 Comm: ksoftirqd/0 Not tainted 5.1.0-rc6 rockchip-linux#21 Hardware name: Marvell 8040 MACCHIATOBin Double-shot (DT) pstate: 40000005 (nZcv daif -PAN -UAO) pc : mall_classify+0x28/0x78 [cls_matchall] lr : tcf_classify+0x78/0x138 sp : ffffff80109db9d0 x29: ffffff80109db9d0 x28: ffffffc426058800 x27: 0000000000000000 x26: ffffffc425b0dd00 x25: 0000000020000000 x24: 0000000000000000 x23: ffffff80109dbac0 x22: 0000000000000001 x21: ffffffc428ab5100 x20: ffffffc425b0dd00 x19: ffffff80109dbac0 x18: 0000000000000000 x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000000 x14: 0000000000000000 x13: ffffffbf108ad288 x12: dead000000000200 x11: 00000000f0000000 x10: 0000000000000001 x9 : ffffffbf1089a220 x8 : 0000000000000001 x7 : ffffffbebffaa950 x6 : 0000000000000000 x5 : 000000442d6ba000 x4 : 0000000000000000 x3 : ffffff8008735ad8 x2 : ffffff80109dbac0 x1 : ffffffc425b0dd00 x0 : ffffff8010592078 Call trace: mall_classify+0x28/0x78 [cls_matchall] tcf_classify+0x78/0x138 __netif_receive_skb_core+0x29c/0xa20 __netif_receive_skb_one_core+0x34/0x60 __netif_receive_skb+0x28/0x78 netif_receive_skb_internal+0x2c/0xc0 napi_gro_receive+0x1a0/0x1d8 mvpp2_poll+0x928/0xb18 [mvpp2] net_rx_action+0x108/0x378 __do_softirq+0x128/0x320 run_ksoftirqd+0x44/0x60 smpboot_thread_fn+0x168/0x1b0 kthread+0x12c/0x130 ret_from_fork+0x10/0x1c Code: aa0203f3 aa1e03e0 d503201f f9400684 (b9403480) ---[ end trace fc71e2ef7b8ab5a5 ]--- Kernel panic - not syncing: Fatal exception in interrupt SMP: stopping secondary CPUs Kernel Offset: disabled CPU features: 0x002,00002000 Memory Limit: none Rebooting in 1 seconds.. Fix this by adding a NULL check in mall_classify(). Fixes: ed76f5e ("net: sched: protect filter_chain list with filter_chain_lock mutex") Signed-off-by: Matteo Croce <mcroce@redhat.com> Acked-by: Cong Wang <xiyou.wangcong@gmail.com> Signed-off-by: David S. Miller <davem@davemloft.net> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
0lvin
pushed a commit
to free-z4u/roc-rk3328-cc-official
that referenced
this issue
Oct 5, 2019
Crash dump shows following instructions crash> bt PID: 0 TASK: ffffffffbe412480 CPU: 0 COMMAND: "swapper/0" #0 [ffff891ee0003868] machine_kexec at ffffffffbd063ef1 FireflyTeam#1 [ffff891ee00038c8] __crash_kexec at ffffffffbd12b6f2 FireflyTeam#2 [ffff891ee0003998] crash_kexec at ffffffffbd12c84c FireflyTeam#3 [ffff891ee00039b8] oops_end at ffffffffbd030f0a FireflyTeam#4 [ffff891ee00039e0] no_context at ffffffffbd074643 FireflyTeam#5 [ffff891ee0003a40] __bad_area_nosemaphore at ffffffffbd07496e FireflyTeam#6 [ffff891ee0003a90] bad_area_nosemaphore at ffffffffbd074a64 FireflyTeam#7 [ffff891ee0003aa0] __do_page_fault at ffffffffbd074b0a FireflyTeam#8 [ffff891ee0003b18] do_page_fault at ffffffffbd074fc8 FireflyTeam#9 [ffff891ee0003b50] page_fault at ffffffffbda01925 [exception RIP: qlt_schedule_sess_for_deletion+15] RIP: ffffffffc02e526f RSP: ffff891ee0003c08 RFLAGS: 00010046 RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffffffffc0307847 RDX: 00000000000020e6 RSI: ffff891edbc377c8 RDI: 0000000000000000 RBP: ffff891ee0003c18 R8: ffffffffc02f0b20 R9: 0000000000000250 R10: 0000000000000258 R11: 000000000000b780 R12: ffff891ed9b43000 R13: 00000000000000f0 R14: 0000000000000006 R15: ffff891edbc377c8 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 FireflyTeam#10 [ffff891ee0003c20] qla2x00_fcport_event_handler at ffffffffc02853d3 [qla2xxx] FireflyTeam#11 [ffff891ee0003cf0] __dta_qla24xx_async_gnl_sp_done_333 at ffffffffc0285a1d [qla2xxx] FireflyTeam#12 [ffff891ee0003de8] qla24xx_process_response_queue at ffffffffc02a2eb5 [qla2xxx] FireflyTeam#13 [ffff891ee0003e88] qla24xx_msix_rsp_q at ffffffffc02a5403 [qla2xxx] FireflyTeam#14 [ffff891ee0003ec0] __handle_irq_event_percpu at ffffffffbd0f4c59 FireflyTeam#15 [ffff891ee0003f10] handle_irq_event_percpu at ffffffffbd0f4e02 FireflyTeam#16 [ffff891ee0003f40] handle_irq_event at ffffffffbd0f4e90 rockchip-linux#17 [ffff891ee0003f68] handle_edge_irq at ffffffffbd0f8984 rockchip-linux#18 [ffff891ee0003f88] handle_irq at ffffffffbd0305d5 rockchip-linux#19 [ffff891ee0003fb8] do_IRQ at ffffffffbda02a18 --- <IRQ stack> --- rockchip-linux#20 [ffffffffbe403d30] ret_from_intr at ffffffffbda0094e [exception RIP: unknown or invalid address] RIP: 000000000000001f RSP: 0000000000000000 RFLAGS: fff3b8c2091ebb3f RAX: ffffbba5a0000200 RBX: 0000be8cdfa8f9fa RCX: 0000000000000018 RDX: 0000000000000101 RSI: 000000000000015d RDI: 0000000000000193 RBP: 0000000000000083 R8: ffffffffbe403e38 R9: 0000000000000002 R10: 0000000000000000 R11: ffffffffbe56b820 R12: ffff891ee001cf00 R13: ffffffffbd11c0a4 R14: ffffffffbe403d60 R15: 0000000000000001 ORIG_RAX: ffff891ee0022ac0 CS: 0000 SS: ffffffffffffffb9 bt: WARNING: possibly bogus exception frame rockchip-linux#21 [ffffffffbe403dd8] cpuidle_enter_state at ffffffffbd67c6fd rockchip-linux#22 [ffffffffbe403e40] cpuidle_enter at ffffffffbd67c907 rockchip-linux#23 [ffffffffbe403e50] call_cpuidle at ffffffffbd0d98f3 rockchip-linux#24 [ffffffffbe403e60] do_idle at ffffffffbd0d9b42 rockchip-linux#25 [ffffffffbe403e98] cpu_startup_entry at ffffffffbd0d9da3 rockchip-linux#26 [ffffffffbe403ec0] rest_init at ffffffffbd81d4aa rockchip-linux#27 [ffffffffbe403ed0] start_kernel at ffffffffbe67d2ca rockchip-linux#28 [ffffffffbe403f28] x86_64_start_reservations at ffffffffbe67c675 rockchip-linux#29 [ffffffffbe403f38] x86_64_start_kernel at ffffffffbe67c6eb rockchip-linux#30 [ffffffffbe403f50] secondary_startup_64 at ffffffffbd0000d5 Fixes: 040036b ("scsi: qla2xxx: Delay loop id allocation at login") Cc: <stable@vger.kernel.org> # v4.17+ Signed-off-by: Chuck Anderson <chuck.anderson@oracle.com> Signed-off-by: Himanshu Madhani <himanshu.madhani@cavium.com> Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
0lvin
pushed a commit
to free-z4u/roc-rk3328-cc-official
that referenced
this issue
Oct 12, 2019
pid_task() dereferences rcu protected tasks array. But there is no rcu_read_lock() in shutdown_umh() routine so that rcu_read_lock() is needed. get_pid_task() is wrapper function of pid_task. it holds rcu_read_lock() then calls pid_task(). if task isn't NULL, it increases reference count of task. test commands: %modprobe bpfilter %modprobe -rv bpfilter splat looks like: [15102.030932] ============================= [15102.030957] WARNING: suspicious RCU usage [15102.030985] 4.19.0-rc7+ rockchip-linux#21 Not tainted [15102.031010] ----------------------------- [15102.031038] kernel/pid.c:330 suspicious rcu_dereference_check() usage! [15102.031063] other info that might help us debug this: [15102.031332] rcu_scheduler_active = 2, debug_locks = 1 [15102.031363] 1 lock held by modprobe/1570: [15102.031389] #0: 00000000580ef2b0 (bpfilter_lock){+.+.}, at: stop_umh+0x13/0x52 [bpfilter] [15102.031552] stack backtrace: [15102.031583] CPU: 1 PID: 1570 Comm: modprobe Not tainted 4.19.0-rc7+ rockchip-linux#21 [15102.031607] Hardware name: To be filled by O.E.M. To be filled by O.E.M./Aptio CRB, BIOS 5.6.5 07/08/2015 [15102.031628] Call Trace: [15102.031676] dump_stack+0xc9/0x16b [15102.031723] ? show_regs_print_info+0x5/0x5 [15102.031801] ? lockdep_rcu_suspicious+0x117/0x160 [15102.031855] pid_task+0x134/0x160 [15102.031900] ? find_vpid+0xf0/0xf0 [15102.032017] shutdown_umh.constprop.1+0x1e/0x53 [bpfilter] [15102.032055] stop_umh+0x46/0x52 [bpfilter] [15102.032092] __x64_sys_delete_module+0x47e/0x570 [ ... ] Fixes: d2ba09c ("net: add skeleton of bpfilter kernel module") Signed-off-by: Taehee Yoo <ap420073@gmail.com> Acked-by: Alexei Starovoitov <ast@kernel.org> Signed-off-by: David S. Miller <davem@davemloft.net>
兄弟，你的烧录步骤根据的是什么手册？另外 你的 loader 用的是哪个？我们也想快速验证一下想法 |
fanck0605
pushed a commit
to fanck0605/friendlywrt-kernel
that referenced
this issue
Apr 27, 2020
[ Upstream commit 1bc7896 ] When experimenting with bpf_send_signal() helper in our production environment (5.2 based), we experienced a deadlock in NMI mode: friendlyarm#5 [ffffc9002219f770] queued_spin_lock_slowpath at ffffffff8110be24 friendlyarm#6 [ffffc9002219f770] _raw_spin_lock_irqsave at ffffffff81a43012 friendlyarm#7 [ffffc9002219f780] try_to_wake_up at ffffffff810e7ecd friendlyarm#8 [ffffc9002219f7e0] signal_wake_up_state at ffffffff810c7b55 rockchip-linux#9 [ffffc9002219f7f0] __send_signal at ffffffff810c8602 rockchip-linux#10 [ffffc9002219f830] do_send_sig_info at ffffffff810ca31a rockchip-linux#11 [ffffc9002219f868] bpf_send_signal at ffffffff8119d227 rockchip-linux#12 [ffffc9002219f988] bpf_overflow_handler at ffffffff811d4140 rockchip-linux#13 [ffffc9002219f9e0] __perf_event_overflow at ffffffff811d68cf rockchip-linux#14 [ffffc9002219fa10] perf_swevent_overflow at ffffffff811d6a09 rockchip-linux#15 [ffffc9002219fa38] ___perf_sw_event at ffffffff811e0f47 rockchip-linux#16 [ffffc9002219fc30] __schedule at ffffffff81a3e04d rockchip-linux#17 [ffffc9002219fc90] schedule at ffffffff81a3e219 rockchip-linux#18 [ffffc9002219fca0] futex_wait_queue_me at ffffffff8113d1b9 rockchip-linux#19 [ffffc9002219fcd8] futex_wait at ffffffff8113e529 rockchip-linux#20 [ffffc9002219fdf0] do_futex at ffffffff8113ffbc rockchip-linux#21 [ffffc9002219fec0] __x64_sys_futex at ffffffff81140d1c rockchip-linux#22 [ffffc9002219ff38] do_syscall_64 at ffffffff81002602 rockchip-linux#23 [ffffc9002219ff50] entry_SYSCALL_64_after_hwframe at ffffffff81c00068 The above call stack is actually very similar to an issue reported by Commit eac9153 ("bpf/stackmap: Fix deadlock with rq_lock in bpf_get_stack()") by Song Liu. The only difference is bpf_send_signal() helper instead of bpf_get_stack() helper. The above deadlock is triggered with a perf_sw_event. Similar to Commit eac9153, the below almost identical reproducer used tracepoint point sched/sched_switch so the issue can be easily caught. /* stress_test.c */ #include <stdio.h> #include <stdlib.h> #include <sys/mman.h> #include <pthread.h> #include <sys/types.h> #include <sys/stat.h> #include <fcntl.h> #define THREAD_COUNT 1000 char *filename; void *worker(void *p) { void *ptr; int fd; char *pptr; fd = open(filename, O_RDONLY); if (fd < 0) return NULL; while (1) { struct timespec ts = {0, 1000 + rand() % 2000}; ptr = mmap(NULL, 4096 * 64, PROT_READ, MAP_PRIVATE, fd, 0); usleep(1); if (ptr == MAP_FAILED) { printf("failed to mmap\n"); break; } munmap(ptr, 4096 * 64); usleep(1); pptr = malloc(1); usleep(1); pptr[0] = 1; usleep(1); free(pptr); usleep(1); nanosleep(&ts, NULL); } close(fd); return NULL; } int main(int argc, char *argv[]) { void *ptr; int i; pthread_t threads[THREAD_COUNT]; if (argc < 2) return 0; filename = argv[1]; for (i = 0; i < THREAD_COUNT; i++) { if (pthread_create(threads + i, NULL, worker, NULL)) { fprintf(stderr, "Error creating thread\n"); return 0; } } for (i = 0; i < THREAD_COUNT; i++) pthread_join(threads[i], NULL); return 0; } and the following command: 1. run `stress_test /bin/ls` in one windown 2. hack bcc trace.py with the following change: # --- a/tools/trace.py # +++ b/tools/trace.py @@ -513,6 +513,7 @@ BPF_PERF_OUTPUT(%s); __data.tgid = __tgid; __data.pid = __pid; bpf_get_current_comm(&__data.comm, sizeof(__data.comm)); + bpf_send_signal(10); %s %s %s.perf_submit(%s, &__data, sizeof(__data)); 3. in a different window run ./trace.py -p $(pidof stress_test) t:sched:sched_switch The deadlock can be reproduced in our production system. Similar to Song's fix, the fix is to delay sending signal if irqs is disabled to avoid deadlocks involving with rq_lock. With this change, my above stress-test in our production system won't cause deadlock any more. I also implemented a scale-down version of reproducer in the selftest (a subsequent commit). With latest bpf-next, it complains for the following potential deadlock. [ 32.832450] -> friendlyarm#1 (&p->pi_lock){-.-.}: [ 32.833100] _raw_spin_lock_irqsave+0x44/0x80 [ 32.833696] task_rq_lock+0x2c/0xa0 [ 32.834182] task_sched_runtime+0x59/0xd0 [ 32.834721] thread_group_cputime+0x250/0x270 [ 32.835304] thread_group_cputime_adjusted+0x2e/0x70 [ 32.835959] do_task_stat+0x8a7/0xb80 [ 32.836461] proc_single_show+0x51/0xb0 ... [ 32.839512] -> #0 (&(&sighand->siglock)->rlock){....}: [ 32.840275] __lock_acquire+0x1358/0x1a20 [ 32.840826] lock_acquire+0xc7/0x1d0 [ 32.841309] _raw_spin_lock_irqsave+0x44/0x80 [ 32.841916] __lock_task_sighand+0x79/0x160 [ 32.842465] do_send_sig_info+0x35/0x90 [ 32.842977] bpf_send_signal+0xa/0x10 [ 32.843464] bpf_prog_bc13ed9e4d3163e3_send_signal_tp_sched+0x465/0x1000 [ 32.844301] trace_call_bpf+0x115/0x270 [ 32.844809] perf_trace_run_bpf_submit+0x4a/0xc0 [ 32.845411] perf_trace_sched_switch+0x10f/0x180 [ 32.846014] __schedule+0x45d/0x880 [ 32.846483] schedule+0x5f/0xd0 ... [ 32.853148] Chain exists of: [ 32.853148] &(&sighand->siglock)->rlock --> &p->pi_lock --> &rq->lock [ 32.853148] [ 32.854451] Possible unsafe locking scenario: [ 32.854451] [ 32.855173] CPU0 CPU1 [ 32.855745] ---- ---- [ 32.856278] lock(&rq->lock); [ 32.856671] lock(&p->pi_lock); [ 32.857332] lock(&rq->lock); [ 32.857999] lock(&(&sighand->siglock)->rlock); Deadlock happens on CPU0 when it tries to acquire &sighand->siglock but it has been held by CPU1 and CPU1 tries to grab &rq->lock and cannot get it. This is not exactly the callstack in our production environment, but sympotom is similar and both locks are using spin_lock_irqsave() to acquire the lock, and both involves rq_lock. The fix to delay sending signal when irq is disabled also fixed this issue. Signed-off-by: Yonghong Song <yhs@fb.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Cc: Song Liu <songliubraving@fb.com> Link: https://lore.kernel.org/bpf/20200304191104.2796501-1-yhs@fb.com Signed-off-by: Sasha Levin <sashal@kernel.org> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
friendlyarm
pushed a commit
to friendlyarm/kernel-rockchip
that referenced
this issue
Aug 31, 2020
[ Upstream commit e24c644 ] I compiled with AddressSanitizer and I had these memory leaks while I was using the tep_parse_format function: Direct leak of 28 byte(s) in 4 object(s) allocated from: #0 0x7fb07db49ffe in __interceptor_realloc (/lib/x86_64-linux-gnu/libasan.so.5+0x10dffe) #1 0x7fb07a724228 in extend_token /home/pduplessis/repo/linux/tools/lib/traceevent/event-parse.c:985 #2 0x7fb07a724c21 in __read_token /home/pduplessis/repo/linux/tools/lib/traceevent/event-parse.c:1140 #3 0x7fb07a724f78 in read_token /home/pduplessis/repo/linux/tools/lib/traceevent/event-parse.c:1206 #4 0x7fb07a725191 in __read_expect_type /home/pduplessis/repo/linux/tools/lib/traceevent/event-parse.c:1291 #5 0x7fb07a7251df in read_expect_type /home/pduplessis/repo/linux/tools/lib/traceevent/event-parse.c:1299 #6 0x7fb07a72e6c8 in process_dynamic_array_len /home/pduplessis/repo/linux/tools/lib/traceevent/event-parse.c:2849 #7 0x7fb07a7304b8 in process_function /home/pduplessis/repo/linux/tools/lib/traceevent/event-parse.c:3161 #8 0x7fb07a730900 in process_arg_token /home/pduplessis/repo/linux/tools/lib/traceevent/event-parse.c:3207 rockchip-linux#9 0x7fb07a727c0b in process_arg /home/pduplessis/repo/linux/tools/lib/traceevent/event-parse.c:1786 rockchip-linux#10 0x7fb07a731080 in event_read_print_args /home/pduplessis/repo/linux/tools/lib/traceevent/event-parse.c:3285 rockchip-linux#11 0x7fb07a731722 in event_read_print /home/pduplessis/repo/linux/tools/lib/traceevent/event-parse.c:3369 rockchip-linux#12 0x7fb07a740054 in __tep_parse_format /home/pduplessis/repo/linux/tools/lib/traceevent/event-parse.c:6335 rockchip-linux#13 0x7fb07a74047a in __parse_event /home/pduplessis/repo/linux/tools/lib/traceevent/event-parse.c:6389 rockchip-linux#14 0x7fb07a740536 in tep_parse_format /home/pduplessis/repo/linux/tools/lib/traceevent/event-parse.c:6431 rockchip-linux#15 0x7fb07a785acf in parse_event ../../../src/fs-src/fs.c:251 rockchip-linux#16 0x7fb07a785ccd in parse_systems ../../../src/fs-src/fs.c:284 rockchip-linux#17 0x7fb07a786fb3 in read_metadata ../../../src/fs-src/fs.c:593 rockchip-linux#18 0x7fb07a78760e in ftrace_fs_source_init ../../../src/fs-src/fs.c:727 rockchip-linux#19 0x7fb07d90c19c in add_component_with_init_method_data ../../../../src/lib/graph/graph.c:1048 rockchip-linux#20 0x7fb07d90c87b in add_source_component_with_initialize_method_data ../../../../src/lib/graph/graph.c:1127 rockchip-linux#21 0x7fb07d90c92a in bt_graph_add_source_component ../../../../src/lib/graph/graph.c:1152 rockchip-linux#22 0x55db11aa632e in cmd_run_ctx_create_components_from_config_components ../../../src/cli/babeltrace2.c:2252 rockchip-linux#23 0x55db11aa6fda in cmd_run_ctx_create_components ../../../src/cli/babeltrace2.c:2347 rockchip-linux#24 0x55db11aa780c in cmd_run ../../../src/cli/babeltrace2.c:2461 rockchip-linux#25 0x55db11aa8a7d in main ../../../src/cli/babeltrace2.c:2673 rockchip-linux#26 0x7fb07d5460b2 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x270b2) The token variable in the process_dynamic_array_len function is allocated in the read_expect_type function, but is not freed before calling the read_token function. Free the token variable before calling read_token in order to plug the leak. Signed-off-by: Philippe Duplessis-Guindon <pduplessis@efficios.com> Reviewed-by: Steven Rostedt (VMware) <rostedt@goodmis.org> Link: https://lore.kernel.org/linux-trace-devel/20200730150236.5392-1-pduplessis@efficios.com Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com> Signed-off-by: Sasha Levin <sashal@kernel.org>
friendlyarm
pushed a commit
to friendlyarm/kernel-rockchip
that referenced
this issue
Aug 31, 2021
[ Upstream commit 43e8f76 ] When using kprobe on powerpc booke series processor, Oops happens as show bellow: / # echo "p:myprobe do_nanosleep" > /sys/kernel/debug/tracing/kprobe_events / # echo 1 > /sys/kernel/debug/tracing/events/kprobes/myprobe/enable / # sleep 1 [ 50.076730] Oops: Exception in kernel mode, sig: 5 [#1] [ 50.077017] BE PAGE_SIZE=4K SMP NR_CPUS=24 QEMU e500 [ 50.077221] Modules linked in: [ 50.077462] CPU: 0 PID: 77 Comm: sleep Not tainted 5.14.0-rc4-00022-g251a1524293d rockchip-linux#21 [ 50.077887] NIP: c0b9c4e0 LR: c00ebecc CTR: 00000000 [ 50.078067] REGS: c3883de0 TRAP: 0700 Not tainted (5.14.0-rc4-00022-g251a1524293d) [ 50.078349] MSR: 00029000 <CE,EE,ME> CR: 24000228 XER: 20000000 [ 50.078675] [ 50.078675] GPR00: c00ebdf0 c3883e90 c313e300 c3883ea0 00000001 00000000 c3883ecc 00000001 [ 50.078675] GPR08: c100598c c00ea250 00000004 00000000 24000222 102490c2 bff4180c 101e60d4 [ 50.078675] GPR16: 00000000 102454ac 00000040 10240000 10241100 102410f8 10240000 00500000 [ 50.078675] GPR24: 00000002 00000000 c3883ea0 00000001 00000000 0000c350 3b9b8d50 00000000 [ 50.080151] NIP [c0b9c4e0] do_nanosleep+0x0/0x190 [ 50.080352] LR [c00ebecc] hrtimer_nanosleep+0x14c/0x1e0 [ 50.080638] Call Trace: [ 50.080801] [c3883e90] [c00ebdf0] hrtimer_nanosleep+0x70/0x1e0 (unreliable) [ 50.081110] [c3883f00] [c00ec004] sys_nanosleep_time32+0xa4/0x110 [ 50.081336] [c3883f40] [c001509c] ret_from_syscall+0x0/0x28 [ 50.081541] --- interrupt: c00 at 0x100a4d08 [ 50.081749] NIP: 100a4d08 LR: 101b5234 CTR: 00000003 [ 50.081931] REGS: c3883f50 TRAP: 0c00 Not tainted (5.14.0-rc4-00022-g251a1524293d) [ 50.082183] MSR: 0002f902 <CE,EE,PR,FP,ME> CR: 24000222 XER: 00000000 [ 50.082457] [ 50.082457] GPR00: 000000a2 bf980040 1024b4d0 bf980084 bf980084 64000000 00555345 fefefeff [ 50.082457] GPR08: 7f7f7f7f 101e0000 00000069 00000003 28000422 102490c2 bff4180c 101e60d4 [ 50.082457] GPR16: 00000000 102454ac 00000040 10240000 10241100 102410f8 10240000 00500000 [ 50.082457] GPR24: 00000002 bf9803f4 10240000 00000000 00000000 100039e0 00000000 102444e8 [ 50.083789] NIP [100a4d08] 0x100a4d08 [ 50.083917] LR [101b5234] 0x101b5234 [ 50.084042] --- interrupt: c00 [ 50.084238] Instruction dump: [ 50.084483] 4bfffc40 60000000 60000000 60000000 9421fff0 39400402 914200c0 38210010 [ 50.084841] 4bfffc20 00000000 00000000 00000000 <7fe00008> 7c0802a6 7c892378 93c10048 [ 50.085487] ---[ end trace f6fffe98e2fa8f3e ]--- [ 50.085678] Trace/breakpoint trap There is no real mode for booke arch and the MMU translation is always on. The corresponding MSR_IS/MSR_DS bit in booke is used to switch the address space, but not for real mode judgment. Fixes: 21f8b2f ("powerpc/kprobes: Ignore traps that happened in real mode") Signed-off-by: Pu Lehui <pulehui@huawei.com> Signed-off-by: Michael Ellerman <mpe@ellerman.id.au> Link: https://lore.kernel.org/r/20210809023658.218915-1-pulehui@huawei.com Signed-off-by: Sasha Levin <sashal@kernel.org>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
你好，我使用RK3229 echo板。
启动的时候到Starting kernel ...就挂了。下面是具体的log:
这是我的extlinux.conf
请帮忙看看这个问题，谢谢
The text was updated successfully, but these errors were encountered: