Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

task txg_sync:1328 blocked for more than 120 seconds. #4011

Closed
ioquatix opened this issue Nov 14, 2015 · 1 comment
Closed

task txg_sync:1328 blocked for more than 120 seconds. #4011

ioquatix opened this issue Nov 14, 2015 · 1 comment

Comments

@ioquatix
Copy link

Running on a system with 8GB of memory. The same system and similar problem as discussed in #2934

% pacman -Q | egrep "zfs|spl"
spl-git 0.6.5.3_r0_g7e85f6b_4.2.5_1-1
spl-utils-git 0.6.5.3_r0_g7e85f6b_4.2.5_1-1
zfs-git 0.6.5.3_r0_g9aaf60b_4.2.5_1-1
zfs-utils-git 0.6.5.3_r0_g9aaf60b_4.2.5_1-1
[293623.087748] INFO: task txg_sync:1328 blocked for more than 120 seconds.
[293623.087866]       Tainted: P           O    4.2.5-1-ARCH #1
[293623.087918] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[293623.087990] txg_sync        D 0000000000000000     0  1328      2 0x00000000
[293623.087997]  ffff8802200cba18 0000000000000046 ffff880090f18000 ffff8800cd793e80
[293623.088002]  ffff8802200cba48 ffff8802200cc000 ffff88022fc15200 7fffffffffffffff
[293623.088005]  ffff8800ccdfec50 0000000000000001 ffff8802200cba38 ffffffff8157283e
[293623.088009] Call Trace:
[293623.088021]  [<ffffffff8157283e>] schedule+0x3e/0x90
[293623.088026]  [<ffffffff81575205>] schedule_timeout+0x1b5/0x240
[293623.088033]  [<ffffffff810b4592>] ? __wake_up_common+0x52/0x90
[293623.088136]  [<ffffffff8101e769>] ? read_tsc+0x9/0x10
[293623.088140]  [<ffffffff81571d94>] io_schedule_timeout+0xa4/0x110
[293623.088174]  [<ffffffffa032fb60>] ? zio_taskq_member.isra.2+0x90/0x90 [zfs]
[293623.088182]  [<ffffffffa01ded28>] cv_wait_common+0xb8/0x140 [spl]
[293623.088186]  [<ffffffff810b4c80>] ? wake_atomic_t_function+0x60/0x60
[293623.088192]  [<ffffffffa01dee08>] __cv_wait_io+0x18/0x20 [spl]
[293623.088211]  [<ffffffffa0331d2e>] zio_wait+0x10e/0x1f0 [zfs]
[293623.088236]  [<ffffffffa02bb468>] dsl_pool_sync+0xb8/0x430 [zfs]
[293623.088262]  [<ffffffffa02d6fa9>] spa_sync+0x369/0xb30 [zfs]
[293623.088267]  [<ffffffff810b4c96>] ? autoremove_wake_function+0x16/0x40
[293623.088292]  [<ffffffffa02e85ae>] txg_sync_thread+0x3be/0x630 [zfs]
[293623.088374]  [<ffffffff810a9071>] ? put_prev_entity+0x31/0x420
[293623.088399]  [<ffffffffa02e81f0>] ? txg_delay+0x180/0x180 [zfs]
[293623.088405]  [<ffffffffa01d9ed1>] thread_generic_wrapper+0x71/0x80 [spl]
[293623.088411]  [<ffffffffa01d9e60>] ? __thread_exit+0x20/0x20 [spl]
[293623.088416]  [<ffffffff81092578>] kthread+0xd8/0xf0
[293623.088421]  [<ffffffff810924a0>] ? kthread_worker_fn+0x170/0x170
[293623.088424]  [<ffffffff8157665f>] ret_from_fork+0x3f/0x70
[293623.088428]  [<ffffffff810924a0>] ? kthread_worker_fn+0x170/0x170
[642705.138572] INFO: task txg_sync:1328 blocked for more than 120 seconds.
[642705.138684]       Tainted: P           O    4.2.5-1-ARCH #1
[642705.138736] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[642705.138808] txg_sync        D 0000000000000000     0  1328      2 0x00000000
[642705.138815]  ffff8802200cba18 0000000000000046 ffff880220894b00 ffff8800cd793e80
[642705.138820]  0000000000000000 ffff8802200cc000 ffff88022fc15200 7fffffffffffffff
[642705.138824]  ffff8800ccdfaf10 0000000000000001 ffff8802200cba38 ffffffff8157283e
[642705.138827] Call Trace:
[642705.138839]  [<ffffffff8157283e>] schedule+0x3e/0x90
[642705.138844]  [<ffffffff81575205>] schedule_timeout+0x1b5/0x240
[642705.138879]  [<ffffffffa032fb60>] ? zio_taskq_member.isra.2+0x90/0x90 [zfs]
[642705.138884]  [<ffffffff81572ae2>] ? preempt_schedule_common+0x22/0x40
[642705.138888]  [<ffffffff81572b24>] ? preempt_schedule+0x24/0x30
[642705.138893]  [<ffffffff81003012>] ? ___preempt_schedule+0x12/0x14
[642705.138898]  [<ffffffff8101e769>] ? read_tsc+0x9/0x10
[642705.138902]  [<ffffffff81571d94>] io_schedule_timeout+0xa4/0x110
[642705.138922]  [<ffffffffa032fb60>] ? zio_taskq_member.isra.2+0x90/0x90 [zfs]
[642705.138931]  [<ffffffffa01ded28>] cv_wait_common+0xb8/0x140 [spl]
[642705.138935]  [<ffffffff810b4c80>] ? wake_atomic_t_function+0x60/0x60
[642705.138941]  [<ffffffffa01dee08>] __cv_wait_io+0x18/0x20 [spl]
[642705.138960]  [<ffffffffa0331d2e>] zio_wait+0x10e/0x1f0 [zfs]
[642705.138985]  [<ffffffffa02bb676>] dsl_pool_sync+0x2c6/0x430 [zfs]
[642705.139011]  [<ffffffffa02d6fa9>] spa_sync+0x369/0xb30 [zfs]
[642705.139015]  [<ffffffff810b4c96>] ? autoremove_wake_function+0x16/0x40
[642705.139040]  [<ffffffffa02e85ae>] txg_sync_thread+0x3be/0x630 [zfs]
[642705.139046]  [<ffffffff810a9071>] ? put_prev_entity+0x31/0x420
[642705.139071]  [<ffffffffa02e81f0>] ? txg_delay+0x180/0x180 [zfs]
[642705.139077]  [<ffffffffa01d9ed1>] thread_generic_wrapper+0x71/0x80 [spl]
[642705.139083]  [<ffffffffa01d9e60>] ? __thread_exit+0x20/0x20 [spl]
[642705.139087]  [<ffffffff81092578>] kthread+0xd8/0xf0
[642705.139092]  [<ffffffff810924a0>] ? kthread_worker_fn+0x170/0x170
[642705.139096]  [<ffffffff8157665f>] ret_from_fork+0x3f/0x70
[642705.139100]  [<ffffffff810924a0>] ? kthread_worker_fn+0x170/0x170
[847977.118177] INFO: task jbd2/sde2-8:135 blocked for more than 120 seconds.
[847977.118290]       Tainted: P           O    4.2.5-1-ARCH #1
[847977.118342] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[847977.118414] jbd2/sde2-8     D 0000000000000001     0   135      2 0x00000000
[847977.118422]  ffff8800cfaafc78 0000000000000046 ffff8801ae14b200 ffff8802251fcb00
[847977.118426]  0000000000000001 ffff8800cfab0000 ffff8800cfaafd70 ffff8800cfaafd58
[847977.118430]  ffff8802251fcb00 ffff880104d69600 ffff8800cfaafc98 ffffffff8157283e
[847977.118434] Call Trace:
[847977.118446]  [<ffffffff8157283e>] schedule+0x3e/0x90
[847977.118456]  [<ffffffffa012d4e1>] jbd2_journal_commit_transaction+0x231/0x1890 [jbd2]
[847977.118463]  [<ffffffff810a8239>] ? dequeue_entity+0x149/0x690
[847977.118468]  [<ffffffff810a522b>] ? set_next_entity+0xab/0x440
[847977.118473]  [<ffffffff810b4c80>] ? wake_atomic_t_function+0x60/0x60
[847977.118478]  [<ffffffff810dca8e>] ? try_to_del_timer_sync+0x5e/0x90
[847977.118484]  [<ffffffffa0133b4a>] kjournald2+0xca/0x250 [jbd2]
[847977.118488]  [<ffffffff810b4c80>] ? wake_atomic_t_function+0x60/0x60
[847977.118493]  [<ffffffffa0133a80>] ? commit_timeout+0x10/0x10 [jbd2]
[847977.118498]  [<ffffffff81092578>] kthread+0xd8/0xf0
[847977.118503]  [<ffffffff810924a0>] ? kthread_worker_fn+0x170/0x170
[847977.118507]  [<ffffffff8157665f>] ret_from_fork+0x3f/0x70
[847977.118511]  [<ffffffff810924a0>] ? kthread_worker_fn+0x170/0x170
[847977.118515] INFO: task systemd-journal:161 blocked for more than 120 seconds.
[847977.118625]       Tainted: P           O    4.2.5-1-ARCH #1
[847977.118676] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[847977.118746] systemd-journal D ffff88022fc95200     0   161      1 0x00000000
[847977.118750]  ffff88022517b7c8 0000000000000086 ffff8802261f8c80 ffff8800cf85f080
[847977.118754]  ffff88022517b7c8 ffff88022517c000 ffff88022fc95200 7fffffffffffffff
[847977.118758]  0000000000000002 ffffffff81573110 ffff88022517b7e8 ffffffff8157283e
[847977.118761] Call Trace:
[847977.118765]  [<ffffffff81573110>] ? bit_wait+0x50/0x50
[847977.118769]  [<ffffffff8157283e>] schedule+0x3e/0x90
[847977.118772]  [<ffffffff81575205>] schedule_timeout+0x1b5/0x240
[847977.118778]  [<ffffffff81245808>] ? kernfs_path+0x48/0x60
[847977.118783]  [<ffffffff812aaebb>] ? cfq_del_cfqq_rr+0x10b/0x280
[847977.118787]  [<ffffffff8101e769>] ? read_tsc+0x9/0x10
[847977.118791]  [<ffffffff81573110>] ? bit_wait+0x50/0x50
[847977.118795]  [<ffffffff81571d94>] io_schedule_timeout+0xa4/0x110
[847977.118799]  [<ffffffff810b4744>] ? prepare_to_wait_exclusive+0x54/0x80
[847977.118802]  [<ffffffff81573145>] bit_wait_io+0x35/0x50
[847977.118806]  [<ffffffff81572f2b>] __wait_on_bit_lock+0x4b/0xa0
[847977.118810]  [<ffffffff812ba5b2>] ? radix_tree_lookup_slot+0x22/0x50
[847977.118814]  [<ffffffff81573110>] ? bit_wait+0x50/0x50
[847977.118818]  [<ffffffff81573002>] out_of_line_wait_on_bit_lock+0x82/0xb0
[847977.118822]  [<ffffffff810b4cc0>] ? autoremove_wake_function+0x40/0x40
[847977.118827]  [<ffffffff81203f67>] __lock_buffer+0x27/0x30
[847977.118833]  [<ffffffffa012bb6c>] do_get_write_access+0x1ec/0x510 [jbd2]
[847977.118836]  [<ffffffff81205929>] ? __find_get_block+0x109/0x110
[847977.118841]  [<ffffffffa012bec7>] jbd2_journal_get_write_access+0x37/0x70 [jbd2]
[847977.118857]  [<ffffffffa018c391>] __ext4_journal_get_write_access+0x31/0x70 [ext4]
[847977.118867]  [<ffffffffa015cde5>] ext4_reserve_inode_write+0x75/0xa0 [ext4]
[847977.118876]  [<ffffffffa0160388>] ? ext4_dirty_inode+0x48/0x70 [ext4]
[847977.118885]  [<ffffffffa015ce5e>] ext4_mark_inode_dirty+0x4e/0x220 [ext4]
[847977.118894]  [<ffffffffa0160388>] ext4_dirty_inode+0x48/0x70 [ext4]
[847977.118897]  [<ffffffff811fc648>] __mark_inode_dirty+0x48/0x380
[847977.118902]  [<ffffffff811ea579>] generic_update_time+0x79/0xc0
[847977.118906]  [<ffffffff811ea6e4>] file_update_time+0xb4/0x110
[847977.118915]  [<ffffffffa0160584>] ext4_page_mkwrite+0x64/0x450 [ext4]
[847977.118918]  [<ffffffff810b4592>] ? __wake_up_common+0x52/0x90
[847977.118922]  [<ffffffff810b4cc0>] ? autoremove_wake_function+0x40/0x40
[847977.118926]  [<ffffffff81185376>] do_page_mkwrite+0x56/0xb0
[847977.118930]  [<ffffffff811878a9>] do_wp_page+0x279/0x5e0
[847977.118934]  [<ffffffff8118a59c>] handle_mm_fault+0xc8c/0x18b0
[847977.118938]  [<ffffffff811d5ac1>] ? cp_new_stat+0x121/0x150
[847977.118942]  [<ffffffff81063439>] __do_page_fault+0x189/0x450
[847977.118945]  [<ffffffff81063722>] do_page_fault+0x22/0x30
[847977.118949]  [<ffffffff81578198>] page_fault+0x28/0x30
[847977.119077] INFO: task kworker/u8:2:11862 blocked for more than 120 seconds.
[847977.119180]       Tainted: P           O    4.2.5-1-ARCH #1
[847977.119231] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[847977.119301] kworker/u8:2    D ffff88022fc95200     0 11862      2 0x00000000
[847977.119308] Workqueue: writeback wb_workfn (flush-8:64)
[847977.119311]  ffff880204f937d8 0000000000000046 ffff8802261f8c80 ffff8801ae14b200
[847977.119315]  ffff8802251fcb00 ffff880204f94000 0000000000000000 ffff8800cf8be870
[847977.119318]  00000000003052ed ffff880104d69600 ffff880204f937f8 ffffffff8157283e
[847977.119321] Call Trace:
[847977.119326]  [<ffffffff8157283e>] schedule+0x3e/0x90
[847977.119331]  [<ffffffffa012a08d>] wait_transaction_locked+0x8d/0xd0 [jbd2]
[847977.119335]  [<ffffffff810b4c80>] ? wake_atomic_t_function+0x60/0x60
[847977.119340]  [<ffffffffa012a4a5>] start_this_handle+0x355/0x5c0 [jbd2]
[847977.119344]  [<ffffffff81575b7c>] ? _raw_write_unlock+0x1c/0x20
[847977.119348]  [<ffffffff811b3baa>] ? kmem_cache_alloc+0x1da/0x200
[847977.119353]  [<ffffffffa012aaac>] jbd2__journal_start+0xdc/0x1f0 [jbd2]
[847977.119364]  [<ffffffffa015d451>] ? ext4_writepages+0x421/0xcd0 [ext4]
[847977.119374]  [<ffffffffa018c0dd>] __ext4_journal_start_sb+0x6d/0x120 [ext4]
[847977.119383]  [<ffffffffa015d451>] ext4_writepages+0x421/0xcd0 [ext4]
[847977.119388]  [<ffffffff810146ec>] ? __switch_to+0x25c/0x4a0
[847977.119392]  [<ffffffff810146ec>] ? __switch_to+0x25c/0x4a0
[847977.119396]  [<ffffffff812c85b5>] ? find_next_bit+0x15/0x20
[847977.119401]  [<ffffffff8116592e>] do_writepages+0x1e/0x30
[847977.119404]  [<ffffffff811fc9c5>] __writeback_single_inode+0x45/0x320
[847977.119408]  [<ffffffff811fd20d>] writeback_sb_inodes+0x29d/0x4e0
[847977.119412]  [<ffffffff811fd4dc>] __writeback_inodes_wb+0x8c/0xc0
[847977.119415]  [<ffffffff811fd744>] wb_writeback+0x234/0x2e0
[847977.119419]  [<ffffffff811fde52>] wb_workfn+0x132/0x510
[847977.119424]  [<ffffffff81099d02>] ? finish_task_switch+0x62/0x1b0
[847977.119428]  [<ffffffff8108c5db>] process_one_work+0x14b/0x440
[847977.119431]  [<ffffffff8108c918>] worker_thread+0x48/0x4a0
[847977.119435]  [<ffffffff8108c8d0>] ? process_one_work+0x440/0x440
[847977.119439]  [<ffffffff81092578>] kthread+0xd8/0xf0
[847977.119443]  [<ffffffff810924a0>] ? kthread_worker_fn+0x170/0x170
[847977.119447]  [<ffffffff8157665f>] ret_from_fork+0x3f/0x70
[847977.119451]  [<ffffffff810924a0>] ? kthread_worker_fn+0x170/0x170
@ioquatix
Copy link
Author

I'm not sure if this issue is still relevant or not so feel free to close it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants
@ioquatix @behlendorf and others