Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cpu hotplug-unplug in parallel with suspend resume.. guest becomes unresponsive.. #16

Closed
bssrikanth opened this issue Sep 27, 2017 · 3 comments

Comments

@bssrikanth
Copy link

bssrikanth commented Sep 27, 2017

I tried cpu hotplug-unplug in parallel with suspend resume.. guest becomes unresponsive..
i. On host run
terminal 1:
for i in {1..100};do sleep 2;virsh suspend srikanth_1710_Cdrom;sleep 2;virsh resume srikanth_1710_Cdrom;done
terminal 2:
for i in {1..100};do sleep 5;virsh setvcpus srikanth_1710_Cdrom 4 --live;sleep 5;virsh setvcpus srikanth_1710_Cdrom 2 --live;done

ii. After/while both above commands complete, will start seeing below messages on guest console:

[ 1209.368538] INFO: task jbd2/dm-0-8:1088 blocked for more than 120 seconds.
[ 1209.368633] Not tainted 4.12.0-11-generic #12-Ubuntu
[ 1209.368694] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

And the guest goes unresponsive..

I could'nt run sosreport on guest since it was unresponsive..

I tested same scenario as mentioned in previous comment with latest hostos guest.. I am hitting similar stall issues there as well...
I am attaching the traces of hostos here..
Guest is hung.. so could not get sosreports...

So these stall issues present on Ubuntu 17.10 [kernel: 4.13.0-11] and HostOS guest [kernel: 4.13.0-4.rel ]

In an internal comment I will post machine access details and guest details..

cde:info Mirrored with LTC bug #159338 </cde:info>

@bssrikanth
Copy link
Author

Stall_hostosguest.txt

@bssrikanth
Copy link
Author

@sam has fix for this issue, patch @ http://www.spinics.net/lists/kvm-ppc/msg12981.html ..
This bug is track verification of the issue when HostOS gets the above patch

@cdeadmin
Copy link

cdeadmin commented Dec 4, 2017

------- Comment From bssrikanth@in.ibm.com 2017-12-04 04:54:15 EDT-------
Tested on 4.14 rel branch.. issue seems to be fixed from the tests conducted. Closing.

@cdeadmin cdeadmin closed this as completed Dec 4, 2017
malcolmcrossley pushed a commit to NXPower/linux that referenced this issue Mar 2, 2018
commit efdab99 upstream.

syzkaller reported:

   WARNING: CPU: 0 PID: 12927 at arch/x86/kernel/traps.c:780 do_debug+0x222/0x250
   CPU: 0 PID: 12927 Comm: syz-executor Tainted: G           OE    4.15.0-rc2+ open-power-host-os#16
   RIP: 0010:do_debug+0x222/0x250
   Call Trace:
    <#DB>
    debug+0x3e/0x70
   RIP: 0010:copy_user_enhanced_fast_string+0x10/0x20
    </#DB>
    _copy_from_user+0x5b/0x90
    SyS_timer_create+0x33/0x80
    entry_SYSCALL_64_fastpath+0x23/0x9a

The testcase sets a watchpoint (with perf_event_open) on a buffer that is
passed to timer_create() as the struct sigevent argument.  In timer_create(),
copy_from_user()'s rep movsb triggers the BP.  The testcase also sets
the debug registers for the guest.

However, KVM only restores host debug registers when the host has active
watchpoints, which triggers a race condition when running the testcase with
multiple threads.  The guest's DR6.BS bit can escape to the host before
another thread invokes timer_create(), and do_debug() complains.

The fix is to respect do_debug()'s dr6 invariant when leaving KVM.

Reported-by: Dmitry Vyukov <dvyukov@google.com>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Radim Krčmář <rkrcmar@redhat.com>
Cc: David Hildenbrand <david@redhat.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Reviewed-by: David Hildenbrand <david@redhat.com>
Signed-off-by: Wanpeng Li <wanpeng.li@hotmail.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Radim Krčmář <rkrcmar@redhat.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
malcolmcrossley pushed a commit to NXPower/linux that referenced this issue Mar 2, 2018
[ Upstream commit 5811767 ]

According to LS1021A RM, the value of PAL can be set so that the start of the
IP header in the receive data buffer is aligned to a 32-bit boundary. Normally,
setting PAL = 2 provides minimal padding to ensure such alignment of the IP
header.

However every incoming packet's 8-byte time stamp will be inserted into the
packet data buffer as padding alignment bytes when hardware time stamping is
enabled.

So we set the padding 8+2 here to avoid the flooded alignment faults:

root@128:~# cat /proc/cpu/alignment
User:           0
System:         17539 (inet_gro_receive+0x114/0x2c0)
Skipped:        0
Half:           0
Word:           0
DWord:          0
Multi:          17539
User faults:    2 (fixup)

Also shown when exception report enablement

CPU: 0 PID: 161 Comm: irq/66-eth1_g0_ Not tainted 4.1.21-rt13-WR8.0.0.0_preempt-rt open-power-host-os#16
Hardware name: Freescale LS1021A
[<8001b420>] (unwind_backtrace) from [<8001476c>] (show_stack+0x20/0x24)
[<8001476c>] (show_stack) from [<807cfb48>] (dump_stack+0x94/0xac)
[<807cfb48>] (dump_stack) from [<80025d70>] (do_alignment+0x720/0x958)
[<80025d70>] (do_alignment) from [<80009224>] (do_DataAbort+0x40/0xbc)
[<80009224>] (do_DataAbort) from [<80015398>] (__dabt_svc+0x38/0x60)
Exception stack(0x86ad1cc0 to 0x86ad1d08)
1cc0: f9b3e080 86b3d072 2d78d287 00000000 866816c0 86b3d05e 86e785d0 00000000
1ce0: 00000011 0000000e 80840ab0 86ad1d3c 86ad1d08 86ad1d08 806d7fc0 806d806c
1d00: 40070013 ffffffff
[<80015398>] (__dabt_svc) from [<806d806c>] (inet_gro_receive+0x114/0x2c0)
[<806d806c>] (inet_gro_receive) from [<80660eec>] (dev_gro_receive+0x21c/0x3c0)
[<80660eec>] (dev_gro_receive) from [<8066133c>] (napi_gro_receive+0x44/0x17c)
[<8066133c>] (napi_gro_receive) from [<804f0538>] (gfar_clean_rx_ring+0x39c/0x7d4)
[<804f0538>] (gfar_clean_rx_ring) from [<804f0bf4>] (gfar_poll_rx_sq+0x58/0xe0)
[<804f0bf4>] (gfar_poll_rx_sq) from [<80660b10>] (net_rx_action+0x27c/0x43c)
[<80660b10>] (net_rx_action) from [<80033638>] (do_current_softirqs+0x1e0/0x3dc)
[<80033638>] (do_current_softirqs) from [<800338c4>] (__local_bh_enable+0x90/0xa8)
[<800338c4>] (__local_bh_enable) from [<8008025c>] (irq_forced_thread_fn+0x70/0x84)
[<8008025c>] (irq_forced_thread_fn) from [<800805e8>] (irq_thread+0x16c/0x244)
[<800805e8>] (irq_thread) from [<8004e490>] (kthread+0xe8/0x104)
[<8004e490>] (kthread) from [<8000fda8>] (ret_from_fork+0x14/0x2c)

Signed-off-by: Zumeng Chen <zumeng.chen@gmail.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
Signed-off-by: Sasha Levin <alexander.levin@microsoft.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants