New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kernel Panics on Dell PowerEdge R730xd #2492

Open
obion opened this Issue Aug 13, 2018 · 2 comments

Comments

Projects
None yet
3 participants
@obion

obion commented Aug 13, 2018

Issue Report

Hi everyone!
We've just migrated to a bare metal environment and after day or two in production our servers started to kernel panic. Currently happened 2 times on 2 different servers within 2 days.
Yesterday's crash we suspected our application caused that, as in dmesg were lots of traps for backend process. We fixed that, those trap errors disappeared, but unfortunately today we also had a kernel panic on another server. That time there visible IPVS errors, but not sure if these are related.
Please help finding the root cause of that.
Thank you!

Bug

Container Linux Version

1745.7.0

core@k8s-worker-002 ~ $ cat /etc/os-release
NAME="Container Linux by CoreOS"
ID=coreos
VERSION=1745.7.0
VERSION_ID=1745.7.0
BUILD_ID=2018-06-14-0909
PRETTY_NAME="Container Linux by CoreOS 1745.7.0 (Rhyolite)"
ANSI_COLOR="38;5;75"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://issues.coreos.com"
COREOS_BOARD="amd64-usr"

Environment

Dell Inc. PowerEdge R730xd/072T6D, BIOS 2.8.0 005/17/2018
Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz / 48 core
Bare Metal
4x NIC Bonding LACP mode
20 network interrupts distributed among odd core numbers
Kubernetes v1.11 with IPVS as a service balancer
Calico v2.6.8 without IP-to-IP
~90 pods per host

Expected Behavior

No kernel panics

Actual Behavior

Second kernel panic during 48h on different physical servers after we put on them load.

Reproduction Steps

Put big load. Wait.

Other Information

From today's server dmesg from pstore

Panic#1 Part1
<3>[523530.453193] IPVS: rr: UDP 10.233.60.50:9888 - no destination available
<3>[523530.453371] IPVS: rr: UDP 10.233.60.50:9888 - no destination available
<3>[523530.453392] IPVS: rr: UDP 10.233.60.50:9888 - no destination available
<3>[523530.453412] IPVS: rr: UDP 10.233.60.50:9888 - no destination available
<3>[523530.454352] IPVS: rr: UDP 10.233.60.50:9888 - no destination available
<4>[523535.452412] net_ratelimit: 5147 callbacks suppressed
<3>[523535.452414] IPVS: rr: UDP 10.233.60.50:9888 - no destination available
<3>[523535.454119] IPVS: rr: UDP 10.233.60.50:9888 - no destination available
<3>[523535.466678] IPVS: rr: UDP 10.233.60.50:9888 - no destination available
<3>[523535.471096] IPVS: rr: UDP 10.233.60.50:9888 - no destination available
<3>[523535.471916] IPVS: rr: UDP 10.233.60.50:9888 - no destination available
<3>[523535.474078] IPVS: rr: UDP 10.233.60.50:9888 - no destination available
<3>[523535.484189] IPVS: rr: UDP 10.233.60.50:9888 - no destination available
<3>[523535.486912] IPVS: rr: UDP 10.233.60.50:9888 - no destination available
<3>[523535.487081] IPVS: rr: UDP 10.233.60.50:9888 - no destination available
<3>[523535.491054] IPVS: rr: UDP 10.233.60.50:9888 - no destination available
<4>[523540.452831] net_ratelimit: 5516 callbacks suppressed
<3>[523540.452834] IPVS: rr: UDP 10.233.60.50:9888 - no destination available
<3>[523540.456450] IPVS: rr: UDP 10.233.60.50:9888 - no destination available
<3>[523540.457445] IPVS: rr: UDP 10.233.60.50:9888 - no destination available
<3>[523540.457498] IPVS: rr: UDP 10.233.60.50:9888 - no destination available
<3>[523540.457926] IPVS: rr: UDP 10.233.60.50:9888 - no destination available
<3>[523540.458148] IPVS: rr: UDP 10.233.60.50:9888 - no destination available
<3>[523540.459454] IPVS: rr: UDP 10.233.60.50:9888 - no destination available
<3>[523540.460032] IPVS: rr: UDP 10.233.60.50:9888 - no destination available
<3>[523540.460109] IPVS: rr: UDP 10.233.60.50:9888 - no destination available
<3>[523540.460125] IPVS: rr: UDP 10.233.60.50:9888 - no destination available
<4>[523545.454370] net_ratelimit: 4889 callbacks suppressed
<3>[523545.454373] IPVS: rr: UDP 10.233.60.50:9888 - no destination available
<3>[523545.454580] IPVS: rr: UDP 10.233.60.50:9888 - no destination available
<3>[523545.455180] IPVS: rr: UDP 10.233.60.50:9888 - no destination available
<3>[523545.456797] IPVS: rr: UDP 10.233.60.50:9888 - no destination available
<3>[523545.460364] IPVS: rr: UDP 10.233.60.50:9888 - no destination available
<3>[523545.463375] IPVS: rr: UDP 10.233.60.50:9888 - no destination available
<3>[523545.466362] IPVS: rr: UDP 10.233.60.50:9888 - no destination available
<3>[523545.467139] IPVS: rr: UDP 10.233.60.50:9888 - no destination available
<3>[523545.467930] IPVS: rr: UDP 10.233.60.50:9888 - no destination available
<3>[523545.467945] IPVS: rr: UDP 10.233.60.50:9888 - no destination available
<4>[523550.455076] net_ratelimit: 4303 callbacks suppressed
<3>[523550.455079] IPVS: rr: UDP 10.233.60.50:9888 - no destination available
<3>[523550.455249] IPVS: rr: UDP 10.233.60.50:9888 - no destination available
<3>[523550.457817] IPVS: rr: UDP 10.233.60.50:9888 - no destination available
<3>[523550.460828] IPVS: rr: UDP 10.233.60.50:9888 - no destination available
<3>[523550.462873] IPVS: rr: UDP 10.233.60.50:9888 - no destination available
<3>[523550.464045] IPVS: rr: UDP 10.233.60.50:9888 - no destination available
<3>[523550.465138] IPVS: rr: UDP 10.233.60.50:9888 - no destination available
<3>[523550.466258] IPVS: rr: UDP 10.233.60.50:9888 - no destination available
<3>[523550.466319] IPVS: rr: UDP 10.233.60.50:9888 - no destination available
<3>[523550.467085] IPVS: rr: UDP 10.233.60.50:9888 - no destination available
<6>[532218.652773] IPv6: ADDRCONF(NETDEV_UP): cali32d2f49420e: link is not ready
<6>[532218.660583] IPv6: ADDRCONF(NETDEV_CHANGE): cali32d2f49420e: link becomes ready
<6>[532242.263821] IPv6: ADDRCONF(NETDEV_UP): cali9b7c770abfa: link is not ready
<6>[532242.272067] IPv6: ADDRCONF(NETDEV_CHANGE): cali9b7c770abfa: link becomes ready
<4>[533593.611284] ------------[ cut here ]------------
<4>[533593.616635] WARNING: CPU: 41 PID: 40201 at ../source/include/net/dst.h:256 nf_xfrm_me_harder+0x11e/0x130 [nf_nat]
<4>[533593.628287] Modules linked in: nfnetlink_queue nfnetlink_log nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 nfs lockd grace sunrpc fscache xt_multiport iptable_mangle iptable_raw ip_set_hash_net veth xt_set ip_set_hash_ipportnet ip_set_bitmap_port ip_set_hash_ipportip ip_set_hash_ipport ip_set dummy xt_addrtype ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6_tables ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_comment xt_mark iptable_nat nf_nat_ipv4 iptable_filter xt_conntrack nf_nat nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo overlay nls_ascii nls_cp437 vfat fat xfs sb_edac edac_core coretemp x86_pkg_temp_thermal ipmi_ssif i2c_core kvm_intel kvm evdev irqbypass dcdbas ipmi_si mei_me ipmi_devintf mei ipmi_msghandler button sch_fq_codel br_netfilter bridge stp llc nf_conntrack_ipv4
<4>[533593.708353]  nf_defrag_ipv4 ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs nf_conntrack libcrc32c bonding ext4 crc32c_generic crc16 mbcache jbd2 fscrypto dm_verity dm_bufio sd_mod crc32c_intel aesni_intel ehci_pci aes_x86_64 ahci tg3 crypto_simd libahci cryptd hwmon ehci_hcd glue_helper megaraid_sas ptp usbcore libata pps_core usb_common scsi_mod libphy dm_mirror dm_region_hash dm_log dm_mod dax
<4>[533593.748514] CPU: 41 PID: 40201 Comm: java Not tainted 4.14.48-coreos-r2 #1
<4>[533593.756789] Hardware name: Dell Inc. PowerEdge R730xd/072T6D, BIOS 2.8.0 005/17/2018
<4>[533593.766447] task: ffff9223488a3c00 task.stack: ffffa1bcec0ec000
<4>[533593.773656] RIP: 0010:nf_xfrm_me_harder+0x11e/0x130 [nf_nat]
<4>[533593.780563] RSP: 0018:ffff923f3f503c50 EFLAGS: 00010246
<4>[533593.786987] RAX: 0000000000000000 RBX: ffffffff9e2d74c0 RCX: 000000000000ecd0
<4>[533593.795962] RDX: 0000000000000001 RSI: ffff923d5c0bda00 RDI: 0000000000000000
<4>[533593.805020] RBP: ffff92373f931ee8 R08: 0000000000000000 R09: 0000000000000018
<4>[533593.814000] R10: 0000000000000001 R11: 00000000e358b3a2 R12: ffff923f3f503d20
<4>[533593.822977] R13: ffff923f3615de00 R14: ffff922e665d7078 R15: 0000000000000008
<4>[533593.831956] FS:  00007fd88e3e3700(0000) GS:ffff923f3f500000(0000) knlGS:0000000000000000
<4>[533593.842000] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
<4>[533593.849007] CR2: 00007fd57002a000 CR3: 0000001fd9acc004 CR4: 00000000003606e0
<4>[533593.857978] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
<4>[533593.866957] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
<4>[533593.876012] Call Trace:
<4>[533593.879330]  <IRQ>
<4>[533593.882165]  nf_nat_ipv4_out+0xbc/0xd0 [nf_nat_ipv4]
<4>[533593.888299]  nf_hook_slow+0x43/0xc0
<4>[533593.892786]  ip_output+0xd2/0xe0
<4>[533593.896977]  ? ip_fragment.constprop.45+0x80/0x80
<4>[533593.902815]  ip_forward+0x35f/0x450
<4>[533593.907293]  ? ip_frag_mem+0x10/0x10
<4>[533593.911873]  ip_rcv+0x287/0x3a0
<4>[533593.915965]  ? inet_del_offload+0x40/0x40
<4>[533593.921031]  __netif_receive_skb_core+0x432/0xb50
<4>[533593.926874]  ? process_backlog+0x97/0x150
<4>[533593.931936]  process_backlog+0x97/0x150
<4>[533593.936807]  net_rx_action+0x149/0x3d0
<4>[533593.941580]  __do_softirq+0xe7/0x2ca
<4>[533593.946158]  do_softirq_own_stack+0x2a/0x40
<4>[533593.951416]  </IRQ>
<4>[533593.954348]  do_softirq.part.14+0x49/0x50
<4>[533593.959408]  __local_bh_enable_ip+0x55/0x60
<4>[533593.964664]  ip_finish_output2+0x189/0x3c0
<4>[533593.969825]  ? ip_output+0x6c/0xe0
<4>[533593.974207]  ip_output+0x6c/0xe0
<4>[533593.978397]  ? ip_fragment.constprop.45+0x80/0x80
<4>[533593.984240]  tcp_transmit_skb+0x516/0x9b0
<4>[533593.989298]  tcp_write_xmit+0x1a8/0xec0
<4>[533593.994171]  ? _copy_from_iter_full+0x9c/0x240
<4>[533593.999712]  __tcp_push_pending_frames+0x31/0xd0
<4>[533594.005456]  tcp_sendmsg_locked+0xb06/0xe60
<4>[533594.010715]  tcp_sendmsg+0x27/0x40
<4>[533594.015095]  sock_sendmsg+0x36/0x40
<4>[533594.019573]  sock_write_iter+0x8f/0xf0
<4>[533594.024348]  __vfs_write+0x101/0x160
<4>[533594.028924]  vfs_write+0xad/0x1a0
<4>[533594.033209]  SyS_write+0x52/0xc0
<4>[533594.037402]  do_syscall_64+0x67/0x120
<4>[533594.042079]  entry_SYSCALL_64_after_hwframe+0x3d/0xa2
<4>[533594.048309] RIP: 0033:0x7fda8cda69ff
<4>[533594.052884] RSP: 002b:00007fd88e3e2210 EFLAGS: 00000293 ORIG_RAX: 0000000000000001
<4>[533594.062335] RAX: ffffffffffffffda RBX: 000000000000011f RCX: 00007fda8cda69ff
<4>[533594.071302] RDX: 0000000000000ef7 RSI: 00007fd770003310 RDI: 000000000000011f
<4>[533594.080270] RBP: 00007fd770003310 R08: 0000000000000000 R09: 00000000d1a6a078
<4>[533594.089328] R10: 00000000000056ec R11: 0000000000000293 R12: 0000000000000ef7
<4>[533594.098305] R13: 0000000000000ef7 R14: 00007fd88e3e22a0 R15: 00007fd9500aa000
<4>[533594.107286] Code: ff ff ff eb cd 48 83 e7 fe 48 89 04 24 e8 9b 50 f8 dc 48 8b 04 24 eb 94 85 c0 74 0f 8d 50 01 f0 0f b1 11 0f 84 58 ff ff ff eb ed <0f> 0b e9 4f ff ff ff e8 e6 36 b0 dc 66 0f 1f 44 00 00 0f 1f 44
<4>[533594.129826] ---[ end trace 1e3cb5257592ce51 ]---
<4>[533594.135647] net_ratelimit: 2114 callbacks suppressed
<4>[533594.135649] dst_release: dst:ffff923d5c0bda00 refcnt:-1
<6>[533595.118929] IPv6: ADDRCONF(NETDEV_UP): calib68ab8be17e: link is not ready
<6>[533595.128547] IPv6: ADDRCONF(NETDEV_CHANGE): calib68ab8be17e: link becomes ready
<6>[534197.214522] IPv6: ADDRCONF(NETDEV_UP): calid610df92c4a: link is not ready
<6>[534197.222653] IPv6: ADDRCONF(NETDEV_CHANGE): calid610df92c4a: link becomes ready
<6>[534279.204579] IPv6: ADDRCONF(NETDEV_UP): cali1e3823c0139: link is not ready
<6>[534279.212933] IPv6: ADDRCONF(NETDEV_CHANGE): cali1e3823c0139: link becomes ready
<6>[536753.607372] IPv6: ADDRCONF(NETDEV_UP): calibef31f30569: link is not ready
<6>[536753.615429] IPv6: ADDRCONF(NETDEV_CHANGE): calibef31f30569: link becomes ready
<6>[536757.162800] IPv6: ADDRCONF(NETDEV_UP): cali82fea6f8feb: link is not ready
<6>[536757.170740] IPv6: ADDRCONF(NETDEV_CHANGE): cali82fea6f8feb: link becomes ready
<6>[536790.105460] IPv6: ADDRCONF(NETDEV_UP): cali59290af2ca8: link is not ready
<6>[536790.113366] IPv6: ADDRCONF(NETDEV_CHANGE): cali59290af2ca8: link becomes ready
<6>[536800.533350] IPv6: ADDRCONF(NETDEV_UP): caliacb61e6a651: link is not ready
<6>[536800.541171] IPv6: ADDRCONF(NETDEV_CHANGE): caliacb61e6a651: link becomes ready
<6>[536877.390039] IPv6: ADDRCONF(NETDEV_UP): calic7835cba3f4: link is not ready
<6>[536877.398124] IPv6: ADDRCONF(NETDEV_CHANGE): calic7835cba3f4: link becomes ready
<6>[536974.873552] IPv6: ADDRCONF(NETDEV_UP): cali50b4f9a1642: link is not ready
<6>[536974.882143] IPv6: ADDRCONF(NETDEV_CHANGE): cali50b4f9a1642: link becomes ready
<6>[537027.884894] IPv6: ADDRCONF(NETDEV_UP): calidcd77743fe7: link is not ready
<6>[537027.892694] IPv6: ADDRCONF(NETDEV_CHANGE): calidcd77743fe7: link becomes ready
<0>[542318.000341] NMI watchdog: Watchdog detected hard LOCKUP on cpu 0
<4>[542318.000342] Modules linked in: nfnetlink_queue nfnetlink_log nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 nfs lockd grace sunrpc fscache xt_multiport iptable_mangle iptable_raw ip_set_hash_net veth xt_set ip_set_hash_ipportnet ip_set_bitmap_port ip_set_hash_ipportip ip_set_hash_ipport ip_set dummy xt_addrtype ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6_tables ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_comment xt_mark iptable_nat nf_nat_ipv4 iptable_filter xt_conntrack nf_nat nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo overlay nls_ascii nls_cp437 vfat fat xfs sb_edac edac_core coretemp x86_pkg_temp_thermal ipmi_ssif i2c_core kvm_intel kvm evdev irqbypass dcdbas ipmi_si mei_me ipmi_devintf mei ipmi_msghandler button sch_fq_codel br_netfilter bridge stp llc nf_conntrack_ipv4
<4>[542318.000372]  nf_defrag_ipv4 ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs nf_conntrack libcrc32c bonding ext4 crc32c_generic crc16 mbcache jbd2 fscrypto dm_verity dm_bufio sd_mod crc32c_intel aesni_intel ehci_pci aes_x86_64 ahci tg3 crypto_simd libahci cryptd hwmon ehci_hcd glue_helper megaraid_sas ptp usbcore libata pps_core usb_common scsi_mod libphy dm_mirror dm_region_hash dm_log dm_mod dax
<4>[542318.000389] CPU: 0 PID: 31763 Comm: redis-server Tainted: G        W       4.14.48-coreos-r2 #1
<4>[542318.000390] Hardware name: Dell Inc. PowerEdge R730xd/072T6D, BIOS 2.8.0 005/17/2018
<4>[542318.000391] task: ffff922eb9e01e00 task.stack: ffffa1bce897c000
<4>[542318.000396] RIP: 0010:try_to_wake_up+0x99/0x470
<4>[542318.000396] RSP: 0000:ffffa1bce897faa0 EFLAGS: 00000002
<4>[542318.000397] RAX: 0000000000000001 RBX: ffff923f3b233c00 RCX: ffff923f3f45d1b0
<4>[542318.000398] RDX: 0000000000000001 RSI: 0000000000000003 RDI: ffff923f3b234374
<4>[542318.000399] RBP: 0000000000000000 R08: 0000000000000023 R09: 0000aaaaaaaaaaaa
<4>[542318.000399] R10: ffffa1bce897fa60 R11: 0000000000000008 R12: ffff923f3b234374
<4>[542318.000400] R13: 0000000000000000 R14: 0000000000000046 R15: 0000000000000003
<4>[542318.000401] FS:  00007fc1ded29b88(0000) GS:ffff922f3f800000(0000) knlGS:0000000000000000
<4>[542318.000401] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
<4>[542318.000402] CR2: 00007fc039dd5518 CR3: 0000001ef3eb8006 CR4: 00000000003606f0
<4>[542318.000403] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
<4>[542318.000404] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
<4>[542318.000404] Call Trace:
<4>[542318.000412]  stop_two_cpus+0x22e/0x270
<4>[542318.000414]  ? cpu_stopper_thread+0x100/0x100
<4>[542318.000415]  ? cpu_stopper_thread+0x100/0x100
<4>[542318.000417]  ? __migrate_swap_task.part.82+0x80/0x80
<4>[542318.000418]  ? migrate_swap+0x9a/0x120
<4>[542318.000419]  migrate_swap+0x9a/0x120
<4>[542318.000423]  task_numa_migrate+0x447/0x8b0
<4>[542318.000428]  ? tcp_schedule_loss_probe+0x129/0x170
<4>[542318.000430]  task_numa_fault+0xa0e/0xd60
<4>[542318.000433]  ? should_numa_migrate_memory+0x52/0x120
<4>[542318.000435]  ? __handle_mm_fault+0xbb4/0x1180
<4>[542318.000436]  __handle_mm_fault+0xbb4/0x1180
<4>[542318.000437]  handle_mm_fault+0xaa/0x1e0
<4>[542318.000442]  __do_page_fault+0x243/0x4c0
<4>[542318.000445]  ? page_fault+0x2f/0x50
<4>[542318.000447]  page_fault+0x45/0x50
<4>[542318.000449] RIP: 3450b5b9:0x7fc1de6183b0
<4>[542318.000450] RSP: 39dd5518:00007fc006a41a93 EFLAGS: 7fc1de618370
<4>[542318.000450] Code: 5b 5d 41 5c 41 5d 41 5e 41 5f c3 0f 1f 44 00 00 44 8b 43 3c 8b 43 5c 85 c0 0f 85 c1 01 00 00 8b 43 38 85 c0 74 09 f3 90 8b 43 38 <85> c0 75 f7 48 8b 53 10 31 c0 83 e2 02 74 15 f6 43 26 01 75 0f
<0>[542318.000468] Kernel panic - not syncing: Hard LOCKUP
<4>[542318.000469] CPU: 0 PID: 31763 Comm: redis-server Tainted: G        W       4.14.48-coreos-r2 #1
<4>[542318.000470] Hardware name: Dell Inc. PowerEdge R730xd/072T6D, BIOS 2.8.0 005/17/2018
<4>[542318.000470] Call Trace:
<4>[542318.000472]  <NMI>
<4>[542318.000474]  dump_stack+0x5c/0x85
<4>[542318.000476]  panic+0xe4/0x252
<4>[542318.000478]  nmi_panic+0x35/0x40
<4>[542318.000481]  watchdog_overflow_callback+0xe3/0x100
<4>[542318.000484]  __perf_event_overflow+0x52/0xe0
<4>[542318.000487]  intel_pmu_handle_irq+0x25b/0x4e0
<4>[542318.000490]  ? __set_pte_vaddr+0x32/0x50
<4>[542318.000491]  ? __set_pte_vaddr+0x32/0x50
<4>[542318.000493]  ? __native_set_fixmap+0x24/0x30
<4>[542318.000497]  ? ghes_copy_tofrom_phys+0xde/0x260
<4>[542318.000500]  ? perf_event_nmi_handler+0x2d/0x50
<4>[542318.000501]  perf_event_nmi_handler+0x2d/0x50
<4>[542318.000504]  nmi_handle+0x63/0x110
<4>[542318.000505]  default_do_nmi+0x4e/0x100
<4>[542318.000506]  do_nmi+0xe5/0x140
<4>[542318.000508]  end_repeat_nmi+0x16/0x50
<4>[542318.000510] RIP: 0010:try_to_wake_up+0x99/0x470
<4>[542318.000510] RSP: 0000:ffffa1bce897faa0 EFLAGS: 00000002
<4>[542318.000511] RAX: 0000000000000001 RBX: ffff923f3b233c00 RCX: ffff923f3f45d1b0
<4>[542318.000512] RDX: 0000000000000001 RSI: 0000000000000003 RDI: ffff923f3b234374
<4>[542318.000512] RBP: 0000000000000000 R08: 0000000000000023 R09: 0000aaaaaaaaaaaa
<4>[542318.000513] R10: ffffa1bce897fa60 R11: 0000000000000008 R12: ffff923f3b234374
<4>[542318.000513] R13: 0000000000000000 R14: 0000000000000046 R15: 0000000000000003
<4>[542318.000515]  ? try_to_wake_up+0x99/0x470
<4>[542318.000516]  ? try_to_wake_up+0x99/0x470
<4>[542318.000516]  </NMI>
<4>[542318.000518]  stop_two_cpus+0x22e/0x270
<4>[542318.000520]  ? cpu_stopper_thread+0x100/0x100
<4>[542318.000521]  ? cpu_stopper_thread+0x100/0x100
<4>[542318.000522]  ? __migrate_swap_task.part.82+0x80/0x80
<4>[542318.000523]  ? migrate_swap+0x9a/0x120
<4>[542318.000524]  migrate_swap+0x9a/0x120
<4>[542318.000526]  task_numa_migrate+0x447/0x8b0
<4>[542318.000528]  ? tcp_schedule_loss_probe+0x129/0x170
<4>[542318.000530]  task_numa_fault+0xa0e/0xd60
<4>[542318.000532]  ? should_numa_migrate_memory+0x52/0x120
<4>[542318.000533]  ? __handle_mm_fault+0xbb4/0x1180
<4>[542318.000534]  __handle_mm_fault+0xbb4/0x1180
<4>[542318.000535]  handle_mm_fault+0xaa/0x1e0
<4>[542318.000537]  __do_page_fault+0x243/0x4c0
<4>[542318.000539]  ? page_fault+0x2f/0x50
<4>[542318.000540]  page_fault+0x45/0x50
<4>[542318.000541] RIP: 3450b5b9:0x7fc1de6183b0
<4>[542318.000541] RSP: 39dd5518:00007fc006a41a93 EFLAGS: 7fc1de618370
<0>[542319.039804] Shutting down cpus with NMI
<0>[542319.115939] Kernel Offset: 0x1c000000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)

Other 4 dmesg from pstore contain similar info:

Panic#1 Part2
<3>[523425.427985] IPVS: rr: UDP 10.233.60.50:9888 - no destination available
<3>[523425.429559] IPVS: rr: UDP 10.233.60.50:9888 - no destination available
<3>[523425.430642] IPVS: rr: UDP 10.233.60.50:9888 - no destination available
<4>[523430.412996] net_ratelimit: 5122 callbacks suppressed
<3>[523430.412999] IPVS: rr: UDP 10.233.60.50:9888 - no destination available
<3>[523430.414793] IPVS: rr: UDP 10.233.60.50:9888 - no destination available
<3>[523430.430026] IPVS: rr: UDP 10.233.60.50:9888 - no destination available
<3>[523430.430828] IPVS: rr: UDP 10.233.60.50:9888 - no destination available
<3>[523430.431129] IPVS: rr: UDP 10.233.60.50:9888 - no destination available
<3>[523430.433491] IPVS: rr: UDP 10.233.60.50:9888 - no destination available
<3>[523430.433962] IPVS: rr: UDP 10.233.60.50:9888 - no destination available
<3>[523430.434355] IPVS: rr: UDP 10.233.60.50:9888 - no destination available
<3>[523430.435661] IPVS: rr: UDP 10.233.60.50:9888 - no destination available
<3>[523430.436676] IPVS: rr: UDP 10.233.60.50:9888 - no destination available
<4>[523435.414689] net_ratelimit: 5426 callbacks suppressed

screen shot 2018-08-13 at 21 35 17

screen shot 2018-08-13 at 21 42 39

screen shot 2018-08-13 at 21 43 06

That's a dmesg from pstore from yesterday's panic on a different server

Panic#1 Part1
<6>[458841.764663] traps: node[11610] general protection ip:7f3782398532 sp:7fffd9004b60 error:0 in libc-2.19.so[7f3782362000+1a1000]
<6>[458985.756257] traps: node[27359] general protection ip:7f993b138532 sp:7ffe94286ae0 error:0 in libc-2.19.so[7f993b102000+1a1000]
<6>[459007.134595] traps: node[4096] general protection ip:7fa98b7e7532 sp:7ffc768a0220 error:0 in libc-2.19.so[7fa98b7b1000+1a1000]
<6>[459066.680632] traps: node[25009] general protection ip:7f596a7ec532 sp:7ffca417a270 error:0 in libc-2.19.so[7f596a7b6000+1a1000]
<6>[459290.661124] traps: node[48614] general protection ip:7f70113e1532 sp:7ffc3e974060 error:0 in libc-2.19.so[7f70113ab000+1a1000]
<6>[459464.257900] traps: node[25616] general protection ip:7fc5928a8532 sp:7ffd96015790 error:0 in libc-2.19.so[7fc592872000+1a1000]
<6>[459554.872733] traps: node[27871] general protection ip:7f816f824532 sp:7ffdb8659d50 error:0 in libc-2.19.so[7f816f7ee000+1a1000]
<6>[459667.327859] traps: node[26286] general protection ip:7f8e6c83b532 sp:7fffab3cfe30 error:0 in libc-2.19.so[7f8e6c805000+1a1000]
<6>[459742.977368] traps: node[17189] general protection ip:7fa45efbe532 sp:7fffd2a58890 error:0 in libc-2.19.so[7fa45ef88000+1a1000]
<6>[459766.270589] traps: node[48324] general protection ip:7f981e555532 sp:7ffd7134af90 error:0 in libc-2.19.so[7f981e51f000+1a1000]
<6>[459820.318674] traps: node[44958] general protection ip:7fe556be8532 sp:7fffbdd44d10 error:0 in libc-2.19.so[7fe556bb2000+1a1000]
<6>[459925.448652] traps: node[32210] general protection ip:7f0d913cd532 sp:7ffe63f599f0 error:0 in libc-2.19.so[7f0d91397000+1a1000]
<6>[459974.131605] traps: node[21015] general protection ip:7f6305356532 sp:7ffea3e71d20 error:0 in libc-2.19.so[7f6305320000+1a1000]
<6>[460044.847324] traps: node[21473] general protection ip:7fa1d6837532 sp:7fff9019b070 error:0 in libc-2.19.so[7fa1d6801000+1a1000]
<6>[460273.235639] traps: node[41311] general protection ip:7f522e8de532 sp:7fff91a77520 error:0 in libc-2.19.so[7f522e8a8000+1a1000]
<6>[460585.508577] traps: node[40278] general protection ip:7f405d755532 sp:7ffd4d2bc500 error:0 in libc-2.19.so[7f405d71f000+1a1000]
<6>[461065.085785] traps: node[30445] general protection ip:7f7a14089532 sp:7ffe984cb790 error:0 in libc-2.19.so[7f7a14053000+1a1000]
<6>[461103.471954] traps: node[563] general protection ip:7fa187481532 sp:7ffc3dda6260 error:0 in libc-2.19.so[7fa18744b000+1a1000]
<6>[461218.275148] traps: node[13319] general protection ip:7fce505dd532 sp:7ffff65c6130 error:0 in libc-2.19.so[7fce505a7000+1a1000]
<6>[461436.096196] traps: node[39633] general protection ip:7f1ebe353532 sp:7ffede773c60 error:0 in libc-2.19.so[7f1ebe31d000+1a1000]
<6>[461499.894894] traps: node[39164] general protection ip:7fe37e2ab532 sp:7ffe0fde1240 error:0 in libc-2.19.so[7fe37e275000+1a1000]
<6>[461762.618159] traps: node[28250] general protection ip:7f7acfb4f532 sp:7ffd60718e40 error:0 in libc-2.19.so[7f7acfb19000+1a1000]
<6>[462014.102217] traps: node[23625] general protection ip:7f017e312532 sp:7ffe6b8c2870 error:0 in libc-2.19.so[7f017e2dc000+1a1000]
<6>[462562.229323] traps: node[6560] general protection ip:7f82bf987532 sp:7ffd8a7d6690 error:0 in libc-2.19.so[7f82bf951000+1a1000]
<6>[462600.645983] traps: node[11467] general protection ip:7fdb54626532 sp:7ffc7eb7d190 error:0 in libc-2.19.so[7fdb545f0000+1a1000]
<6>[462689.979224] traps: node[9751] general protection ip:7fad9d729532 sp:7ffe0df528a0 error:0 in libc-2.19.so[7fad9d6f3000+1a1000]
<6>[462823.339573] traps: node[7065] general protection ip:7fd7fa2b6532 sp:7ffd0352cf90 error:0 in libc-2.19.so[7fd7fa280000+1a1000]
<6>[462863.630107] traps: node[16525] general protection ip:7f1c0d43d532 sp:7ffc764ec650 error:0 in libc-2.19.so[7f1c0d407000+1a1000]
<6>[462885.789171] traps: node[43723] general protection ip:7fa1d74af532 sp:7ffdfd4789f0 error:0 in libc-2.19.so[7fa1d7479000+1a1000]
<6>[462974.431144] traps: node[20547] general protection ip:7fd138ed5532 sp:7ffdfaf48c00 error:0 in libc-2.19.so[7fd138e9f000+1a1000]
<6>[463427.381396] traps: node[10297] general protection ip:7f3fd2be3532 sp:7fffae7aac80 error:0 in libc-2.19.so[7f3fd2bad000+1a1000]
<6>[463465.922857] traps: node[33044] general protection ip:7ff5b597f532 sp:7ffc35a75970 error:0 in libc-2.19.so[7ff5b5949000+1a1000]
<6>[463639.454464] traps: node[43935] general protection ip:7f3bb62a0532 sp:7ffffc2091f0 error:0 in libc-2.19.so[7f3bb626a000+1a1000]
<6>[463655.733297] traps: node[11216] general protection ip:7f5c343ef532 sp:7ffd92b80170 error:0 in libc-2.19.so[7f5c343b9000+1a1000]
<6>[463668.984363] traps: node[32908] general protection ip:7f1ad7f65532 sp:7fffe5a0aca0 error:0 in libc-2.19.so[7f1ad7f2f000+1a1000]
<6>[463720.967430] traps: node[7269] general protection ip:7f2721e90532 sp:7ffe69b13a90 error:0 in libc-2.19.so[7f2721e5a000+1a1000]
<6>[463795.781919] traps: node[12777] general protection ip:7fcc231ec532 sp:7ffe251a4f90 error:0 in libc-2.19.so[7fcc231b6000+1a1000]
<6>[464057.362623] traps: node[5792] general protection ip:7f56dd9aa532 sp:7ffd2469a2b0 error:0 in libc-2.19.so[7f56dd974000+1a1000]
<6>[464128.659672] traps: node[40570] general protection ip:7f39a744b532 sp:7fffeb2e2280 error:0 in libc-2.19.so[7f39a7415000+1a1000]
<6>[464213.349539] traps: node[1511] general protection ip:7fe63485e532 sp:7ffd9495a750 error:0 in libc-2.19.so[7fe634828000+1a1000]
<6>[464226.156691] traps: node[27943] general protection ip:7fba5a17a532 sp:7ffe05a6d020 error:0 in libc-2.19.so[7fba5a144000+1a1000]
<6>[464263.264473] traps: node[39542] general protection ip:7f27caf02532 sp:7ffc6d0597e0 error:0 in libc-2.19.so[7f27caecc000+1a1000]
<6>[464294.299054] traps: node[19204] general protection ip:7f66b5d47532 sp:7ffef75dcd50 error:0 in libc-2.19.so[7f66b5d11000+1a1000]
<6>[464683.386649] traps: node[47602] general protection ip:7f4d137df532 sp:7ffd0f4a3ea0 error:0 in libc-2.19.so[7f4d137a9000+1a1000]
<6>[464761.886593] traps: node[24223] general protection ip:7fd5d8238532 sp:7ffdd26e8bb0 error:0 in libc-2.19.so[7fd5d8202000+1a1000]
<6>[464843.477481] traps: node[17475] general protection ip:7ff70c207532 sp:7ffffc2df100 error:0 in libc-2.19.so[7ff70c1d1000+1a1000]
<6>[464977.665677] traps: node[7201] general protection ip:7f35bfb9c532 sp:7ffc9da14fc0 error:0 in libc-2.19.so[7f35bfb66000+1a1000]
<6>[465066.882301] traps: node[42534] general protection ip:7f56edea9532 sp:7ffd75d3dbf0 error:0 in libc-2.19.so[7f56ede73000+1a1000]
<6>[465223.300440] traps: node[45969] general protection ip:7f05a6126532 sp:7ffc25e4d540 error:0 in libc-2.19.so[7f05a60f0000+1a1000]
<6>[465869.040676] traps: node[27977] general protection ip:7f2f9ae7d532 sp:7ffeb9578f00 error:0 in libc-2.19.so[7f2f9ae47000+1a1000]
<6>[465936.561655] traps: node[9062] general protection ip:7f45448ac532 sp:7ffdf52ef9c0 error:0 in libc-2.19.so[7f4544876000+1a1000]
<6>[466050.042270] traps: node[11073] general protection ip:7f2f2c5c7532 sp:7ffed1eb1be0 error:0 in libc-2.19.so[7f2f2c591000+1a1000]
<6>[466165.415966] traps: node[6036] general protection ip:7ff5736a0532 sp:7ffd90de9480 error:0 in libc-2.19.so[7ff57366a000+1a1000]
<0>[466230.113784] NMI watchdog: Watchdog detected hard LOCKUP on cpu 11
<4>[466230.113786] Modules linked in: nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 nfs lockd grace sunrpc fscache xt_multiport iptable_mangle iptable_raw ip_set_hash_net veth xt_set ip_set_hash_ipportnet ip_set_bitmap_port ip_set_hash_ipportip ip_set_hash_ipport ip_set dummy xt_addrtype ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6_tables ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_comment xt_mark iptable_nat nf_nat_ipv4 iptable_filter xt_conntrack nf_nat nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo overlay nls_ascii nls_cp437 vfat fat xfs sb_edac edac_core coretemp x86_pkg_temp_thermal ipmi_ssif i2c_core kvm_intel kvm dcdbas mei_me ipmi_si irqbypass mei mousedev ipmi_devintf evdev ipmi_msghandler button sch_fq_codel br_netfilter bridge stp llc nf_conntrack_ipv4 nf_defrag_ipv4 ip_vs_sh
<4>[466230.113829]  ip_vs_wrr ip_vs_rr ip_vs nf_conntrack libcrc32c bonding hid_generic usbhid hid ext4 crc32c_generic crc16 mbcache jbd2 fscrypto dm_verity dm_bufio crc32c_intel aesni_intel aes_x86_64 sd_mod crypto_simd ehci_pci tg3 cryptd ehci_hcd hwmon ahci glue_helper libahci megaraid_sas ptp libata usbcore pps_core usb_common scsi_mod libphy dm_mirror dm_region_hash dm_log dm_mod dax
<4>[466230.113857] CPU: 11 PID: 679 Comm: node Not tainted 4.14.48-coreos-r2 #1
<4>[466230.113858] Hardware name: Dell Inc. PowerEdge R730xd/072T6D, BIOS 2.8.0 005/17/2018
<4>[466230.113860] task: ffff9f4e28d01e00 task.stack: ffffb171a0294000
<4>[466230.113868] RIP: 0010:native_queued_spin_lock_slowpath+0x175/0x190
<4>[466230.113869] RSP: 0000:ffffb171a0297af0 EFLAGS: 00000046
<4>[466230.113870] RAX: 0000000000000000 RBX: ffff9f5ebf15d1a8 RCX: ffff9f5ebf162040
<4>[466230.113871] RDX: 0000000000000023 RSI: 0000000000900101 RDI: ffff9f4ebf81d1a8
<4>[466230.113872] RBP: ffffb171a0297be0 R08: 0000000000300000 R09: 0000000000000003
<4>[466230.113873] R10: ffffb171a0297c00 R11: 0000000000000131 R12: ffffb171a0297b58
<4>[466230.113875] R13: ffff9f4ebf81d1a8 R14: ffff9f4ebf81d1a0 R15: ffff9f5ebf15d1a0
<4>[466230.113876] FS:  00007f966a92c700(0000) GS:ffff9f5ebf140000(0000) knlGS:0000000000000000
<4>[466230.113878] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
<4>[466230.113879] CR2: 00001f3ec61ed018 CR3: 0000000b3c9c4006 CR4: 00000000003606e0
<4>[466230.113880] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
<4>[466230.113881] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
<4>[466230.113882] Call Trace:
<4>[466230.113890]  _raw_spin_lock_irq+0x24/0x27
<4>[466230.113896]  stop_two_cpus+0x146/0x270
<4>[466230.113900]  ? cpu_stopper_thread+0x100/0x100
<4>[466230.113902]  ? cpu_stopper_thread+0x100/0x100
<4>[466230.113906]  ? __migrate_swap_task.part.82+0x80/0x80
<4>[466230.113909]  ? migrate_swap+0x9a/0x120
<4>[466230.113910]  migrate_swap+0x9a/0x120
<4>[466230.113915]  task_numa_migrate+0x447/0x8b0
<4>[466230.113919]  task_numa_fault+0xa0e/0xd60
<4>[466230.113923]  ? get_futex_key+0x297/0x340
<4>[466230.113925]  ? futex_wake+0x93/0x180
<4>[466230.113927]  ? should_numa_migrate_memory+0x52/0x120
<4>[466230.113930]  ? __handle_mm_fault+0xbb4/0x1180
<4>[466230.113932]  __handle_mm_fault+0xbb4/0x1180
<4>[466230.113935]  handle_mm_fault+0xaa/0x1e0
<4>[466230.113940]  __do_page_fault+0x243/0x4c0
<4>[466230.113943]  ? page_fault+0x2f/0x50
<4>[466230.113945]  page_fault+0x45/0x50
<4>[466230.113948] RIP: 592f660:0x98
<4>[466230.113949] RSP: c8392e49:00007f966a92bdb0 EFLAGS: 3498f4182309
<4>[466230.113950] Code: 12 83 e0 03 83 ea 01 48 c1 e0 04 48 63 d2 48 05 40 20 02 00 48 03 04 d5 80 26 00 b7 48 89 08 8b 41 08 85 c0 75 09 f3 90 8b 41 08 <85> c0 74 f7 4c 8b 09 4d 85 c9 0f 84 5e ff ff ff 41 0f 0d 09 e9
<0>[466230.113979] Kernel panic - not syncing: Hard LOCKUP
<4>[466230.113980] CPU: 11 PID: 679 Comm: node Not tainted 4.14.48-coreos-r2 #1
<4>[466230.113981] Hardware name: Dell Inc. PowerEdge R730xd/072T6D, BIOS 2.8.0 005/17/2018
<4>[466230.113982] Call Trace:
<4>[466230.113984]  <NMI>
<4>[466230.113987]  dump_stack+0x5c/0x85
<4>[466230.113989]  panic+0xe4/0x252
<4>[466230.113992]  nmi_panic+0x35/0x40
<4>[466230.113996]  watchdog_overflow_callback+0xe3/0x100
<4>[466230.114000]  __perf_event_overflow+0x52/0xe0
<4>[466230.114005]  intel_pmu_handle_irq+0x25b/0x4e0
<4>[466230.114009]  ? __set_pte_vaddr+0x32/0x50
<4>[466230.114011]  ? __set_pte_vaddr+0x32/0x50
<4>[466230.114014]  ? __native_set_fixmap+0x24/0x30
<4>[466230.114019]  ? ghes_copy_tofrom_phys+0xde/0x260
<4>[466230.114023]  ? perf_event_nmi_handler+0x2d/0x50
<4>[466230.114025]  perf_event_nmi_handler+0x2d/0x50
<4>[466230.114028]  nmi_handle+0x63/0x110
<4>[466230.114030]  default_do_nmi+0x4e/0x100
<4>[466230.114032]  do_nmi+0xe5/0x140
<4>[466230.114035]  end_repeat_nmi+0x16/0x50
<4>[466230.114038] RIP: 0010:native_queued_spin_lock_slowpath+0x175/0x190
<4>[466230.114039] RSP: 0000:ffffb171a0297af0 EFLAGS: 00000046
<4>[466230.114040] RAX: 0000000000000000 RBX: ffff9f5ebf15d1a8 RCX: ffff9f5ebf162040
<4>[466230.114041] RDX: 0000000000000023 RSI: 0000000000900101 RDI: ffff9f4ebf81d1a8
<4>[466230.114042] RBP: ffffb171a0297be0 R08: 0000000000300000 R09: 0000000000000003
<4>[466230.114043] R10: ffffb171a0297c00 R11: 0000000000000131 R12: ffffb171a0297b58
<4>[466230.114044] R13: ffff9f4ebf81d1a8 R14: ffff9f4ebf81d1a0 R15: ffff9f5ebf15d1a0
<4>[466230.114047]  ? native_queued_spin_lock_slowpath+0x175/0x190
<4>[466230.114049]  ? native_queued_spin_lock_slowpath+0x175/0x190
<4>[466230.114050]  </NMI>
<4>[466230.114052]  _raw_spin_lock_irq+0x24/0x27
<4>[466230.114054]  stop_two_cpus+0x146/0x270
<4>[466230.114056]  ? cpu_stopper_thread+0x100/0x100
<4>[466230.114058]  ? cpu_stopper_thread+0x100/0x100
<4>[466230.114060]  ? __migrate_swap_task.part.82+0x80/0x80
<4>[466230.114062]  ? migrate_swap+0x9a/0x120
<4>[466230.114063]  migrate_swap+0x9a/0x120
<4>[466230.114066]  task_numa_migrate+0x447/0x8b0
<4>[466230.114069]  task_numa_fault+0xa0e/0xd60
<4>[466230.114070]  ? get_futex_key+0x297/0x340
<4>[466230.114072]  ? futex_wake+0x93/0x180
<4>[466230.114074]  ? should_numa_migrate_memory+0x52/0x120
<4>[466230.114076]  ? __handle_mm_fault+0xbb4/0x1180
<4>[466230.114078]  __handle_mm_fault+0xbb4/0x1180
<4>[466230.114080]  handle_mm_fault+0xaa/0x1e0
<4>[466230.114082]  __do_page_fault+0x243/0x4c0
<4>[466230.114085]  ? page_fault+0x2f/0x50
<4>[466230.114087]  page_fault+0x45/0x50
<4>[466230.114088] RIP: 592f660:0x98
<4>[466230.114089] RSP: c8392e49:00007f966a92bdb0 EFLAGS: 3498f4182309
<0>[466230.872342] NMI watchdog: Watchdog detected hard LOCKUP on cpu 31
<4>[466230.872343] Modules linked in: nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 nfs lockd grace sunrpc fscache xt_multiport iptable_mangle iptable_raw ip_set_hash_net veth xt_set ip_set_hash_ipportnet ip_set_bitmap_port ip_set_hash_ipportip ip_set_hash_ipport ip_set dummy xt_addrtype ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6_tables ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_comment xt_mark iptable_nat nf_nat_ipv4 iptable_filter xt_conntrack nf_nat nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo overlay nls_ascii nls_cp437 vfat fat xfs sb_edac edac_core coretemp x86_pkg_temp_thermal ipmi_ssif i2c_core kvm_intel kvm dcdbas mei_me ipmi_si irqbypass mei mousedev ipmi_devintf evdev ipmi_msghandler button sch_fq_codel br_netfilter bridge stp llc nf_conntrack_ipv4 nf_defrag_ipv4 ip_vs_sh
<4>[466230.872375]  ip_vs_wrr ip_vs_rr ip_vs nf_conntrack libcrc32c bonding hid_generic usbhid hid ext4 crc32c_generic crc16 mbcache jbd2 fscrypto dm_verity dm_bufio crc32c_intel aesni_intel aes_x86_64 sd_mod crypto_simd ehci_pci tg3 cryptd ehci_hcd hwmon ahci glue_helper libahci megaraid_sas ptp libata usbcore pps_core usb_common scsi_mod libphy dm_mirror dm_region_hash dm_log dm_mod dax
<4>[466230.872396] CPU: 31 PID: 10943 Comm: node Not tainted 4.14.48-coreos-r2 #1
<4>[466230.872396] Hardware name: Dell Inc. PowerEdge R730xd/072T6D, BIOS 2.8.0 005/17/2018
<4>[466230.872398] task: ffff9f4bc8e00000 task.stack: ffffb1718dfa8000
<4>[466230.872401] RIP: 0010:native_queued_spin_lock_slowpath+0x172/0x190
<4>[466230.872402] RSP: 0000:ffffb1718dfabaf0 EFLAGS: 00000046
<4>[466230.872403] RAX: 0000000000000000 RBX: ffff9f5ebf3dd1a8 RCX: ffff9f5ebf3e2040
<4>[466230.872405] RDX: 0000000000000021 RSI: 0000000000880101 RDI: ffff9f4ebf81d1a8
<4>[466230.872405] RBP: ffffb1718dfabbe0 R08: 0000000000800000 R09: 0000000000000003
<4>[466230.872407] R10: ffffb1718dfabc00 R11: 000000000000c208 R12: ffffb1718dfabb58
<4>[466230.872408] R13: ffff9f4ebf81d1a8 R14: ffff9f4ebf81d1a0 R15: ffff9f5ebf3dd1a0
<4>[466230.872409] FS:  00007fe0fdbe0740(0000) GS:ffff9f5ebf3c0000(0000) knlGS:0000000000000000
<4>[466230.872410] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
<4>[466230.872412] CR2: 0000000005f2d538 CR3: 0000000d3583c004 CR4: 00000000003606e0
<4>[466230.872413] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
<4>[466230.872414] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
<4>[466230.872414] Call Trace:
<4>[466230.872418]  _raw_spin_lock_irq+0x24/0x27
<4>[466230.872420]  stop_two_cpus+0x146/0x270
<4>[466230.872424]  ? cpu_stopper_thread+0x100/0x100
<4>[466230.872426]  ? cpu_stopper_thread+0x100/0x100
<4>[466230.872428]  ? __migrate_swap_task.part.82+0x80/0x80
<4>[466230.872430]  ? migrate_swap+0x9a/0x120
<4>[466230.872431]  migrate_swap+0x9a/0x120
<4>[466230.872434]  task_numa_migrate+0x447/0x8b0
<4>[466230.872438]  task_numa_fault+0xa0e/0xd60
<4>[466230.872442]  ? migrate_pages+0x582/0xa50
<4>[466230.872444]  ? __handle_mm_fault+0xbb4/0x1180
<4>[466230.872446]  __handle_mm_fault+0xbb4/0x1180
<4>[466230.872448]  handle_mm_fault+0xaa/0x1e0
<4>[466230.872451]  __do_page_fault+0x243/0x4c0
<4>[466230.872454]  ? page_fault+0x2f/0x50
<4>[466230.872456]  page_fault+0x45/0x50
<4>[466230.872457] RIP: 22170f0:0x7ffce1bb72e0
<4>[466230.872458] RSP: 0000:00007ffce1bba310 EFLAGS: 02216ec0
<4>[466230.872460] Code: ce c1 ea 12 83 e0 03 83 ea 01 48 c1 e0 04 48 63 d2 48 05 40 20 02 00 48 03 04 d5 80 26 00 b7 48 89 08 8b 41 08 85 c0 75 09 f3 90 <8b> 41 08 85 c0 74 f7 4c 8b 09 4d 85 c9 0f 84 5e ff ff ff 41 0f
<0>[466231.178868] Shutting down cpus with NMI
<0>[466231.275975] Kernel Offset: 0x35000000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
@ajeddeloh

This comment has been minimized.

Show comment
Hide comment
@ajeddeloh

ajeddeloh Aug 13, 2018

As a first step, can you test with the latest beta, stable and alpha. They all have different kernels so if one of them works that would greatly narrow down the problem.

ajeddeloh commented Aug 13, 2018

As a first step, can you test with the latest beta, stable and alpha. They all have different kernels so if one of them works that would greatly narrow down the problem.

@obion

This comment has been minimized.

Show comment
Hide comment
@obion

obion Aug 14, 2018

Ok! Let's try. It will take some time.

obion commented Aug 14, 2018

Ok! Let's try. It will take some time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment