Linux kernel panic and reboot under some conditions with cpu-manager-policy set to static

[phsiao]

Is this a BUG REPORT or FEATURE REQUEST?:

/kind bug

/sig node

What happened:

We have been running with cpu-manager-policy set to static for several weeks without issues, but starting this week we have servers spontaneously reboot when pods with resources meet the cpu-manager-policy=static criteria starting/restarting on them. It does not happen every time. These pods used to do httpGet for liveness and readiness probes, but this week they have changed to use exec and use curl directly. We have other pods using exec for probes, but they don't get updated/restarted as often.

Turning off cpu-manager-policy=static have stopped the reboot.

We did not get kernel dump due to misconfiguration, but this error showed up several times before the reboot happens.

Dec  4 18:14:48 7yjhsd2 kernel: list_add corruption. next->prev should be prev (ffff8810283b60e0), but was dead000000000200. (next=ffff88104913e6e0).
Dec  4 18:14:48 7yjhsd2 kernel: ------------[ cut here ]------------
Dec  4 18:14:48 7yjhsd2 kernel: WARNING: CPU: 13 PID: 0 at lib/list_debug.c:25 __list_add_valid+0x3a/0x80
Dec  4 18:14:48 7yjhsd2 kernel: Modules linked in: vfat fat uas usb_storage binfmt_misc tcp_diag inet_diag xt_statistic xt_nat xt_recent veth mpt3sas raid_class scsi_transport_sas mp
Dec  4 18:14:48 7yjhsd2 kernel: ip_tables xfs libcrc32c sd_mod mgag200 i2c_algo_bit crc32c_intel drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci libahci drm tg3
Dec  4 18:14:48 7yjhsd2 kernel: CPU: 13 PID: 0 Comm: swapper/13 Tainted: G        W       4.13.8-1.el7.elrepo.x86_64 #1
Dec  4 18:14:48 7yjhsd2 kernel: Hardware name: Dell Inc. PowerEdge R630/02C2CP, BIOS 2.5.5 08/16/2017
Dec  4 18:14:48 7yjhsd2 kernel: task: ffff88105ba5c5c0 task.stack: ffffc90006460000
Dec  4 18:14:48 7yjhsd2 kernel: RIP: 0010:__list_add_valid+0x3a/0x80
Dec  4 18:14:48 7yjhsd2 kernel: RSP: 0000:ffff88105ef83ca8 EFLAGS: 00010046
Dec  4 18:14:48 7yjhsd2 kernel: RAX: 0000000000000075 RBX: ffff881049948800 RCX: 0000000000000000
Dec  4 18:14:48 7yjhsd2 kernel: RDX: 0000000000000000 RSI: 00000000000000f6 RDI: 0000000000000300
Dec  4 18:14:48 7yjhsd2 kernel: RBP: ffff88105ef83ca8 R08: 0000000000000000 R09: 00000000000010dd
Dec  4 18:14:48 7yjhsd2 kernel: R10: 0000000000000004 R11: 00000000000010dc R12: ffff88105ef9c000
Dec  4 18:14:48 7yjhsd2 kernel: R13: ffff8810499488e0 R14: ffff8810283b60e0 R15: ffff88104913e6e0
Dec  4 18:14:48 7yjhsd2 kernel: FS:  0000000000000000(0000) GS:ffff88105ef80000(0000) knlGS:0000000000000000
Dec  4 18:14:48 7yjhsd2 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec  4 18:14:48 7yjhsd2 kernel: CR2: 0000000000b40ca0 CR3: 0000000001c09000 CR4: 00000000003406e0
Dec  4 18:14:48 7yjhsd2 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Dec  4 18:14:48 7yjhsd2 kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Dec  4 18:14:48 7yjhsd2 kernel: Call Trace:
Dec  4 18:14:48 7yjhsd2 kernel: <IRQ>
Dec  4 18:14:48 7yjhsd2 kernel: enqueue_entity+0x274/0x720
Dec  4 18:14:48 7yjhsd2 kernel: enqueue_task_fair+0x7d/0x8a0
Dec  4 18:14:48 7yjhsd2 kernel: ? __update_load_avg_blocked_se.isra.34+0xa4/0xf0
Dec  4 18:14:48 7yjhsd2 kernel: activate_task+0x51/0xc0
Dec  4 18:14:48 7yjhsd2 kernel: ? sched_clock+0x9/0x10
Dec  4 18:14:48 7yjhsd2 kernel: ttwu_do_activate+0x49/0x90
Dec  4 18:14:48 7yjhsd2 kernel: try_to_wake_up+0x1e1/0x470
Dec  4 18:14:48 7yjhsd2 kernel: wake_up_state+0x10/0x20
Dec  4 18:14:48 7yjhsd2 kernel: signal_wake_up_state+0x19/0x30
Dec  4 18:14:48 7yjhsd2 kernel: complete_signal+0xf6/0x1f0
Dec  4 18:14:48 7yjhsd2 kernel: send_sigqueue+0x100/0x200
Dec  4 18:14:48 7yjhsd2 kernel: ? find_next_bit+0xb/0x10
Dec  4 18:14:48 7yjhsd2 kernel: posix_timer_event+0x3b/0x50
Dec  4 18:14:48 7yjhsd2 kernel: posix_timer_fn+0x4a/0xc0
Dec  4 18:14:48 7yjhsd2 kernel: ? posix_timer_event+0x50/0x50
Dec  4 18:14:48 7yjhsd2 kernel: __hrtimer_run_queues+0xe5/0x230
Dec  4 18:14:48 7yjhsd2 kernel: hrtimer_interrupt+0xa8/0x1a0
Dec  4 18:14:48 7yjhsd2 kernel: local_apic_timer_interrupt+0x35/0x60
Dec  4 18:14:48 7yjhsd2 kernel: smp_apic_timer_interrupt+0x38/0x50
Dec  4 18:14:48 7yjhsd2 kernel: apic_timer_interrupt+0x93/0xa0
Dec  4 18:14:48 7yjhsd2 kernel: RIP: 0010:poll_idle+0x2e/0x5d
Dec  4 18:14:48 7yjhsd2 kernel: RSP: 0000:ffffc90006463e58 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff10
Dec  4 18:14:48 7yjhsd2 kernel: RAX: 0000000000200000 RBX: ffffe8ffff780300 RCX: 0000000000000000
Dec  4 18:14:48 7yjhsd2 kernel: RDX: 0000000000000000 RSI: ffff88105ba5c5c0 RDI: ffffe8ffff780300
Dec  4 18:14:48 7yjhsd2 kernel: RBP: ffffc90006463e58 R08: 0000000000006731 R09: 00000000000011b4
Dec  4 18:14:48 7yjhsd2 kernel: R10: 000000000000030b R11: 0000000000000018 R12: 0000000000000000
Dec  4 18:14:48 7yjhsd2 kernel: R13: ffffffff81d6b9f8 R14: ffffe8ffff780300 R15: ffffffff81d6b9e0
Dec  4 18:14:48 7yjhsd2 kernel: </IRQ>
Dec  4 18:14:48 7yjhsd2 kernel: cpuidle_enter_state+0x9e/0x280
Dec  4 18:14:48 7yjhsd2 kernel: cpuidle_enter+0x17/0x20
Dec  4 18:14:48 7yjhsd2 kernel: call_cpuidle+0x23/0x40
Dec  4 18:14:48 7yjhsd2 kernel: do_idle+0x172/0x1e0
Dec  4 18:14:48 7yjhsd2 kernel: cpu_startup_entry+0x73/0x80
Dec  4 18:14:48 7yjhsd2 kernel: start_secondary+0x159/0x1a0
Dec  4 18:14:48 7yjhsd2 kernel: secondary_startup_64+0x9f/0x9f
Dec  4 18:14:48 7yjhsd2 kernel: Code: 19 48 8b 32 48 39 f0 75 42 48 39 c7 74 23 48 39 fa 74 1e b8 01 00 00 00 5d c3 48 89 c1 48 c7 c7 b0 ae a8 81 31 c0 e8 29 b2 d3 ff <0f> ff 31 c0 5
Dec  4 18:14:48 7yjhsd2 kernel: ---[ end trace ea9cc7c7b92eb9c8 ]---

What you expected to happen:

Servers should kernel panic and reboot.

How to reproduce it (as minimally and precisely as possible):

Still working on this.

Anything else we need to know?:

Talked to @ConnorDoyle and decided to file a bug here. The bug may be elsewhere but cpu-manager-policy=static is part of the issue from what we observe.

Environment:

• Kubernetes version (use kubectl version): 1.8.3
• Cloud provider or hardware configuration: bare metal, Dell PowerEdge servers
• OS (e.g. from /etc/os-release): CentOS 7
• Kernel (e.g. uname -a): 4.13.8-1.el7.elrepo.x86_64
• Install tools: custom
• Others:

/cc @ConnorDoyle @sjenning

[ConnorDoyle]

/assign

[balajismaniam]

/assign

[sjenning]

Any reason you are running a ELRepo kernel instead of the standard CentOS kernel?

[phsiao]

It is a result of us trying to find a combination of kernel + docker + ceph/rbd that run more stable together. Newer kernel + newer docker worked better for us in practice. We don't yet have a engineering process to find and verify a combination, so it is mostly from experience and observation.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close