New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Killing container with id docker **** Need to kill Pod #56891

Closed
xingxing9688 opened this Issue Dec 6, 2017 · 3 comments

Comments

Projects
None yet
5 participants
@xingxing9688

xingxing9688 commented Dec 6, 2017

[Wed Dec 6 23:54:11 2017] java invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=-998
[Wed Dec 6 23:54:11 2017] java cpuset=bf2d7043c1267b4dfd8fdb356653a73070a8db8dfd7e2e4d74700edcaef7b548 mems_allowed=0
[Wed Dec 6 23:54:11 2017] CPU: 2 PID: 5282 Comm: java Tainted: G B ------------ T 3.10.0-327.4.5.el7.x86_64 #1
[Wed Dec 6 23:54:11 2017] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.8.2-0-g33fbe13 by qemu-project.org 04/01/2014
[Wed Dec 6 23:54:11 2017] ffff8806db522280 00000000cdb595c3 ffff88066838fca8 ffffffff8163515c
[Wed Dec 6 23:54:11 2017] ffff88066838fd38 ffffffff816300fc ffff8805d6075cc0 0000000000000001
[Wed Dec 6 23:54:11 2017] 0000000000000003 ffff88066838fd08 ffffffff8116c946 ffff8807dab827e8
[Wed Dec 6 23:54:11 2017] Call Trace:
[Wed Dec 6 23:54:11 2017] [] dump_stack+0x19/0x1b
[Wed Dec 6 23:54:11 2017] [] dump_header+0x8e/0x214
[Wed Dec 6 23:54:11 2017] [] ? find_lock_task_mm+0x56/0xc0
[Wed Dec 6 23:54:11 2017] [] oom_kill_process+0x24e/0x3b0
[Wed Dec 6 23:54:11 2017] [] ? has_capability_noaudit+0x1e/0x30
[Wed Dec 6 23:54:11 2017] [] mem_cgroup_oom_synchronize+0x555/0x580
[Wed Dec 6 23:54:11 2017] [] ? mem_cgroup_charge_common+0xc0/0xc0
[Wed Dec 6 23:54:11 2017] [] pagefault_out_of_memory+0x14/0x90
[Wed Dec 6 23:54:11 2017] [] mm_fault_error+0x68/0x12b
[Wed Dec 6 23:54:11 2017] [] __do_page_fault+0x3e2/0x450
[Wed Dec 6 23:54:11 2017] [] trace_do_page_fault+0x43/0x110
[Wed Dec 6 23:54:11 2017] [] do_async_page_fault+0x29/0xe0
[Wed Dec 6 23:54:11 2017] [] async_page_fault+0x28/0x30
[Wed Dec 6 23:54:11 2017] Task in /kubepods/pod95387fa4-da9c-11e7-9d36-5254b24cbf5e/bf2d7043c1267b4dfd8fdb356653a73070a8db8dfd7e2e4d74700edcaef7b548 killed as a result of limit of /kubepods/pod95387fa4-da9c-11e7-9d36-5254b24cbf5e/bf2d7043c1267b4dfd8fdb356653a73070a8db8dfd7e2e4d74700edcaef7b548
[Wed Dec 6 23:54:11 2017] memory: usage 524252kB, limit 524288kB, failcnt 152667
[Wed Dec 6 23:54:11 2017] memory+swap: usage 1048576kB, limit 1048576kB, failcnt 6
[Wed Dec 6 23:54:11 2017] kmem: usage 0kB, limit 9007199254740991kB, failcnt 0
[Wed Dec 6 23:54:11 2017] Memory cgroup stats for /kubepods/pod95387fa4-da9c-11e7-9d36-5254b24cbf5e/bf2d7043c1267b4dfd8fdb356653a73070a8db8dfd7e2e4d74700edcaef7b548: cache:16KB rss:524236KB rss_huge:354304KB mapped_file:0KB swap:524324KB inactive_anon:264540KB active_anon:259688KB inactive_file:12KB active_file:4KB unevictable:0KB
[Wed Dec 6 23:54:11 2017] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
[Wed Dec 6 23:54:11 2017] [ 5270] 0 5270 5064680 119053 1675 137996 -998 java
[Wed Dec 6 23:54:11 2017] Memory cgroup out of memory: Kill process 28386 (java) score 0 or sacrifice child
[Wed Dec 6 23:54:11 2017] Killed process 5270 (java) total-vm:20258720kB, anon-rss:462616kB, file-rss:13596kB
[Wed Dec 6 23:58:46 2017] libceph: osd0 10.39.1.55:6800 socket closed (con state OPEN)
[Thu Dec 7 00:05:14 2017] java invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=-998
[Thu Dec 7 00:05:14 2017] java cpuset=6895a3e13439535852d3184ac2aadd26d7363cc035f171545ec7d4ce74073b0b mems_allowed=0
[Thu Dec 7 00:05:14 2017] CPU: 11 PID: 28456 Comm: java Tainted: G B ------------ T 3.10.0-327.4.5.el7.x86_64 #1
[Thu Dec 7 00:05:14 2017] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.8.2-0-g33fbe13 by qemu-project.org 04/01/2014
[Thu Dec 7 00:05:14 2017] ffff8807f0f9c500 000000000854d4f0 ffff880631cbbca8 ffffffff8163515c
[Thu Dec 7 00:05:14 2017] ffff880631cbbd38 ffffffff816300fc ffff8805d6075c30 0000000000000001
[Thu Dec 7 00:05:14 2017] 0000000000000003 ffff880631cbbd08 ffffffff8116c946 ffff880614dbd5e8
[Thu Dec 7 00:05:14 2017] Call Trace:
[Thu Dec 7 00:05:14 2017] [] dump_stack+0x19/0x1b
[Thu Dec 7 00:05:14 2017] [] dump_header+0x8e/0x214
[Thu Dec 7 00:05:14 2017] [] ? find_lock_task_mm+0x56/0xc0
[Thu Dec 7 00:05:14 2017] [] oom_kill_process+0x24e/0x3b0
[Thu Dec 7 00:05:14 2017] [] ? has_capability_noaudit+0x1e/0x30
[Thu Dec 7 00:05:14 2017] [] mem_cgroup_oom_synchronize+0x555/0x580
[Thu Dec 7 00:05:14 2017] [] ? mem_cgroup_charge_common+0xc0/0xc0
[Thu Dec 7 00:05:14 2017] [] pagefault_out_of_memory+0x14/0x90
[Thu Dec 7 00:05:14 2017] [] mm_fault_error+0x68/0x12b
[Thu Dec 7 00:05:14 2017] [] __do_page_fault+0x3e2/0x450
[Thu Dec 7 00:05:14 2017] [] trace_do_page_fault+0x43/0x110
[Thu Dec 7 00:05:14 2017] [] do_async_page_fault+0x29/0xe0
[Thu Dec 7 00:05:14 2017] [] async_page_fault+0x28/0x30
[Thu Dec 7 00:05:14 2017] Task in /kubepods/pod95387fa4-da9c-11e7-9d36-5254b24cbf5e/6895a3e13439535852d3184ac2aadd26d7363cc035f171545ec7d4ce74073b0b killed as a result of limit of /kubepods/pod95387fa4-da9c-11e7-9d36-5254b24cbf5e/6895a3e13439535852d3184ac2aadd26d7363cc035f171545ec7d4ce74073b0b
[Thu Dec 7 00:05:14 2017] memory: usage 524220kB, limit 524288kB, failcnt 159355
[Thu Dec 7 00:05:14 2017] memory+swap: usage 1048576kB, limit 1048576kB, failcnt 6
[Thu Dec 7 00:05:14 2017] kmem: usage 0kB, limit 9007199254740991kB, failcnt 0
[Thu Dec 7 00:05:14 2017] Memory cgroup stats for /kubepods/pod95387fa4-da9c-11e7-9d36-5254b24cbf5e/6895a3e13439535852d3184ac2aadd26d7363cc035f171545ec7d4ce74073b0b: cache:32KB rss:524188KB rss_huge:370688KB mapped_file:0KB swap:524356KB inactive_anon:267804KB active_anon:256384KB inactive_file:12KB active_file:4KB unevictable:0KB
[Thu Dec 7 00:05:15 2017] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
[Thu Dec 7 00:05:15 2017] [28438] 0 28438 5063909 119961 1661 136409 -998 java
[Thu Dec 7 00:05:15 2017] Memory cgroup out of memory: Kill process 24572 (java) score 0 or sacrifice child
[Thu Dec 7 00:05:15 2017] Killed process 28438 (java) total-vm:20255636kB, anon-rss:466160kB, file-rss:13684kB
[Thu Dec 7 00:13:22 2017] java invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=-998
[Thu Dec 7 00:13:22 2017] java cpuset=1e479e759020fa0b6c72def90973540a2d33982fea2ad67a96373b4da751a820 mems_allowed=0
[Thu Dec 7 00:13:22 2017] CPU: 9 PID: 14881 Comm: java Tainted: G B ------------ T 3.10.0-327.4.5.el7.x86_64 #1
[Thu Dec 7 00:13:22 2017] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.8.2-0-g33fbe13 by qemu-project.org 04/01/2014
[Thu Dec 7 00:13:22 2017] ffff8807f0d6b980 0000000051be382e ffff88039bc07ca8 ffffffff8163515c
[Thu Dec 7 00:13:22 2017] ffff88039bc07d38 ffffffff816300fc ffff88080fe08498 0000000000000001
[Thu Dec 7 00:13:22 2017] 0000000000000003 ffff88039bc07d08 ffffffff8116c946 ffff880054a7a7e8
[Thu Dec 7 00:13:22 2017] Call Trace:
[Thu Dec 7 00:13:22 2017] [] dump_stack+0x19/0x1b
[Thu Dec 7 00:13:22 2017] [] dump_header+0x8e/0x214
[Thu Dec 7 00:13:22 2017] [] ? find_lock_task_mm+0x56/0xc0
[Thu Dec 7 00:13:22 2017] [] oom_kill_process+0x24e/0x3b0
[Thu Dec 7 00:13:22 2017] [] ? has_capability_noaudit+0x1e/0x30
[Thu Dec 7 00:13:22 2017] [] mem_cgroup_oom_synchronize+0x555/0x580
[Thu Dec 7 00:13:22 2017] [] ? mem_cgroup_charge_common+0xc0/0xc0
[Thu Dec 7 00:13:22 2017] [] pagefault_out_of_memory+0x14/0x90
[Thu Dec 7 00:13:22 2017] [] mm_fault_error+0x68/0x12b
[Thu Dec 7 00:13:22 2017] [] __do_page_fault+0x3e2/0x450
[Thu Dec 7 00:13:22 2017] [] trace_do_page_fault+0x43/0x110
[Thu Dec 7 00:13:22 2017] [] do_async_page_fault+0x29/0xe0
[Thu Dec 7 00:13:22 2017] [] async_page_fault+0x28/0x30
[Thu Dec 7 00:13:22 2017] Task in /kubepods/pod95387fa4-da9c-11e7-9d36-5254b24cbf5e/1e479e759020fa0b6c72def90973540a2d33982fea2ad67a96373b4da751a820 killed as a result of limit of /kubepods/pod95387fa4-da9c-11e7-9d36-5254b24cbf5e/1e479e759020fa0b6c72def90973540a2d33982fea2ad67a96373b4da751a820
[Thu Dec 7 00:13:22 2017] memory: usage 524184kB, limit 524288kB, failcnt 166422
[Thu Dec 7 00:13:22 2017] memory+swap: usage 1048576kB, limit 1048576kB, failcnt 15
[Thu Dec 7 00:13:22 2017] kmem: usage 0kB, limit 9007199254740991kB, failcnt 0
[Thu Dec 7 00:13:22 2017] Memory cgroup stats for /kubepods/pod95387fa4-da9c-11e7-9d36-5254b24cbf5e/1e479e759020fa0b6c72def90973540a2d33982fea2ad67a96373b4da751a820: cache:20KB rss:524164KB rss_huge:354304KB mapped_file:0KB swap:524392KB inactive_anon:263468KB active_anon:260648KB inactive_file:12KB active_file:0KB unevictable:0KB
[Thu Dec 7 00:13:22 2017] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
[Thu Dec 7 00:13:22 2017] [24632] 0 24632 5063138 119363 1658 134623 -998 java
[Thu Dec 7 00:13:22 2017] Memory cgroup out of memory: Kill process 14881 (java) score 0 or sacrifice child
[Thu Dec 7 00:13:22 2017] Killed process 24632 (java) total-vm:20252552kB, anon-rss:463684kB, file-rss:13768kB
[Thu Dec 7 00:13:46 2017] libceph: osd0 10.39.1.55:6800 socket closed (con state OPEN)
[Thu Dec 7 00:15:03 2017] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[Thu Dec 7 00:15:03 2017] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[Thu Dec 7 00:20:58 2017] java invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=-998
[Thu Dec 7 00:20:58 2017] java cpuset=e84ed2eecf157a08df53ca510418796ccf5173f5d71e42133320e9178b9c8447 mems_allowed=0
[Thu Dec 7 00:20:58 2017] CPU: 5 PID: 14984 Comm: java Tainted: G B ------------ T 3.10.0-327.4.5.el7.x86_64 #1
[Thu Dec 7 00:20:58 2017] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.8.2-0-g33fbe13 by qemu-project.org 04/01/2014
[Thu Dec 7 00:20:58 2017] ffff88001f3c0b80 00000000358e7705 ffff8807db143ca8 ffffffff8163515c
[Thu Dec 7 00:20:58 2017] ffff8807db143d38 ffffffff816300fc ffff8808131310f0 0000000000000001
[Thu Dec 7 00:20:58 2017] ffff8800bb89d740 ffff8807db143d08 ffffffff8116c946 ffff88073806f868
[Thu Dec 7 00:20:58 2017] Call Trace:
[Thu Dec 7 00:20:58 2017] [] dump_stack+0x19/0x1b
[Thu Dec 7 00:20:58 2017] [] dump_header+0x8e/0x214
[Thu Dec 7 00:20:58 2017] [] ? find_lock_task_mm+0x56/0xc0
[Thu Dec 7 00:20:58 2017] [] oom_kill_process+0x24e/0x3b0
[Thu Dec 7 00:20:58 2017] [] ? has_capability_noaudit+0x1e/0x30
[Thu Dec 7 00:20:58 2017] [] mem_cgroup_oom_synchronize+0x555/0x580
[Thu Dec 7 00:20:58 2017] [] ? mem_cgroup_charge_common+0xc0/0xc0
[Thu Dec 7 00:20:58 2017] [] pagefault_out_of_memory+0x14/0x90
[Thu Dec 7 00:20:58 2017] [] mm_fault_error+0x68/0x12b
[Thu Dec 7 00:20:58 2017] [] __do_page_fault+0x3e2/0x450
[Thu Dec 7 00:20:58 2017] [] trace_do_page_fault+0x43/0x110
[Thu Dec 7 00:20:58 2017] [] do_async_page_fault+0x29/0xe0
[Thu Dec 7 00:20:58 2017] [] async_page_fault+0x28/0x30
[Thu Dec 7 00:20:58 2017] Task in /kubepods/pod95387fa4-da9c-11e7-9d36-5254b24cbf5e/e84ed2eecf157a08df53ca510418796ccf5173f5d71e42133320e9178b9c8447 killed as a result of limit of /kubepods/pod95387fa4-da9c-11e7-9d36-5254b24cbf5e/e84ed2eecf157a08df53ca510418796ccf5173f5d71e42133320e9178b9c8447
[Thu Dec 7 00:20:58 2017] memory: usage 524092kB, limit 524288kB, failcnt 164324
[Thu Dec 7 00:20:58 2017] memory+swap: usage 1048576kB, limit 1048576kB, failcnt 17
[Thu Dec 7 00:20:58 2017] kmem: usage 0kB, limit 9007199254740991kB, failcnt 0
[Thu Dec 7 00:20:58 2017] Memory cgroup stats for /kubepods/pod95387fa4-da9c-11e7-9d36-5254b24cbf5e/e84ed2eecf157a08df53ca510418796ccf5173f5d71e42133320e9178b9c8447: cache:32KB rss:524060KB rss_huge:307200KB mapped_file:12KB swap:524484KB inactive_anon:262104KB active_anon:261936KB inactive_file:4KB active_file:4KB unevictable:0KB
[Thu Dec 7 00:20:58 2017] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
[Thu Dec 7 00:20:58 2017] [14970] 0 14970 5090123 117579 1714 135473 -998 java
[Thu Dec 7 00:20:58 2017] Memory cgroup out of memory: Kill process 3068 (java) score 0 or sacrifice child

kubernetes version 1.7.6
docker version 1.12.6
OS centos 7.2 3.10.0-327.4.5.el7.x86_64

@k8s-merge-robot

This comment has been minimized.

Show comment
Hide comment
@k8s-merge-robot

k8s-merge-robot Dec 6, 2017

Contributor

@xingxing9688
There are no sig labels on this issue. Please add a sig label by:

  1. mentioning a sig: @kubernetes/sig-<group-name>-<group-suffix>
    e.g., @kubernetes/sig-contributor-experience-<group-suffix> to notify the contributor experience sig, OR

  2. specifying the label manually: /sig <label>
    e.g., /sig scalability to apply the sig/scalability label

Note: Method 1 will trigger an email to the group. See the group list and label list.
The <group-suffix> in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals

Contributor

k8s-merge-robot commented Dec 6, 2017

@xingxing9688
There are no sig labels on this issue. Please add a sig label by:

  1. mentioning a sig: @kubernetes/sig-<group-name>-<group-suffix>
    e.g., @kubernetes/sig-contributor-experience-<group-suffix> to notify the contributor experience sig, OR

  2. specifying the label manually: /sig <label>
    e.g., /sig scalability to apply the sig/scalability label

Note: Method 1 will trigger an email to the group. See the group list and label list.
The <group-suffix> in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals

@tianshapjq

This comment has been minimized.

Show comment
Hide comment
@tianshapjq

tianshapjq Dec 7, 2017

Contributor

you should follow the template to describe your problems, just error logs may not enough

Contributor

tianshapjq commented Dec 7, 2017

you should follow the template to describe your problems, just error logs may not enough

@dims

This comment has been minimized.

Show comment
Hide comment
@dims

dims Dec 11, 2017

Member

please open a new issue with description and details please

/close

Member

dims commented Dec 11, 2017

please open a new issue with description and details please

/close

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment