Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Agent's memory utilization always keep increasing. #88

Open
s-takeuchi opened this issue Apr 2, 2022 · 1 comment
Open

Agent's memory utilization always keep increasing. #88

s-takeuchi opened this issue Apr 2, 2022 · 1 comment
Assignees
@s-takeuchi
Labels
bug
Milestone
Future

Comments

@s-takeuchi
Copy link
Owner

Agent's memory utilization always keep increasing.
@s-takeuchi s-takeuchi added the bug label Apr 2, 2022
@s-takeuchi s-takeuchi added this to the Future milestone Apr 2, 2022
@s-takeuchi s-takeuchi self-assigned this Apr 2, 2022
@s-takeuchi
Copy link
Owner Author

s-takeuchi commented Apr 2, 2022 •
edited
Loading

Logs when OOM killer kills servalagt.

Mar 21 09:54:54 ip-172-26-5-41 kernel: rpcbind invoked oom-killer: gfp_mask=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD), nodemask=(null), order=0, oom_score_adj=0
Mar 21 09:54:54 ip-172-26-5-41 kernel: rpcbind cpuset=/ mems_allowed=0
Mar 21 09:54:54 ip-172-26-5-41 kernel: CPU: 0 PID: 2485 Comm: rpcbind Not tainted 4.14.243-185.433.amzn2.x86_64 #1
Mar 21 09:54:54 ip-172-26-5-41 kernel: Hardware name: Xen HVM domU, BIOS 4.11.amazon 08/24/2006
Mar 21 09:54:54 ip-172-26-5-41 kernel: Call Trace:
Mar 21 09:54:54 ip-172-26-5-41 kernel: dump_stack+0x66/0x81
Mar 21 09:54:54 ip-172-26-5-41 kernel: dump_header+0x94/0x21a
Mar 21 09:54:54 ip-172-26-5-41 kernel: ? do_try_to_free_pages+0x2a1/0x330
Mar 21 09:54:54 ip-172-26-5-41 kernel: ? get_page_from_freelist+0x525/0xba0
Mar 21 09:54:54 ip-172-26-5-41 kernel: oom_kill_process+0x223/0x410
Mar 21 09:54:54 ip-172-26-5-41 kernel: out_of_memory+0x112/0x4d0
Mar 21 09:54:54 ip-172-26-5-41 kernel: __alloc_pages_slowpath+0x9b6/0xd20
Mar 21 09:54:54 ip-172-26-5-41 kernel: __alloc_pages_nodemask+0x25d/0x280
Mar 21 09:54:54 ip-172-26-5-41 kernel: filemap_fault+0x213/0x640
Mar 21 09:54:54 ip-172-26-5-41 kernel: __xfs_filemap_fault.constprop.8+0x49/0x120 [xfs]
Mar 21 09:54:54 ip-172-26-5-41 kernel: __do_fault+0x31/0xe0
Mar 21 09:54:54 ip-172-26-5-41 kernel: __handle_mm_fault+0xdd5/0x12b0
Mar 21 09:54:54 ip-172-26-5-41 kernel: handle_mm_fault+0xe0/0x230
Mar 21 09:54:54 ip-172-26-5-41 kernel: __do_page_fault+0x23e/0x4c0
Mar 21 09:54:54 ip-172-26-5-41 kernel: ? page_fault+0x2f/0x50
Mar 21 09:54:54 ip-172-26-5-41 kernel: page_fault+0x45/0x50
Mar 21 09:54:54 ip-172-26-5-41 kernel: RIP: 8b8082e0:0x561a97a370e0
Mar 21 09:54:54 ip-172-26-5-41 kernel: RSP: 000c:0000000000000001 EFLAGS: 00000007
Mar 21 09:54:54 ip-172-26-5-41 kernel: Mem-Info:
Mar 21 09:54:54 ip-172-26-5-41 kernel: active_anon:103447 inactive_anon:63 isolated_anon:0#012 active_file:84 inactive_file:92 isolated_file:0#012 unevictable:0 dirty:0 writeback:0 unstable:0#012 slab_reclaimable:3419 slab_unreclaimable:5598#012 mapped:57 shmem:102 pagetables:4459 bounce:0#012 free:1115 free_pcp:61 free_cma:0
Mar 21 09:54:54 ip-172-26-5-41 kernel: Node 0 active_anon:413788kB inactive_anon:252kB active_file:336kB inactive_file:368kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:228kB dirty:0kB writeback:0kB shmem:408kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
Mar 21 09:54:54 ip-172-26-5-41 kernel: Node 0 DMA free:1860kB min:92kB low:112kB high:132kB active_anon:8992kB inactive_anon:0kB active_file:0kB inactive_file:12kB unevictable:0kB writepending:0kB present:15988kB managed:15904kB mlocked:0kB kernel_stack:228kB pagetables:2308kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Mar 21 09:54:54 ip-172-26-5-41 kernel: lowmem_reserve[]: 0 442 442 442
Mar 21 09:54:54 ip-172-26-5-41 kernel: Node 0 DMA32 free:2600kB min:2644kB low:3304kB high:3964kB active_anon:404796kB inactive_anon:252kB active_file:336kB inactive_file:356kB unevictable:0kB writepending:0kB present:507904kB managed:474908kB mlocked:0kB kernel_stack:3356kB pagetables:15528kB bounce:0kB free_pcp:244kB local_pcp:244kB free_cma:0kB
Mar 21 09:54:54 ip-172-26-5-41 kernel: lowmem_reserve[]: 0 0 0 0
Mar 21 09:54:54 ip-172-26-5-41 kernel: Node 0 DMA: 754kB (UM) 298kB (UM) 4116kB (UM) 1332kB (U) 464kB (U) 0128kB 0256kB 0512kB 01024kB 02048kB 04096kB = 1860kB
Mar 21 09:54:54 ip-172-26-5-41 kernel: Node 0 DMA32: 5264kB (UME) 308kB (UME) 1016kB (UME) 332kB (UE) 064kB 0128kB 0256kB 0512kB 01024kB 02048kB 04096kB = 2600kB
Mar 21 09:54:54 ip-172-26-5-41 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Mar 21 09:54:54 ip-172-26-5-41 kernel: 291 total pagecache pages
Mar 21 09:54:54 ip-172-26-5-41 kernel: 0 pages in swap cache
Mar 21 09:54:54 ip-172-26-5-41 kernel: Swap cache stats: add 0, delete 0, find 0/0
Mar 21 09:54:54 ip-172-26-5-41 kernel: Free swap = 0kB
Mar 21 09:54:54 ip-172-26-5-41 kernel: Total swap = 0kB
Mar 21 09:54:54 ip-172-26-5-41 kernel: 130973 pages RAM
Mar 21 09:54:54 ip-172-26-5-41 kernel: 0 pages HighMem/MovableOnly
Mar 21 09:54:54 ip-172-26-5-41 kernel: 8270 pages reserved
Mar 21 09:54:55 ip-172-26-5-41 kernel: 0 pages hwpoisoned
Mar 21 09:54:55 ip-172-26-5-41 kernel: [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
.....
Mar 21 09:54:55 ip-172-26-5-41 kernel: [18966] 0 18966 65708 4452 37 4 0 0 servalagt
.....
Mar 21 09:54:55 ip-172-26-5-41 kernel: [19000] 0 19000 60452 271 72 4 0 0 sudo
Mar 21 09:54:55 ip-172-26-5-41 kernel: [19001] 0 19001 60452 271 71 4 0 0 sudo
Mar 21 09:54:55 ip-172-26-5-41 kernel: [19002] 0 19002 60452 271 70 3 0 0 sudo
Mar 21 09:54:55 ip-172-26-5-41 kernel: [19003] 0 19003 45704 3932 33 4 0 0 servalagt
Mar 21 09:54:55 ip-172-26-5-41 kernel: [19004] 0 19004 45704 3947 33 4 0 0 servalagt
Mar 21 09:54:55 ip-172-26-5-41 kernel: [19005] 0 19005 45704 3939 33 3 0 0 servalagt
Mar 21 09:54:55 ip-172-26-5-41 kernel: [19006] 0 19006 45704 3932 32 3 0 0 servalagt
Mar 21 09:54:55 ip-172-26-5-41 kernel: [19007] 0 19007 45704 3922 31 3 0 0 servalagt
Mar 21 09:54:55 ip-172-26-5-41 kernel: [19008] 0 19008 45704 3965 32 3 0 0 servalagt
Mar 21 09:54:55 ip-172-26-5-41 kernel: [19009] 0 19009 45704 3939 31 4 0 0 servalagt
Mar 21 09:54:55 ip-172-26-5-41 kernel: [19010] 0 19010 45704 3940 32 3 0 0 servalagt
Mar 21 09:54:55 ip-172-26-5-41 kernel: [19011] 0 19011 45704 3944 31 3 0 0 servalagt
Mar 21 09:54:55 ip-172-26-5-41 kernel: [19012] 0 19012 45704 3937 32 3 0 0 servalagt
Mar 21 09:54:55 ip-172-26-5-41 kernel: [19013] 0 19013 45704 3939 32 4 0 0 servalagt
Mar 21 09:54:55 ip-172-26-5-41 kernel: [19014] 0 19014 45704 3924 33 3 0 0 servalagt
Mar 21 09:54:55 ip-172-26-5-41 kernel: [19015] 0 19015 45704 3941 33 3 0 0 servalagt
Mar 21 09:54:55 ip-172-26-5-41 kernel: [19016] 0 19016 45704 3934 31 3 0 0 servalagt
Mar 21 09:54:55 ip-172-26-5-41 kernel: [19017] 0 19017 45704 3936 33 4 0 0 servalagt
Mar 21 09:54:55 ip-172-26-5-41 kernel: [19018] 0 19018 45704 3931 31 3 0 0 servalagt
Mar 21 09:54:55 ip-172-26-5-41 kernel: [19019] 0 19019 45704 3945 31 3 0 0 servalagt
Mar 21 09:54:55 ip-172-26-5-41 kernel: [19020] 0 19020 45704 3935 32 3 0 0 servalagt
Mar 21 09:54:55 ip-172-26-5-41 kernel: [19021] 0 19021 45704 3943 31 3 0 0 servalagt
Mar 21 09:54:55 ip-172-26-5-41 kernel: [19022] 0 19022 45704 3923 33 4 0 0 servalagt
Mar 21 09:54:55 ip-172-26-5-41 kernel: [10737] 89 10737 22594 254 47 3 0 0 pickup
Mar 21 09:54:55 ip-172-26-5-41 kernel: [10744] 0 10744 28213 254 58 3 0 0 sshd
Mar 21 09:54:55 ip-172-26-5-41 kernel: [10745] 0 10745 28213 255 58 3 0 0 sshd
.....
Mar 21 09:54:55 ip-172-26-5-41 kernel: Out of memory: Kill process 18966 (servalagt) score 35 or sacrifice child
Mar 21 09:54:55 ip-172-26-5-41 kernel: Killed process 18966 (servalagt) total-vm:262832kB, anon-rss:17808kB, file-rss:0kB, shmem-rss:0kB
Mar 21 09:54:55 ip-172-26-5-41 systemd: servalagt.service: main process exited, code=killed, status=9/KILL
Mar 21 09:54:55 ip-172-26-5-41 systemd: Unit servalagt.service entered failed state.
Mar 21 09:54:55 ip-172-26-5-41 systemd: servalagt.service failed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@s-takeuchi s-takeuchi

Labels
bug
Projects
None yet
Milestone
Future
Development

No branches or pull requests
1 participant
@s-takeuchi