Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Memory leaks (IO::Socket::Async?) – HTTP::UserAgent, Cro::HTTP::Client, … #1501

Open
AlexDaniel opened this Issue Feb 8, 2018 · 20 comments

Comments

Projects
None yet
8 participants
@AlexDaniel
Copy link
Member

AlexDaniel commented Feb 8, 2018

The task is simple: fetch a file from some URL without leaking significant amounts of memory.

Using HTTP::UserAgent

Code: https://gist.github.com/AlexDaniel/0cdb3211cfcae90c29da9330649171c2

Result:

103.25MiB maxrss – before doing anything!!
「4b32fcab2192054b82312cde4c235adda0b8ac4d.zst」
1742.917969MiB maxrss 
「eb3123e5e60fa1635ed1ee121cec1290c290044a.zst」
1754.53125MiB maxrss 
「f66f8be09c6023b3b53e32f7571ea536ad87b87f.zst」
1754.53125MiB maxrss 
「b2a3441749878e338b0861b14b3b9433cc902f42.zst」
1827.464844MiB maxrss 
「78980ed447cceff82f5efef16dbe9ee437aae809.zst」
1842.933594MiB maxrss 
「aa94ffc33712c6f3068c6472b79a74fa70aa7b2e.zst」
1882.15625MiB maxrss 
「a0a28432f54d608130f17247f9202f4c4939dfff.zst」
1882.15625MiB maxrss 

So for some reason it spikes up from 103 MiB to 1742 MiB, that by itself is unreasonable but in terms of this ticket is probably irrelevant. What is important is that maxrss keeps growing with almost every subsequent request.

This may be an issue in HTTP::UserAgent, so let's try some other module…

Using Cro::HTTP::Client

First run:

127.859375MiB maxrss – before doing anything!!
「4b32fcab2192054b82312cde4c235adda0b8ac4d.zst」
190.054688MiB maxrss 
「eb3123e5e60fa1635ed1ee121cec1290c290044a.zst」
Segmentation fault

Whoops… that didn't go too well. Curious minds are free to run that in a loop to hunt for this segfault, but we're interested in memory usage:

Second run (and code): https://gist.github.com/AlexDaniel/fbfba45c23bf0dc157f9a1c07cfdea7d

128.660156MiB maxrss – before doing anything!!
「4b32fcab2192054b82312cde4c235adda0b8ac4d.zst」
186.773438MiB maxrss 
「eb3123e5e60fa1635ed1ee121cec1290c290044a.zst」
206.277344MiB maxrss 
「f66f8be09c6023b3b53e32f7571ea536ad87b87f.zst」
212.207031MiB maxrss 
「b2a3441749878e338b0861b14b3b9433cc902f42.zst」
220.429688MiB maxrss 
「78980ed447cceff82f5efef16dbe9ee437aae809.zst」
235.613281MiB maxrss 
「aa94ffc33712c6f3068c6472b79a74fa70aa7b2e.zst」
242.066406MiB maxrss 
「a0a28432f54d608130f17247f9202f4c4939dfff.zst」
257.566406MiB maxrss

Still leaking.

@AlexDaniel

This comment has been minimized.

Copy link
Member Author

AlexDaniel commented Feb 8, 2018

Bonus: Using curl through Proc::Async (Success!?)

Code: https://gist.github.com/AlexDaniel/408489c0409b96ea6d0ee36cce6e584e

Output:

94.746094MiB maxrss – before doing anything!!
「4b32fcab2192054b82312cde4c235adda0b8ac4d.zst」
102.984375MiB maxrss 
「eb3123e5e60fa1635ed1ee121cec1290c290044a.zst」
110.632813MiB maxrss 
「f66f8be09c6023b3b53e32f7571ea536ad87b87f.zst」
115.234375MiB maxrss 
「b2a3441749878e338b0861b14b3b9433cc902f42.zst」
117.605469MiB maxrss 
「78980ed447cceff82f5efef16dbe9ee437aae809.zst」
117.839844MiB maxrss 
「aa94ffc33712c6f3068c6472b79a74fa70aa7b2e.zst」
117.902344MiB maxrss 
「a0a28432f54d608130f17247f9202f4c4939dfff.zst」
117.980469MiB maxrss 

Even if it leaks, it is pretty much unnoticeable given that it settles around 120MiB at the end. Note that this is a completely different issue.

@dogbert17

This comment has been minimized.

Copy link
Contributor

dogbert17 commented Feb 8, 2018

Managed to repro after many attempts. Note that the JIT was enabled:

dilbert@Linux-Mint18 ~ $ perl6-gdb-m scratch.pl6
================================================================================================
This is Rakudo Perl 6 running in the GNU debugger, which often allows the user to generate useful back-
traces to debug or report issues in Rakudo, the MoarVM backend or the currently running code.

This Rakudo version is 2018.01.143.g.8.ba.3.c.86 built on MoarVM version 2018.01.77.g.9.a.029.b.4,
running on linuxmint (18.3.Sylvia) / linux (4.10.0.38.generic)

Type `bt full` to generate a backtrace if applicable, type `q` to quit or `help` for help.
------------------------------------------------------------------------------------------------
Reading symbols from /home/dilbert/.rakudobrew/moar-master/install/bin/moar...done.
Starting program: /home/dilbert/.rakudobrew/moar-master/install/bin/moar --execname=/home/dilbert/.rakudobrew/bin/../moar-master/install/bin/perl6-gdb-m --libpath=/home/dilbert/.rakudobrew/moar-master/install/share/nqp/lib --libpath=/home/dilbert/.rakudobrew/moar-master/install/share/perl6/lib --libpath=/home/dilbert/.rakudobrew/moar-master/install/share/perl6/runtime /home/dilbert/.rakudobrew/moar-master/install/share/perl6/runtime/perl6.moarvm scratch.pl6
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff63ed700 (LWP 28935)]
127.085938MiB maxrss – before doing anything!!
[New Thread 0x7fffef068700 (LWP 28936)]
[New Thread 0x7fffee867700 (LWP 28937)]
[New Thread 0x7fffed9f9700 (LWP 28938)]
[New Thread 0x7fffecbd3700 (LWP 28939)]
[New Thread 0x7fffdffff700 (LWP 28940)]
[New Thread 0x7fffdf7fe700 (LWP 28941)]
「4b32fcab2192054b82312cde4c235adda0b8ac4d.zst」

Thread 5 "moar" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffed9f9700 (LWP 28938)]
0x00007ffff76e7165 in MVM_gc_root_add_frame_registers_to_worklist (tc=tc@entry=0x7fffe00f82b0, worklist=worklist@entry=0x7fffe437b770, frame=frame@entry=0x7fffe44c1ed8) at src/gc/roots.c:430
430	                    MVM_gc_worklist_add(tc, worklist, &frame->args[i].o);
(gdb) bt
#0  0x00007ffff76e7165 in MVM_gc_root_add_frame_registers_to_worklist (tc=tc@entry=0x7fffe00f82b0, worklist=worklist@entry=0x7fffe437b770, frame=frame@entry=0x7fffe44c1ed8) at src/gc/roots.c:430
#1  0x00007ffff76e7525 in MVM_gc_root_add_frame_roots_to_worklist (tc=0x7fffe00f82b0, worklist=0x7fffe437b770, cur_frame=0x7fffe44c1ed8) at src/gc/roots.c:389
#2  0x00007ffff76e84af in process_worklist (tc=tc@entry=0x7fffe00f82b0, worklist=worklist@entry=0x7fffe437b770, wtp=wtp@entry=0x7fffed9f89a0, gen=gen@entry=0 '\000') at src/gc/collect.c:341
#3  0x00007ffff76e85e4 in MVM_gc_collect (tc=0x7fffe00f82b0, what_to_do=<optimized out>, gen=gen@entry=0 '\000') at src/gc/collect.c:123
#4  0x00007ffff76e43cb in run_gc (tc=tc@entry=0x7fffe00f82b0, what_to_do=what_to_do@entry=0 '\000') at src/gc/orchestrate.c:369
#5  0x00007ffff76e4f73 in MVM_gc_enter_from_allocator (tc=tc@entry=0x7fffe00f82b0) at src/gc/orchestrate.c:486
#6  0x00007ffff76e5128 in MVM_gc_allocate_nursery (tc=0x7fffe00f82b0, size=40) at src/gc/allocation.c:32
#7  0x00007ffff76e536b in MVM_gc_allocate_object (tc=0x7fffe00f82b0, st=<optimized out>) at src/gc/allocation.c:86
#8  0x00007ffff76f9db3 in MVM_repr_alloc_init (tc=tc@entry=0x7fffe00f82b0, type=type@entry=0xc9da70) at src/6model/reprconv.c:17
#9  0x00007ffff76fb873 in MVM_repr_box_int (tc=0x7fffe00f82b0, type=0xc9da70, val=16384) at src/6model/reprconv.c:576
#10 0x00007ffff52b41e6 in p6box_i () from /home/dilbert/.rakudobrew/moar-master/install/share/perl6/runtime/dynext/libperl6_ops_moar.so
#11 0x00007fffec37e09c in ?? ()
#12 0x0000000000000008 in ?? ()
#13 0x00007fffcc1f87a0 in ?? ()
#14 0x00007ffff7a4f740 in CSWTCH.13 () from //home/dilbert/.rakudobrew/moar-master/nqp/MoarVM/../../install/lib/libmoar.so
#15 0x00007ffff76cee21 in remove_one_frame (unwind=1 '\001', tc=0x7fffed9f8c70) at src/core/frame.c:753
#16 MVM_frame_unwind_to (tc=0x7fffed9f8c70, tc@entry=0x7fffe00f82b0, frame=frame@entry=0x7fffcc1f87a0, abs_addr=0xc267a0 "", rel_addr=3759112880, rel_addr@entry=0, return_value=return_value@entry=0x0)
    at src/core/frame.c:952
#17 0x00007ffff76af5dd in run_handler (payload=0x7aa5f0, category=32, ex_obj=0x0, lh=..., tc=0x7fffe00f82b0) at src/core/exceptions.c:359
#18 MVM_exception_throwpayload (tc=0x7fffe00f82b0, mode=<optimized out>, cat=32, payload=0x7aa5f0, resume_result=<optimized out>) at src/core/exceptions.c:774
#19 0x00007ffff76bd589 in MVM_interp_run (tc=0xf, tc@entry=0x7fffe00f82b0, initial_invoke=0x1, invoke_data=0x7fffe424c898, invoke_data@entry=0x7fffe00f7b60) at src/core/interp.c:5729
#20 0x00007ffff76d263e in start_thread (data=0x7fffe00f7b60) at src/core/threads.c:85
#21 0x00007ffff6c026ba in start_thread (arg=0x7fffed9f9700) at pthread_create.c:333
#22 0x00007ffff722841d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
(gdb) info threads
  Id   Target Id         Frame 
  1    Thread 0x7ffff7fd9700 (LWP 28931) "moar" 0x00007ffff6c0a827 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x5ee7db0) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  2    Thread 0x7ffff63ed700 (LWP 28935) "moar" pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  3    Thread 0x7fffef068700 (LWP 28936) "moar" pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  4    Thread 0x7fffee867700 (LWP 28937) "moar" pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
* 5    Thread 0x7fffed9f9700 (LWP 28938) "moar" 0x00007ffff76e7165 in MVM_gc_root_add_frame_registers_to_worklist (tc=tc@entry=0x7fffe00f82b0, worklist=worklist@entry=0x7fffe437b770, 
    frame=frame@entry=0x7fffe44c1ed8) at src/gc/roots.c:430
  6    Thread 0x7fffecbd3700 (LWP 28939) "moar" pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  7    Thread 0x7fffdffff700 (LWP 28940) "moar" pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  8    Thread 0x7fffdf7fe700 (LWP 28941) "moar" pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
(gdb) p MVM_dump_backtrace(tc)
   at <unknown>:1  (/home/dilbert/.rakudobrew/moar-master/install/share/perl6/runtime/CORE.setting.moarvm:)
 from site#sources/2FC1AB545260676155D864FD85CA0BA1138DC1BA (Cro::HTTP::RawBodyParser):28  (/home/dilbert/.rakudobrew/moar-master/install/share/perl6/site/precomp/2D7757105A497B93FE7FEAF9E62BC3E1D6908D16.1518111450.73731/2F/2FC1AB545260676155D864FD85CA0BA1138DC1BA:)
 from SETTING::src/core/Supply.pm:1974  (/home/dilbert/.rakudobrew/moar-master/install/share/perl6/runtime/CORE.setting.moarvm:)
 from SETTING::src/core/Lock/Async.pm:170  (/home/dilbert/.rakudobrew/moar-master/install/share/perl6/runtime/CORE.setting.moarvm:run-with-updated-recursion-list)
 from SETTING::src/core/Lock/Async.pm:136  (/home/dilbert/.rakudobrew/moar-master/install/share/perl6/runtime/CORE.setting.moarvm:protect-or-queue-on-recursion)
 from SETTING::src/core/Supply.pm:1972  (/home/dilbert/.rakudobrew/moar-master/install/share/perl6/runtime/CORE.setting.moarvm:run-supply-code)
 from SETTING::src/core/Supply.pm:1915  (/home/dilbert/.rakudobrew/moar-master/install/share/perl6/runtime/CORE.setting.moarvm:)
 from SETTING::src/core/Supply.pm:249  (/home/dilbert/.rakudobrew/moar-master/install/share/perl6/runtime/CORE.setting.moarvm:)
 from SETTING::src/core/Supply.pm:214  (/home/dilbert/.rakudobrew/moar-master/install/share/perl6/runtime/CORE.setting.moarvm:)
 from SETTING::src/core/Lock/Async.pm:170  (/home/dilbert/.rakudobrew/moar-master/install/share/perl6/runtime/CORE.setting.moarvm:run-with-updated-recursion-list)
 from SETTING::src/core/Lock/Async.pm:136  (/home/dilbert/.rakudobrew/moar-master/install/share/perl6/runtime/CORE.setting.moarvm:protect-or-queue-on-recursion)
 from SETTING::src/core/Supply.pm:213  (/home/dilbert/.rakudobrew/moar-master/install/share/perl6/runtime/CORE.setting.moarvm:)
 from SETTING::src/core/Supply.pm:1556  (/home/dilbert/.rakudobrew/moar-master/install/share/perl6/runtime/CORE.setting.moarvm:)
 from SETTING::src/core/Supply.pm:1554  (/home/dilbert/.rakudobrew/moar-master/install/share/perl6/runtime/CORE.setting.moarvm:emit)
 from site#sources/7AA4846CE9534F85E0599D2B72E7B87B0027B049 (Cro::HTTP::ResponseParser):41  (/home/dilbert/.rakudobrew/moar-master/install/share/perl6/site/precomp/2D7757105A497B93FE7FEAF9E62BC3E1D6908D16.1518111450.73731/7A/7AA4846CE9534F85E0599D2B72E7B87B0027B049:)
 from site#sources/7AA4846CE9534F85E0599D2B72E7B87B0027B049 (Cro::HTTP::ResponseParser):39  (/home/dilbert/.rakudobrew/moar-master/install/share/perl6/site/precomp/2D7757105A497B93FE7FEAF9E62BC3E1D6908D16.1518111450.73731/7A/7AA4846CE9534F85E0599D2B72E7B87B0027B049:)
 from SETTING::src/core/Supply.pm:1974  (/home/dilbert/.rakudobrew/moar-master/install/share/perl6/runtime/CORE.setting.moarvm:)
 from SETTING::src/core/Lock/Async.pm:170  (/home/dilbert/.rakudobrew/moar-master/install/share/perl6/runtime/CORE.setting.moarvm:run-with-updated-recursion-list)
 from SETTING::src/core/Lock/Async.pm:136  (/home/dilbert/.rakudobrew/moar-master/install/share/perl6/runtime/CORE.setting.moarvm:protect-or-queue-on-recursion)
 from SETTING::src/core/Supply.pm:1972  (/home/dilbert/.rakudobrew/moar-master/install/share/perl6/runtime/CORE.setting.moarvm:run-supply-code)
 from SETTING::src/core/Supply.pm:1915  (/home/dilbert/.rakudobrew/moar-master/install/share/perl6/runtime/CORE.setting.moarvm:)
 from SETTING::src/core/Supply.pm:1870  (/home/dilbert/.rakudobrew/moar-master/install/share/perl6/runtime/CORE.setting.moarvm:)
 from SETTING::src/core/Supply.pm:1869  (/home/dilbert/.rakudobrew/moar-master/install/share/perl6/runtime/CORE.setting.moarvm:run-emit)
 from <unknown>:1  (/home/dilbert/.rakudobrew/moar-master/install/share/perl6/runtime/CORE.setting.moarvm:)
 from site#sources/C2D80F3820FA6CFD2C5267E9A7B3331ADE6636E9 (Cro::TLS):88  (/home/dilbert/.rakudobrew/moar-master/install/share/perl6/site/precomp/2D7757105A497B93FE7FEAF9E62BC3E1D6908D16.1518111450.73731/C2/C2D80F3820FA6CFD2C5267E9A7B3331ADE6636E9:)
 from SETTING::src/core/Supply.pm:1974  (/home/dilbert/.rakudobrew/moar-master/install/share/perl6/runtime/CORE.setting.moarvm:)
 from SETTING::src/core/Lock/Async.pm:170  (/home/dilbert/.rakudobrew/moar-master/install/share/perl6/runtime/CORE.setting.moarvm:run-with-updated-recursion-list)
 from SETTING::src/core/Lock/Async.pm:136  (/home/dilbert/.rakudobrew/moar-master/install/share/perl6/runtime/CORE.setting.moarvm:protect-or-queue-on-recursion)
 from SETTING::src/core/Supply.pm:1972  (/home/dilbert/.rakudobrew/moar-master/install/share/perl6/runtime/CORE.setting.moarvm:run-supply-code)
 from SETTING::src/core/Supply.pm:1915  (/home/dilbert/.rakudobrew/moar-master/install/share/perl6/runtime/CORE.setting.moarvm:)
 from SETTING::src/core/Supply.pm:1870  (/home/dilbert/.rakudobrew/moar-master/install/share/perl6/runtime/CORE.setting.moarvm:)
 from SETTING::src/core/Supply.pm:1869  (/home/dilbert/.rakudobrew/moar-master/install/share/perl6/runtime/CORE.setting.moarvm:run-emit)
 from <unknown>:1  (/home/dilbert/.rakudobrew/moar-master/install/share/perl6/runtime/CORE.setting.moarvm:)
 from SETTING::src/core/Channel.pm:113  (/home/dilbert/.rakudobrew/moar-master/install/share/perl6/runtime/CORE.setting.moarvm:)
 from SETTING::src/core/Supply.pm:1974  (/home/dilbert/.rakudobrew/moar-master/install/share/perl6/runtime/CORE.setting.moarvm:)
 from SETTING::src/core/Lock/Async.pm:170  (/home/dilbert/.rakudobrew/moar-master/install/share/perl6/runtime/CORE.setting.moarvm:run-with-updated-recursion-list)
 from SETTING::src/core/Lock/Async.pm:136  (/home/dilbert/.rakudobrew/moar-master/install/share/perl6/runtime/CORE.setting.moarvm:protect-or-queue-on-recursion)
 from SETTING::src/core/Supply.pm:1972  (/home/dilbert/.rakudobrew/moar-master/install/share/perl6/runtime/CORE.setting.moarvm:run-supply-code)
 from SETTING::src/core/Supply.pm:1915  (/home/dilbert/.rakudobrew/moar-master/install/share/perl6/runtime/CORE.setting.moarvm:)
 from SETTING::src/core/Supply.pm:391  (/home/dilbert/.rakudobrew/moar-master/install/share/perl6/runtime/CORE.setting.moarvm:)
 from SETTING::src/core/ThreadPoolScheduler.pm:232  (/home/dilbert/.rakudobrew/moar-master/install/share/perl6/runtime/CORE.setting.moarvm:)
 from SETTING::src/core/ThreadPoolScheduler.pm:230  (/home/dilbert/.rakudobrew/moar-master/install/share/perl6/runtime/CORE.setting.moarvm:run-one)
 from SETTING::src/core/ThreadPoolScheduler.pm:270  (/home/dilbert/.rakudobrew/moar-master/install/share/perl6/runtime/CORE.setting.moarvm:)
 from SETTING::src/core/Thread.pm:54  (/home/dilbert/.rakudobrew/moar-master/install/share/perl6/runtime/CORE.setting.moarvm:THREAD-ENTRY)
$1 = void
(gdb) 
@jnthn

This comment has been minimized.

Copy link
Member

jnthn commented Feb 8, 2018

That stack trace matches two others I've seen that are related to the current, known, expression JIT bug.

@MasterDuke17

This comment has been minimized.

Copy link
Contributor

MasterDuke17 commented Feb 9, 2018

Heaptrack info for a run of the HTTP::UserAgent version (with JIT disabled).

https://i.imgur.com/RDboVUa.png
https://i.imgur.com/PBq0ddn.png
https://i.imgur.com/oMzrcR8.png
https://i.imgur.com/FuV7uUb.png

@MasterDuke17

This comment has been minimized.

Copy link
Contributor

MasterDuke17 commented Feb 9, 2018

FWIW, i added some fprintfs to set_size_internal and a run of the HTTP::UserAgent version allocates almost 1k arrays of size greater than 5m.

@MasterDuke17

This comment has been minimized.

Copy link
Contributor

MasterDuke17 commented Feb 9, 2018

Backtraces at one of the 5m array size allocations.

$ gdb --args /home/dan/Source/perl6/install/bin/moar --libpath="/home/dan/Source/perl6/install/share/nqp/lib" --libpath="/home/dan/Source/perl6/install/share/nqp/lib" --libpath="/home/dan/Source/perl6/install/share/perl6/lib" --libpath="/home/dan/Source/perl6/install/share/perl6/runtime" /home/dan/Source/perl6/install/share/perl6/runtime/perl6.moarvm -I ./github/sergot/io-socket-ssl/lib,./github/sergot/http-useragent/lib,./github/sergot/perl6-encode/lib,./github/supernovus/perl6-http-status/lib,./github/perl6-community-modules/uri/lib,./github/perl6/Perl6-MIME-Base64/lib,./github/sergot/datetime-parse/lib,./github/perlpilot/p6-File-Temp/lib,./github/labster/p6-file-directory-tree/lib,./github/sergot/openssl/lib leak-some-memory.p6
GNU gdb (Ubuntu 8.0.1-0ubuntu1) 8.0.1
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/dan/Source/perl6/install/bin/moar...done.
(gdb) b VMArray.c:362
No source file named VMArray.c.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (VMArray.c:362) pending.
(gdb) r
Starting program: /home/dan/Source/perl6/install/bin/moar --libpath=/home/dan/Source/perl6/install/share/nqp/lib --libpath=/home/dan/Source/perl6/install/share/nqp/lib --libpath=/home/dan/Source/perl6/install/share/perl6/lib --libpath=/home/dan/Source/perl6/install/share/perl6/runtime /home/dan/Source/perl6/install/share/perl6/runtime/perl6.moarvm -I ./github/sergot/io-socket-ssl/lib,./github/sergot/http-useragent/lib,./github/sergot/perl6-encode/lib,./github/supernovus/perl6-http-status/lib,./github/perl6-community-modules/uri/lib,./github/perl6/Perl6-MIME-Base64/lib,./github/sergot/datetime-parse/lib,./github/perlpilot/p6-File-Temp/lib,./github/labster/p6-file-directory-tree/lib,./github/sergot/openssl/lib leak-some-memory.p6
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff62a5700 (LWP 27710)]
108.269531MiB maxrss – before doing anything!!

Thread 1 "moar" hit Breakpoint 1, set_size_internal (tc=0x555555758c40, body=0x7fffe92f25e4, n=5140144, repr_data=0x555556887460) at src/6model/reprs/VMArray.c:362
362         fprintf(stderr, "resiging array: new elems = %lu,\t\tnew size = %lu,\t\ttotal = %lu\n", elems, ssize, repr_data->elem_size*ssize);
(gdb) bt
#0  set_size_internal (tc=0x555555758c40, body=0x7fffe92f25e4, n=5140144, repr_data=0x555556887460) at src/6model/reprs/VMArray.c:362
#1  0x00007ffff765107d in set_elems (tc=0x555555758c40, st=0x5555568fa218, root=0x7fffe92f25cc, data=0x7fffe92f25e4, count=5140144) at src/6model/reprs/VMArray.c:466
#2  0x00007ffff75e1226 in MVM_interp_run (tc=0x555555758c40, initial_invoke=0x7ffff76fe4b3 <toplevel_initial_invoke>, invoke_data=0x5555557ccbb0) at src/core/interp.c:2382
#3  0x00007ffff76fe618 in MVM_vm_run_file (instance=0x555555758260, filename=0x7fffffffe0a1 "/home/dan/Source/perl6/install/share/perl6/runtime/perl6.moarvm") at src/moar.c:407
#4  0x00005555555557d7 in main (argc=9, argv=0x7fffffffdc08) at src/main.c:256
(gdb) c MVM_dump_backtrace(tc)
   at SETTING::src/core/Buf.pm:52  (/home/dan/Source/perl6/install/share/perl6/runtime/CORE.setting.moarvm:allocate)
 from SETTING::src/core/Buf.pm:50  (/home/dan/Source/perl6/install/share/perl6/runtime/CORE.setting.moarvm:allocate)
 from gen/moar/Metamodel.nqp:1798  (/home/dan/Source/perl6/install/share/nqp/lib/Perl6/Metamodel.moarvm:)
 from /home/dan/Source/perl6/modules/perl6-all-modules/github/sergot/openssl/lib/OpenSSL.pm6 (OpenSSL):217  (/home/dan/Source/perl6/modules/perl6-all-modules/github/sergot/openssl/lib/.precomp/56DFA1943271D478295725DF68F6EB3827C494D7.1518144305.03459/ED/ED7003A34FD1ABBAE7D2B35E80A0769DB2B14709:read)
 from /home/dan/Source/perl6/modules/perl6-all-modules/github/sergot/io-socket-ssl/lib/IO/Socket/SSL.pm6 (IO::Socket::SSL):107  (/home/dan/Source/perl6/modules/perl6-all-modules/github/sergot/openssl/lib/.precomp/56DFA1943271D478295725DF68F6EB3827C494D7.1518144305.03459/3B/3BCDC32F2483CAD15B7165D1DF140DA653A1A6D2:recv)
 from /home/dan/Source/perl6/modules/perl6-all-modules/github/sergot/http-useragent/lib/HTTP/UserAgent.pm6 (HTTP::UserAgent):201  (/home/dan/Source/perl6/modules/perl6-all-modules/github/sergot/http-useragent/lib/.precomp/56DFA1943271D478295725DF68F6EB3827C494D7.1518144305.03459/A4/A4C28B05728E4361F6FF69CA7C56CC19B0B99C54:get-content)
 from /home/dan/Source/perl6/modules/perl6-all-modules/github/sergot/http-useragent/lib/HTTP/UserAgent.pm6 (HTTP::UserAgent):195  (/home/dan/Source/perl6/modules/perl6-all-modules/github/sergot/http-useragent/lib/.precomp/56DFA1943271D478295725DF68F6EB3827C494D7.1518144305.03459/A4/A4C28B05728E4361F6FF69CA7C56CC19B0B99C54:get-content)
 from /home/dan/Source/perl6/modules/perl6-all-modules/github/sergot/http-useragent/lib/HTTP/UserAgent.pm6 (HTTP::UserAgent):313  (/home/dan/Source/perl6/modules/perl6-all-modules/github/sergot/http-useragent/lib/.precomp/56DFA1943271D478295725DF68F6EB3827C494D7.1518144305.03459/A4/A4C28B05728E4361F6FF69CA7C56CC19B0B99C54:)
 from /home/dan/Source/perl6/modules/perl6-all-modules/github/sergot/http-useragent/lib/HTTP/UserAgent.pm6 (HTTP::UserAgent):311  (/home/dan/Source/perl6/modules/perl6-all-modules/github/sergot/http-useragent/lib/.precomp/56DFA1943271D478295725DF68F6EB3827C494D7.1518144305.03459/A4/A4C28B05728E4361F6FF69CA7C56CC19B0B99C54:)
 from /home/dan/Source/perl6/modules/perl6-all-modules/github/sergot/http-useragent/lib/HTTP/UserAgent.pm6 (HTTP::UserAgent):294  (/home/dan/Source/perl6/modules/perl6-all-modules/github/sergot/http-useragent/lib/.precomp/56DFA1943271D478295725DF68F6EB3827C494D7.1518144305.03459/A4/A4C28B05728E4361F6FF69CA7C56CC19B0B99C54:get-response)
 from /home/dan/Source/perl6/modules/perl6-all-modules/github/sergot/http-useragent/lib/HTTP/UserAgent.pm6 (HTTP::UserAgent):159  (/home/dan/Source/perl6/modules/perl6-all-modules/github/sergot/http-useragent/lib/.precomp/56DFA1943271D478295725DF68F6EB3827C494D7.1518144305.03459/A4/A4C28B05728E4361F6FF69CA7C56CC19B0B99C54:request)
 from /home/dan/Source/perl6/modules/perl6-all-modules/github/sergot/http-useragent/lib/HTTP/UserAgent.pm6 (HTTP::UserAgent):102  (/home/dan/Source/perl6/modules/perl6-all-modules/github/sergot/http-useragent/lib/.precomp/56DFA1943271D478295725DF68F6EB3827C494D7.1518144305.03459/A4/A4C28B05728E4361F6FF69CA7C56CC19B0B99C54:get)
 from /home/dan/Source/perl6/modules/perl6-all-modules/github/sergot/http-useragent/lib/HTTP/UserAgent.pm6 (HTTP::UserAgent):98  (/home/dan/Source/perl6/modules/perl6-all-modules/github/sergot/http-useragent/lib/.precomp/56DFA1943271D478295725DF68F6EB3827C494D7.1518144305.03459/A4/A4C28B05728E4361F6FF69CA7C56CC19B0B99C54:get)
 from /home/dan/Source/perl6/modules/perl6-all-modules/github/sergot/http-useragent/lib/HTTP/UserAgent.pm6 (HTTP::UserAgent):105  (/home/dan/Source/perl6/modules/perl6-all-modules/github/sergot/http-useragent/lib/.precomp/56DFA1943271D478295725DF68F6EB3827C494D7.1518144305.03459/A4/A4C28B05728E4361F6FF69CA7C56CC19B0B99C54:get)
 from /home/dan/Source/perl6/modules/perl6-all-modules/github/sergot/http-useragent/lib/HTTP/UserAgent.pm6 (HTTP::UserAgent):98  (/home/dan/Source/perl6/modules/perl6-all-modules/github/sergot/http-useragent/lib/.precomp/56DFA1943271D478295725DF68F6EB3827C494D7.1518144305.03459/A4/A4C28B05728E4361F6FF69CA7C56CC19B0B99C54:get)
 from leak-some-memory.p6:6  (<ephemeral file>:foo)
 from leak-some-memory.p6:21  (<ephemeral file>:)
 from leak-some-memory.p6:20  (<ephemeral file>:<unit>)
 from leak-some-memory.p6:1  (<ephemeral file>:<unit-outer>)
 from gen/moar/stage2/NQPHLL.nqp:1537  (/home/dan/Source/perl6/install/share/nqp/lib/NQPHLL.moarvm:eval)
 from gen/moar/stage2/NQPHLL.nqp:1774  (/home/dan/Source/perl6/install/share/nqp/lib/NQPHLL.moarvm:evalfiles)
 from gen/moar/stage2/NQPHLL.nqp:1666  (/home/dan/Source/perl6/install/share/nqp/lib/NQPHLL.moarvm:command_eval)
 from src/Perl6/Compiler.nqp:42  (/home/dan/Source/perl6/install/share/nqp/lib/Perl6/Compiler.moarvm:command_eval)
 from gen/moar/stage2/NQPHLL.nqp:1625  (/home/dan/Source/perl6/install/share/nqp/lib/NQPHLL.moarvm:command_line)
 from gen/moar/main.nqp:47  (/home/dan/Source/perl6/install/share/perl6/runtime/perl6.moarvm:MAIN)
 from gen/moar/main.nqp:38  (/home/dan/Source/perl6/install/share/perl6/runtime/perl6.moarvm:<mainline>)
 from <unknown>:1  (/home/dan/Source/perl6/install/share/perl6/runtime/perl6.moarvm:<main>)
 from <unknown>:1  (/home/dan/Source/perl6/install/share/perl6/runtime/perl6.moarvm:<entry>)
Value can't be converted to integer.
(gdb)

AlexDaniel added a commit to perl6/whateverable that referenced this issue Feb 10, 2018

⚠ Make all bots runnable locally (this is so cool!)
Previously if you wanted to run bisectable or some other bot locally
you had to build tens (if not thousands) of rakudo versions to get
something useful out of it. Now whenever it stumbles upon a missing
build it will attempt to pull it from the main server. If you're on
linux x86_64, this means that the bots you start are no worse than the
ones running on the main server.

Note, however, that this does not let you *use* these bots as tools
locally. Well, unless you fire up your own IRC server and configure
the thing to use it. We will get this working next, for now this is a
huge step already (see #122).

Ah, almost forgot to mention, this does not work. See
rakudo/rakudo#1501. I mean, yeah, you can
probably use committable, evalable and some other bots, but bisectable
will run out of memory before you get to any useful result.
@AlexDaniel

This comment has been minimized.

Copy link
Member Author

AlexDaniel commented Feb 11, 2018

Just for the record, @MasterDuke17++ made some PRs to OpenSSL module to improve memory usage:

These PRs reduce maxrss but I believe do not affect the leak.

@AlexDaniel AlexDaniel referenced this issue Feb 11, 2018

Open

Make it easier to run for contributors #122

6 of 16 tasks complete
@bbkr

This comment has been minimized.

Copy link

bbkr commented Feb 15, 2019

I confirm that IO::Socket::Async leaks memory like crazy on Rakudo 2018.12.

client.pl

my $connection = IO::Socket::INET.new( host => 'localhost',  port => 3333 );

loop {
    $connection.put( 'ping' );
    my $status = $connection.get( );
    die unless $status eq 'pong'; # just checking...
}

server.pl

react {
    whenever IO::Socket::Async.listen('0.0.0.0', 3333) -> $connection {
        whenever $connection.Supply.lines -> $line {
            die unless $line eq 'ping'; # just checking...
            $connection.print( "pong\n" );
        }
    }
}

On my machine server.pl leaks 3MB of RSS memory per second. Making async sockets in Perl6 completely unusable for any long running processes. Awaiting for $connection.print Promise makes no difference. Using --optimize=0 also makes no difference.

@lizmat

This comment has been minimized.

Copy link
Contributor

lizmat commented Feb 15, 2019

@bbkr

This comment has been minimized.

Copy link

bbkr commented Feb 15, 2019

Yes, synchronous sockets (as in client.pl) are stable. Only asynchronous (as in server.pl) leak. Leak only occurs when data is transfered, so the faster the CPU and the more ping-pong cycles it can do per second the sooner memory is clogged.

Even your case - 10 MB per minute - can eat almost 1GB of RAM per hour, so the leak is quite severe.

BTW - I've also tested version without lines chained Supply, same result.

@bbkr

This comment has been minimized.

Copy link

bbkr commented Mar 22, 2019

Latest changes in Async code in eb8561b didn't help. IO::Socket::Async still leaks memory like crazy on 2019.03.

For example my server.pl example above stabilizes at 100MB when no clients are connected. Then when client starts to produce messages it growths by 120MB for every 10M of messages exchanged.

Are there any chances this will be addressed in 2019.04? I hate being pushy when it comes to development cycles, but this affects a lot of modules in the ecosystem (and quite important code in my $dayjob as well - this is broken for so long that we're considering dumping Perl6 and switching to Go :( ). Any help here will be greatly appreciated. Thanks!

@lizmat

This comment has been minimized.

Copy link
Contributor

lizmat commented Mar 22, 2019

FWIW, eb8561b was never intended to fix a leak, just reduce memory / nursery pressure

Did the recent malloc changes in Moar by @samcv have a positive effect? This should be in master inn rakudo now.

@AlexDaniel

This comment has been minimized.

Copy link
Member Author

AlexDaniel commented Mar 22, 2019

.oO( malloc changes were not by @samcv, right?

(Edit: no, I confused it with MoarVM/MoarVM@5e36339)

Anyway, @bbkr, if this gives you hope:

<AlexDaniel> jnthn: maybe #1501 (comment)
<AlexDaniel> pretty sure it also affects Cro, and I'm suffering because of leaks too…
<jnthn> AlexDaniel: Maybe next week; really hope to have more time then.

@MasterDuke17

This comment has been minimized.

Copy link
Contributor

MasterDuke17 commented Mar 23, 2019

I did some experiments with malloc_trim, but @samcv actually created the PR (MoarVM/MoarVM#1072), though it hasn't been merged yet.

@AlexDaniel

This comment has been minimized.

Copy link
Member Author

AlexDaniel commented Mar 23, 2019

That looks very promising.

(oops, closed by accident)

@AlexDaniel AlexDaniel closed this Mar 23, 2019

@AlexDaniel AlexDaniel reopened this Mar 23, 2019

@Kaiepi

This comment has been minimized.

Copy link
Contributor

Kaiepi commented Mar 30, 2019

The solution for this is OS-dependent.

For OpenBSD, using freezero instead of free wherever applicable may be the answer, but it's a huge job to write and I'm still researching better solutions. This is what happens when I run the Cro::HTTP::Client test using freezero in src/io/asyncsocket.c and src/strings/utf8.c:

153.445313MiB maxrss – before doing anything!!
「4b32fcab2192054b82312cde4c235adda0b8ac4d.zst」
186.664063MiB maxrss 
「eb3123e5e60fa1635ed1ee121cec1290c290044a.zst」
203.773438MiB maxrss 
「f66f8be09c6023b3b53e32f7571ea536ad87b87f.zst」
212.054688MiB maxrss 
「b2a3441749878e338b0861b14b3b9433cc902f42.zst」
219.449219MiB maxrss 
「78980ed447cceff82f5efef16dbe9ee437aae809.zst」
232.742188MiB maxrss 
「aa94ffc33712c6f3068c6472b79a74fa70aa7b2e.zst」
232.742188MiB maxrss 
「a0a28432f54d608130f17247f9202f4c4939dfff.zst」
232.742188MiB maxrss 

The leak may not happen on FreeBSD/NetBSD since they use jemalloc. I'll test this on my FreeBSD VPS once I have access to SSH again around the end of next week. If it doesn't happen, maybe MoarVM could use jemalloc instead of having to test and fix the leak for every supported OS.

@Kaiepi

This comment has been minimized.

Copy link
Contributor

Kaiepi commented Mar 31, 2019

I decided not to wait and built MoarVM to use jemalloc on my own machine. Can confirm the leak doesn't happen with it:

148.945313MiB maxrss – before doing anything!!
「4b32fcab2192054b82312cde4c235adda0b8ac4d.zst」
218.054688MiB maxrss 
「eb3123e5e60fa1635ed1ee121cec1290c290044a.zst」
218.054688MiB maxrss 
「f66f8be09c6023b3b53e32f7571ea536ad87b87f.zst」
220.722656MiB maxrss 
「b2a3441749878e338b0861b14b3b9433cc902f42.zst」
232.898438MiB maxrss 
「78980ed447cceff82f5efef16dbe9ee437aae809.zst」
245.289063MiB maxrss 
「aa94ffc33712c6f3068c6472b79a74fa70aa7b2e.zst」
245.289063MiB maxrss 
「a0a28432f54d608130f17247f9202f4c4939dfff.zst」
245.289063MiB maxrss 
@MasterDuke17

This comment has been minimized.

Copy link
Contributor

MasterDuke17 commented Apr 5, 2019

FWIW, I tested @bbkr's server/client examples with jemalloc LD_PRELOADed. Without jemalloc I get (apparently) unbounded memory growth in the server process. With jemalloc I get cycles of growing and shrinking, with the max never increasing.

@jnthn

This comment has been minimized.

Copy link
Member

jnthn commented Apr 5, 2019

I tried it out after fixing a couple of other leaks today, wondering if it might have helped this case too. Looking at top said no. But running it under massif for a while didn't show growing heap use (though I want to run it for longer to verify that). That'd support the various fragmentation theories, with alternative malloc libraries not being vulnerable to them and malloc_trim perhaps mitigating them.

I'm wondering if this could be due to the thread that does the initial allocation of the buffer not reliably being the one that frees it (note: I need to verify this claim, but it's feasible), and that not sitting too well with some malloc implementations. I know we had to put in mitigation for such producer/consumer scenarios in MoarVM's fixed size allocator in the past.

jnthn added a commit to MoarVM/MoarVM that referenced this issue Apr 8, 2019

Avoid unbounded growth of active async task list
This seems to resolve the leak in the client/server example code given
in rakudo/rakudo#1501.
@jnthn

This comment has been minimized.

Copy link
Member

jnthn commented Apr 8, 2019

Gathered more output over a longer time period. Spotted something. Fixed it in MoarVM/MoarVM@adefbe4. That seems to get rid of the leak in the client/server example from @bbkr.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.