Segfault + seemingly crazy memory gobbling

[jnthn]

This program reliably segfaults for me:

.sub bb
    $P0 = new 'ByteBuffer'
    $P0 = 100
    $I0 = 0
  loop:
    $I1 = $I0 + 1
    $P0[$I0] = $I1
    inc $I0
    unless $I0 == 100 goto loop
    .return ($P0)
.end

.sub main :main
    $P0 = new 'ByteBuffer'
    $P0 = 1
    $P0[0] = 1
    $I0 = 0
loop:
    $P1 = 'bb'()
    $S0 = $P0.'get_string'('binary')
    $S1 = $P1.'get_string'('binary')
    $S0 = concat $S0, $S1
    $P0 = new 'ByteBuffer'
    $P0 = $S0
    inc $I0
    unless $I0 == 10000 goto loop
.end

It does so in the function move_one_buffer in string_gc.c

Also rather worrying is that - even though the longest buffer string it should ever construct is 10000 * 100 bytes (and I guess it segfaults before it gets there) - at the point it segfaults it has eaten 2.7 GB of RAM!

[cotto]

I'm not able to reproduce this segfault on master on x86_64/Ubuntu 11.10 with either an optimized or an unoptimized build. can you specify your platform?

[leto]

Which sha1/version and platform did this happen on? Can we get a backtrace?

[jnthn]

Happens with RELEASE_3_9_0. I've built with MSVC++ compiler, on Windows, 64-bit.

Also, try increasing the number of iterations. 5000 is not enough to trigger it on my box, for example.

[moritz]

Note that I also wasn't able to reproduce this segfault in pure PIR, but I could when doing the same in Rakudo. See https://rt.perl.org/rt3//Public/Bug/Display.html?id=101912 for the sample code. (64bit linux on RELEASE_3_9_0)

[jnthn]

Yes - I guess the extra bits of memory/GC activity in the Rakudo version bring out the issue. Note that it explodes in a different place in the Rakudo one than where I had this PIR version doing so.

[Whiteknight]

Can you reproduce the issue if you use something like a StringBuilder PMC instead of doing the raw string concats directly? Also, can you make the issue appear if you don't use the ByteBuffer PMC to setup the substrings?

I'm trying to narrow down possible causes. I'll try to look at this one today.

[jnthn]

Having another look, I think that at least the segfault is specific to ByteBuffer. It keeps track of the strstart pointer, however it appears that this may actually change over the lifetime of the string. That seems to be what move_one_buffer in string_gc.c is doing. The segfaults all look like memory corruption of some sort, and trying to access a buffer that got moved would probably cause that.

[NotFound]

I've been able to reproduce it by adding 'sweep 1' to the loop.

Yes, looks like ByteBuffer is depending in internal immutability of STRING, which has abandoned some time ago to allow some optimization in concat operations. Fix coming.

[NotFound]

Fixed for me in 65e6ab7, please verify.

[jnthn]

The good news: that ByteBuffer fix makes the Rakudo bug that was originally reported go away. :-)

The bad news: Something is still not right. The PIR example I have gave still segfaults during string heap compaction. Here's a full stack trace.

msvcr100.dll!memcpy()  Line 226
libparrot.dll!move_one_buffer(parrot_interp_t * interp, Memory_Block * pool, buffer_t * old_buf)  Line 958
libparrot.dll!move_buffer_callback(parrot_interp_t * interp, buffer_t * b, void * data)  Line 824
libparrot.dll!gc_gms_iterate_live_strings(parrot_interp_t * interp, void (parrot_interp_t *, buffer_t *, void *)* callback, void * data)  Line 1702 + 0x3d bytes
libparrot.dll!compact_pool(parrot_interp_t * interp, GC_Statistics * stats, Variable_Size_Pool * pool)  Line 788
libparrot.dll!gc_gms_mark_and_sweep(parrot_interp_t * interp, unsigned __int64 flags)  Line 855
libparrot.dll!gc_gms_allocate_pmc_header(parrot_interp_t * interp, unsigned __int64 flags)  Line 1412
libparrot.dll!Parrot_gc_new_pmc_header(parrot_interp_t * interp, unsigned __int64 flags)  Line 312 + 0xd bytes
libparrot.dll!get_new_pmc_header(parrot_interp_t * interp, __int64 base_type, unsigned __int64 flags)  Line 491 + 0xb bytes
libparrot.dll!Parrot_pmc_new(parrot_interp_t * interp, __int64 base_type)  Line 160
libparrot.dll!Parrot_new_p_sc(__int64 * cur_opcode, parrot_interp_t * interp)  Line 18761 + 0x8 bytes
libparrot.dll!runops_fast_core(parrot_interp_t * interp, runcore_t * runcore_unused, __int64 * pc)  Line 503 + 0x23 bytes
libparrot.dll!runops_int(parrot_interp_t * interp, unsigned __int64 offset)  Line 225
libparrot.dll!runops(parrot_interp_t * interp, unsigned __int64 offs)  Line 128
libparrot.dll!Parrot_pcc_invoke_from_sig_object(parrot_interp_t * interp, PMC * sub_obj, PMC * call_object)  Line 338
libparrot.dll!Parrot_pcc_invoke_sub_from_c_args(parrot_interp_t * interp, PMC * sub_obj, const char * sig, ...)  Line 140
libparrot.dll!Parrot_pf_execute_bytecode_program(parrot_interp_t * interp, PMC * pbc, PMC * args)  Line 2695
libparrot.dll!Parrot_api_run_bytecode(PMC * interp_pmc, PMC * pbc, PMC * args)  Line 162
parrot.exe!main(int argc, const char * * argv)  Line 172 + 0x4d bytes
parrot.exe!__tmainCRTStartup()  Line 555 + 0x19 bytes
kernel32.dll!0000000076d3652d()     
[Frames below may be incorrect and/or missing, no symbols loaded for kernel32.dll]  
ntdll.dll!0000000076e6c521()    

[NotFound]

One more fix added in 393e5e7, it should avoid negative effects of heap compaction.

The method get_chars is not fixed yet, and it probably has the same problem. It may be more complicated than the other cases because of the size in codepoints, I'm not sure about the correctness of partial_scan in corner cases,

[jnthn]

Tried it, but I still get the segfault in the same place with the original PIR program I posted. And it's swallowed 2.7 GB of RAM by then (very, very quickly, I should add).

[leto]

Can this snippet be written as a test to make it easy to, for
instance, run it on a bunch of different platforms, like the gcc
compile farm?

On Wed, Oct 26, 2011 at 9:48 AM, Jonathan Worthington
reply@reply.github.com
wrote:

Tried it, but I still get the segfault in the same place with the original PIR program I posted. And it's swallowed 2.7 GB of RAM by then (very, very quickly, I should add).

Reply to this email directly or view it on GitHub:
#182 (comment)

Jonathan "Duke" Leto jonathan@leto.net
Leto Labs LLC
209.691.DUKE // http://labs.leto.net
NOTE: Personal email is only checked twice a day at 10am/2pm PST,
please call/text for time-sensitive matters.

[leto]

@jnthn is this still a problem or can this issue be closed?

[coke]

I fail to segfault on OS X:

$ sw_vers
ProductName:    Mac OS X
ProductVersion: 10.8.2
BuildVersion:   12C60

Rev: 32ac823

[leto]

Closing this issue, please create another or reopen if this is still a problem.