OutOfMemoryError with GC-allocs

[dlangBugzillaToGithub]

a11e99z reported this on 2019-06-07T14:46:30Z

Transferred from https://issues.dlang.org/show_bug.cgi?id=19947

CC List

• Gregor Mückl
• ZombineDev
• Rainer Schuetze

Description

program bellow crashed

// dmd -m32 -release -O gctest.d
import std, core.memory;
void main() {
    long r = 0; // using results = dont drop code
    for(int i = 0; i < 1000; i++) {
        auto x = new int[10000000];
        r += x.sum();
    }
    write( r );
}

C:\content\downloadz\dlang>gctest.exe
core.exception.OutOfMemoryError@src\core\exception.d(702): Memory allocation failed
----------------

with -m64 no crashes but prog has more memory than 2GB. 
it cannot be false positive coz array contains all 0s (int.init)
imo something wrong with gc-roots or gc-marks.
code by LDC works fine.

[dlangBugzillaToGithub]

black80 commented on 2019-06-07T14:50:17Z

Windows Server 2019 (x64)
DMD 2.086.0
LDC 1.16.0-beta2

[dlangBugzillaToGithub]

dfj1esp02 commented on 2019-06-07T15:58:55Z

It can be false positive, see issue 15723

[dlangBugzillaToGithub]

black80 commented on 2019-06-07T21:30:19Z

maybe u right.
imo gc shouldnt scan all data segment for pointers:
when we assign to some globals some pointers or explicitly called GC.addRange/addRoot. and when I transfer something to C(or others)-libs.. well, its my responsibility only, not compiler or GC

[dlangBugzillaToGithub]

gregormueckl commented on 2019-06-08T17:40:28Z

I don't think that it can be false positives. Adding a GC.collect() call will clean up the allocations; so the issue must rather be that collection is not started for some reason. 

See also original forum discussion: https://forum.dlang.org/thread/obkpctudtnnfojrbrwiv@forum.dlang.org

[dlangBugzillaToGithub]

r.sagitario commented on 2019-06-12T20:35:51Z

To avoid false pointers in the heap and the data segment, use
--DRT-gcopt=gc:precise --DRT-scanDataSeg=precise

With these the program runs to completion.

[dlangBugzillaToGithub]

gregormueckl commented on 2019-06-21T08:16:14Z

(In reply to Rainer Schuetze from comment #5)
> To avoid false pointers in the heap and the data segment, use
> --DRT-gcopt=gc:precise --DRT-scanDataSeg=precise
> 
> With these the program runs to completion.

How does this square with the fact that calling GC.collect() inside the for loop also makes the program run to completion? Also, the array is initialized with 0, so any false false pointers would be invisible to the user code.

[dlangBugzillaToGithub]

r.sagitario commented on 2019-06-21T09:00:10Z

> How does this square with the fact that calling GC.collect()
> inside the for loop also makes the program run to completion?

When running GC.collect() after the allocation, you are changing the amount of memory that the GC overallocates before running the next collection (2 times the live memory at the time of the previous collection). This changes the probability of getting false pointers.

> Also, the array is initialized with 0, so any false false pointers
> would be invisible to the user code.

The content of the int[] array is not scanned, the false pointers in the data segment (and the stack, -gx might help) are the issue here. These can pin the large allocations. Address randomization (ASLR) makes it rather unpredictable, too.