runtime: garbage collection ineffective on 32-bit #909

Closed
peterGo opened this Issue Jul 8, 2010 · 83 comments

Comments

Projects
None yet
@peterGo
Contributor

peterGo commented Jul 8, 2010

GOARCH=386 Garbage Collection Is Ineffectual: mmap: errno=0xc

package main

import (
    "fmt"
    "runtime"
)

func fs() []float64 {
    r := make([]float64, 923521)
    return r
}

func main() {
    s := fs()
    for i := 0; i < 1000; i++ {
        s = fs()
        m := runtime.MemStats
        fmt.Printf("i %d; Alloc %d; TotalAlloc %d\n", i, m.Alloc, m.TotalAlloc)
    }
    m := runtime.MemStats
    fmt.Printf("end; Alloc %d; TotalAlloc %d\n", m.Alloc, m.TotalAlloc)
    _ = s
}

Expected results obtained with 6g, GOARCH=amd64, and GOOS=linux, for hg id 5af6f6656531
tip, using 6.0GB real memory:
i 0; Alloc 15086544; TotalAlloc 15141352
i 1; Alloc 15156656; TotalAlloc 23020488
i 2; Alloc 22545888; TotalAlloc 30798840
i 3; Alloc 30004752; TotalAlloc 38646824
i 4; Alloc 15295920; TotalAlloc 46520408
i 5; Alloc 22685152; TotalAlloc 54298760
i 6; Alloc 30074384; TotalAlloc 62077112
 . .
i 996; Alloc 30074384; TotalAlloc 7771093592
i 997; Alloc 15295920; TotalAlloc 7778897544
i 998; Alloc 22685152; TotalAlloc 7786675896
i 999; Alloc 30074384; TotalAlloc 7794454248
  end; Alloc 30074432; TotalAlloc 7794838296

Actual results obtained with 8g, GOARCH=386, and GOOS=linux, for hg id 5af6f6656531 tip,
using 0.5GB real memory:
i 0; Alloc 14975824; TotalAlloc 16730200
i 1; Alloc 22402248; TotalAlloc 24448472
i 2; Alloc 29828344; TotalAlloc 32166408
i 3; Alloc 37254440; TotalAlloc 39884344
i 4; Alloc 44680536; TotalAlloc 47602280
i 5; Alloc 44717256; TotalAlloc 55320216
i 6; Alloc 52106488; TotalAlloc 63001288
. . .
i 427; Alloc 2814607576; TotalAlloc 3310449848
i 428; Alloc 2822033672; TotalAlloc 3318167784
i 429; Alloc 2829459768; TotalAlloc 3325885720
i 430; Alloc 2836885864; TotalAlloc 3333603656
mmap: errno=0xc

For a complete description of the problem q.v. conjugate gradient method out of memory
http://groups.google.com/group/golang-nuts/browse_thread/thread/6fb3e3b7ae04d42a
@peterGo

This comment has been minimized.

Show comment
Hide comment
@peterGo

peterGo Jul 8, 2010

Contributor

Comment 1:

If you reduce the size of the allocation to a 10th or a 100th to 'make([]float64,
92352)' or 'make([]float64, 9235)', the garbage collector becomes effective.
Contributor

peterGo commented Jul 8, 2010

Comment 1:

If you reduce the size of the allocation to a 10th or a 100th to 'make([]float64,
92352)' or 'make([]float64, 9235)', the garbage collector becomes effective.
@adg

This comment has been minimized.

Show comment
Hide comment
@adg

adg Jul 9, 2010

Contributor

Comment 2:

Owner changed to r...@golang.org.

Status changed to Accepted.

Contributor

adg commented Jul 9, 2010

Comment 2:

Owner changed to r...@golang.org.

Status changed to Accepted.

@rsc

This comment has been minimized.

Show comment
Hide comment
@rsc

rsc Sep 10, 2010

Contributor

Comment 3:

Issue #1091 has been merged into this issue.

Contributor

rsc commented Sep 10, 2010

Comment 3:

Issue #1091 has been merged into this issue.

@adg

This comment has been minimized.

Show comment
Hide comment
@adg

adg Oct 19, 2010

Contributor

Comment 4:

Issue #1210 has been merged into this issue.

Contributor

adg commented Oct 19, 2010

Comment 4:

Issue #1210 has been merged into this issue.

@davecheney

This comment has been minimized.

Show comment
Hide comment
@davecheney

davecheney Apr 29, 2011

Contributor

Comment 5:

Hi Peter,
I ran your sample code on the current tip, a11092559c78, and it completed as expected on
8g. Is the issue still reproducible for you?
Cheers
Dave
Contributor

davecheney commented Apr 29, 2011

Comment 5:

Hi Peter,
I ran your sample code on the current tip, a11092559c78, and it completed as expected on
8g. Is the issue still reproducible for you?
Cheers
Dave
@rsc

This comment has been minimized.

Show comment
Hide comment
@rsc

rsc Apr 29, 2011

Contributor

Comment 6:

We've done some things to ameliorate the problem but it's not fixed.
Russ
Contributor

rsc commented Apr 29, 2011

Comment 6:

We've done some things to ameliorate the problem but it's not fixed.
Russ
@rsc

This comment has been minimized.

Show comment
Hide comment
@rsc

rsc Jun 7, 2011

Contributor

Comment 7:

Issue #1920 has been merged into this issue.

Contributor

rsc commented Jun 7, 2011

Comment 7:

Issue #1920 has been merged into this issue.

@peterGo

This comment has been minimized.

Show comment
Hide comment
@peterGo

peterGo Jun 7, 2011

Contributor

Comment 8:

Should issue #1925, rather than issue #1920, have been merged into this issue?
Contributor

peterGo commented Jun 7, 2011

Comment 8:

Should issue #1925, rather than issue #1920, have been merged into this issue?
@rsc

This comment has been minimized.

Show comment
Hide comment
@rsc

rsc Jun 7, 2011

Contributor

Comment 9:

It doesn't really matter.  1920 may be a dup of 1925 or it may be a dup of 909.
1925 is really a separate issue; such a trivial program should not hit 909.
Contributor

rsc commented Jun 7, 2011

Comment 9:

It doesn't really matter.  1920 may be a dup of 1925 or it may be a dup of 909.
1925 is really a separate issue; such a trivial program should not hit 909.
@FlorianUekermann

This comment has been minimized.

Show comment
Hide comment
@FlorianUekermann

FlorianUekermann Jul 7, 2011

Contributor

Comment 10:

package main
import "fmt"
func main() {
  for {
    vec := make([]float64, 1000000)
    fmt.Println(vec[0])
  }
}
This crashes on 386 linux with 8g from last weekly on a mashine with 1.6GB free memory:
runtime: memory allocated by OS not in usable range
runtime: out of memory: cannot allocate 8060928-byte block (533069824 in use)
throw: out of memory
Is this a dublicate of 909 or something else? Looks like 909 to me, but testcase is a
bit different, I'm not sure. If it is, can we expect it to get fixed anytime soon?,
since it renders go on 386 unusable for lots of very common usecases (at least for me).
Contributor

FlorianUekermann commented Jul 7, 2011

Comment 10:

package main
import "fmt"
func main() {
  for {
    vec := make([]float64, 1000000)
    fmt.Println(vec[0])
  }
}
This crashes on 386 linux with 8g from last weekly on a mashine with 1.6GB free memory:
runtime: memory allocated by OS not in usable range
runtime: out of memory: cannot allocate 8060928-byte block (533069824 in use)
throw: out of memory
Is this a dublicate of 909 or something else? Looks like 909 to me, but testcase is a
bit different, I'm not sure. If it is, can we expect it to get fixed anytime soon?,
since it renders go on 386 unusable for lots of very common usecases (at least for me).
@adg

This comment has been minimized.

Show comment
Hide comment
@adg

adg Jul 7, 2011

Contributor

Comment 11:

This is the same issue. But I'm confused why this is such a problem. Why would you want
to do so much allocation when (in this simple example, at least) it seems easily avoided?
Contributor

adg commented Jul 7, 2011

Comment 11:

This is the same issue. But I'm confused why this is such a problem. Why would you want
to do so much allocation when (in this simple example, at least) it seems easily avoided?
@FlorianUekermann

This comment has been minimized.

Show comment
Hide comment
@FlorianUekermann

FlorianUekermann Jul 8, 2011

Contributor

Comment 12:

@adg: You make it sound like a cornercase.. ...it's not. Allocating a vector with around
1.000.000 floats is not that uncommon if you process scientific data. One vector in this
example should be about 8Mb, the program crashed at 530Mb, thats only 70 vectors. I am
most certain "much" is a bit more than 70 vectors in some very common cases.
I really don't want to complain, I like go and the compilers, they are great on 64bit,
but this is a serious bug. I'm a bit surprised that you think that it isn't.
I'm still not convinced that it is the same bug though, since my problem is not only the
GC, but that the memory allocation fails when >1Gb free memory is available.
Contributor

FlorianUekermann commented Jul 8, 2011

Comment 12:

@adg: You make it sound like a cornercase.. ...it's not. Allocating a vector with around
1.000.000 floats is not that uncommon if you process scientific data. One vector in this
example should be about 8Mb, the program crashed at 530Mb, thats only 70 vectors. I am
most certain "much" is a bit more than 70 vectors in some very common cases.
I really don't want to complain, I like go and the compilers, they are great on 64bit,
but this is a serious bug. I'm a bit surprised that you think that it isn't.
I'm still not convinced that it is the same bug though, since my problem is not only the
GC, but that the memory allocation fails when >1Gb free memory is available.
@adg

This comment has been minimized.

Show comment
Hide comment
@adg

adg Jul 11, 2011

Contributor

Comment 13:

Sorry, I missed the part where you had more available physical memory than you were
trying to allocate. That seems like a bug that has nothing to do with the GC. File a
separate issue please.
Contributor

adg commented Jul 11, 2011

Comment 13:

Sorry, I missed the part where you had more available physical memory than you were
trying to allocate. That seems like a bug that has nothing to do with the GC. File a
separate issue please.
@rsc

This comment has been minimized.

Show comment
Hide comment
@rsc

rsc Dec 9, 2011

Contributor

Comment 14:

Labels changed: added priority-later.

Contributor

rsc commented Dec 9, 2011

Comment 14:

Labels changed: added priority-later.

@rsc

This comment has been minimized.

Show comment
Hide comment
@rsc

rsc Dec 12, 2011

Contributor

Comment 15:

Labels changed: added priority-go1.

Contributor

rsc commented Dec 12, 2011

Comment 15:

Labels changed: added priority-go1.

@robpike

This comment has been minimized.

Show comment
Hide comment
@robpike

robpike Jan 13, 2012

Contributor

Comment 16:

Owner changed to builder@golang.org.

Contributor

robpike commented Jan 13, 2012

Comment 16:

Owner changed to builder@golang.org.

@rsc

This comment has been minimized.

Show comment
Hide comment
@rsc

rsc Feb 19, 2012

Contributor

Comment 18:

Issue #1925 has been merged into this issue.

Contributor

rsc commented Feb 19, 2012

Comment 18:

Issue #1925 has been merged into this issue.

@rsc

This comment has been minimized.

Show comment
Hide comment
@rsc

rsc Feb 19, 2012

Contributor

Comment 19:

Nice test program from issue #1925:
package main
import (
    "fmt"
    "runtime"
    "net/http"
    _ "net/http/pprof"
)
var st runtime.MemStats
func main() {
    runtime.MemProfileRate = 1
    for i := 0; i < 10; i++ {
        a := make([]byte, 5000000)
        if a == nil {
        }
        a = nil
        runtime.GC()
        runtime.ReadMemStats(&st)
        fmt.Println(i, st.Alloc, st.Sys, st.HeapObjects)
    }
    fmt.Println()
    for i := 0; i < 10; i++ {
        a := make([]byte, 20000000)
        if a == nil {
        }
        a = nil
        runtime.GC()
        runtime.ReadMemStats(&st)
        fmt.Println(i, st.Alloc, st.Sys, st.HeapObjects)
    }
    http.ListenAndServe(":8000", nil)
}
I have a few changes to ameliorate the effect of large static tables for Go 1, but the
general problem remains.
Contributor

rsc commented Feb 19, 2012

Comment 19:

Nice test program from issue #1925:
package main
import (
    "fmt"
    "runtime"
    "net/http"
    _ "net/http/pprof"
)
var st runtime.MemStats
func main() {
    runtime.MemProfileRate = 1
    for i := 0; i < 10; i++ {
        a := make([]byte, 5000000)
        if a == nil {
        }
        a = nil
        runtime.GC()
        runtime.ReadMemStats(&st)
        fmt.Println(i, st.Alloc, st.Sys, st.HeapObjects)
    }
    fmt.Println()
    for i := 0; i < 10; i++ {
        a := make([]byte, 20000000)
        if a == nil {
        }
        a = nil
        runtime.GC()
        runtime.ReadMemStats(&st)
        fmt.Println(i, st.Alloc, st.Sys, st.HeapObjects)
    }
    http.ListenAndServe(":8000", nil)
}
I have a few changes to ameliorate the effect of large static tables for Go 1, but the
general problem remains.
@rsc

This comment has been minimized.

Show comment
Hide comment
@rsc

rsc Feb 19, 2012

Contributor

Comment 20:

This CL should fix the 'static tables cause confusion' problem, but once you remove the
static tables you quickly find that dynamic data can cause confusion too, sadly.
changeset:   12574:24411588e821
user:        Russ Cox <rsc@golang.org>
date:        Sun Feb 19 03:19:52 2012 -0500
summary:     gc, ld: tag data as no-pointers and allocate in separate section
Contributor

rsc commented Feb 19, 2012

Comment 20:

This CL should fix the 'static tables cause confusion' problem, but once you remove the
static tables you quickly find that dynamic data can cause confusion too, sadly.
changeset:   12574:24411588e821
user:        Russ Cox <rsc@golang.org>
date:        Sun Feb 19 03:19:52 2012 -0500
summary:     gc, ld: tag data as no-pointers and allocate in separate section
@rsc

This comment has been minimized.

Show comment
Hide comment
@rsc

rsc Feb 19, 2012

Contributor

Comment 21:

The rest of the issue will have to wait until after Go 1.
Or maybe all the 32-bit systems will be replaced by 64-bit ones.

Labels changed: added priority-later, removed priority-go1.

Owner changed to ---.

Contributor

rsc commented Feb 19, 2012

Comment 21:

The rest of the issue will have to wait until after Go 1.
Or maybe all the 32-bit systems will be replaced by 64-bit ones.

Labels changed: added priority-later, removed priority-go1.

Owner changed to ---.

@gopherbot

This comment has been minimized.

Show comment
Hide comment
@gopherbot

gopherbot Feb 29, 2012

Comment 22 by mstplbrg:

I stumbled upon this bug because a Go program of mine runs out of memory on my 386
machine :-(.
I tried the latest version of Go (go version says "go version weekly.2012-02-22
+ca9790d6a51a"), but the problem is still present. The output of the test program from
comment #19 is:
./test909 
0 5482240 9018492 684
1 10521560 14695548 705
2 15559640 20372604 707
3 20597720 26049660 709
4 25635800 31726716 711
5 30673880 37403772 713
6 35711960 43080828 715
7 40750040 48757884 717
8 45788120 54434940 719
9 50826200 60111996 721
0 65862696 82672764 723
1 85900592 105233532 731
2 105938224 127794300 733
3 125975856 150355068 735
4 146013488 172915836 737
5 166051120 195476604 739
6 186088752 218037372 741
7 206126384 240598140 743
8 226164016 263158908 745
9 246201648 285719676 747
Am I misunderstanding your message / the test program or does your fix not actually work?
Thanks.

Comment 22 by mstplbrg:

I stumbled upon this bug because a Go program of mine runs out of memory on my 386
machine :-(.
I tried the latest version of Go (go version says "go version weekly.2012-02-22
+ca9790d6a51a"), but the problem is still present. The output of the test program from
comment #19 is:
./test909 
0 5482240 9018492 684
1 10521560 14695548 705
2 15559640 20372604 707
3 20597720 26049660 709
4 25635800 31726716 711
5 30673880 37403772 713
6 35711960 43080828 715
7 40750040 48757884 717
8 45788120 54434940 719
9 50826200 60111996 721
0 65862696 82672764 723
1 85900592 105233532 731
2 105938224 127794300 733
3 125975856 150355068 735
4 146013488 172915836 737
5 166051120 195476604 739
6 186088752 218037372 741
7 206126384 240598140 743
8 226164016 263158908 745
9 246201648 285719676 747
Am I misunderstanding your message / the test program or does your fix not actually work?
Thanks.
@rsc

This comment has been minimized.

Show comment
Hide comment
@rsc

rsc Feb 29, 2012

Contributor

Comment 23:

Comment #19 does say "the general problem remains".
My changes helped, but they are not a complete fix.
Contributor

rsc commented Feb 29, 2012

Comment 23:

Comment #19 does say "the general problem remains".
My changes helped, but they are not a complete fix.
@gopherbot

This comment has been minimized.

Show comment
Hide comment
@gopherbot

gopherbot Feb 29, 2012

Comment 24 by raumzeitlabor:

I hate to ask this, but do you have a rough time frame for when this bug will be
adressed? Order of weeks, months, two-digit months?
I’m just asking because I really need a solution for this. I could live without it for
a few more weeks, but if it’s going to take longer than that, I’ll have to
reimplement my program in a different language (not meant as pressure, just a fact).

Comment 24 by raumzeitlabor:

I hate to ask this, but do you have a rough time frame for when this bug will be
adressed? Order of weeks, months, two-digit months?
I’m just asking because I really need a solution for this. I could live without it for
a few more weeks, but if it’s going to take longer than that, I’ll have to
reimplement my program in a different language (not meant as pressure, just a fact).
@rsc

This comment has been minimized.

Show comment
Hide comment
@rsc

rsc Feb 29, 2012

Contributor

Comment 25:

My guess would be 2-digit months.  My suggestion would be to
try to arrange to run on a 64-bit machine instead of reimplementing
your entire program.  Sorry.
Contributor

rsc commented Feb 29, 2012

Comment 25:

My guess would be 2-digit months.  My suggestion would be to
try to arrange to run on a 64-bit machine instead of reimplementing
your entire program.  Sorry.
@gopherbot

This comment has been minimized.

Show comment
Hide comment
@gopherbot

gopherbot Feb 29, 2012

Comment 26 by raumzeitlabor:

Alright, thanks for your answer. I’ll have to reimplement because my budget doesn’t
allow switching to a new machine :).

Comment 26 by raumzeitlabor:

Alright, thanks for your answer. I’ll have to reimplement because my budget doesn’t
allow switching to a new machine :).
@gopherbot

This comment has been minimized.

Show comment
Hide comment
@gopherbot

gopherbot Feb 29, 2012

Comment 27 by pcrosby:

We're having the same problem as in comment #10...We've had to develop several
workarounds (including restarting a long-running app when approaching the 512MB
allocation limit, which is far from ideal).  I'd love it if this was fixed as moving all
our servers to 64 bit isn't an affordable option at the moment.

Comment 27 by pcrosby:

We're having the same problem as in comment #10...We've had to develop several
workarounds (including restarting a long-running app when approaching the 512MB
allocation limit, which is far from ideal).  I'd love it if this was fixed as moving all
our servers to 64 bit isn't an affordable option at the moment.
@gopherbot

This comment has been minimized.

Show comment
Hide comment
@gopherbot

gopherbot Mar 3, 2012

Comment 28 by mstplbrg:

FYI: I was able to solve this problem (in my case) by using "coffer": I put up a fixed
version at http://github.com/mstap/coffer, forked from http://github.com/mcgoo/coffer.
I just swapped all my buffers which used buffer := bytes.NewBuffer(make([]byte, 0,
readBufferBytes)) before with buffer, _ := coffer.NewMemCoffer(readBufferBytes+1). My
old buffer.Reset() becomes buffer.Seek(0, 0), but apart from that, coffer was a drop-in
replacement in my case. Note that buffer.Close() free()s the memory and *needs* to be
called.
This is not a beautiful solution, but it gets the job done until this problem is
properly fixed.

Comment 28 by mstplbrg:

FYI: I was able to solve this problem (in my case) by using "coffer": I put up a fixed
version at http://github.com/mstap/coffer, forked from http://github.com/mcgoo/coffer.
I just swapped all my buffers which used buffer := bytes.NewBuffer(make([]byte, 0,
readBufferBytes)) before with buffer, _ := coffer.NewMemCoffer(readBufferBytes+1). My
old buffer.Reset() becomes buffer.Seek(0, 0), but apart from that, coffer was a drop-in
replacement in my case. Note that buffer.Close() free()s the memory and *needs* to be
called.
This is not a beautiful solution, but it gets the job done until this problem is
properly fixed.
@gopherbot

This comment has been minimized.

Show comment
Hide comment
@gopherbot

gopherbot Mar 12, 2012

Comment 29 by zhigangc:

I spent some time digging into this issue and have found the following:
1. GC is Ineffectual because some memory blocks in the GO's heap can be mistakenly
marked as referenced, but in fact they are not.
2. The "fake" references are from the static variables defined in various go packages.
These variables are not pointers. However, as GC scans the data section for potential
references to the heap, they are treated as "pointers" and therefore the entire heap
blocks which these "pointers" happen to "reference" can never be reclaimed even when
they should be.
3. The attached test programs, mem.go can easily illustrate how these fake pointers
prevent GC from freeing used memory on both the tip (12661:426b1101b166) and r60.3 on
32-bit linux.
  To run the test programs, please unzip the attachment.
  To run it on Go 1, go to "go0" and run "go run mem.go". Make sure you are tip hash is 12661:426b1101b166, the most recent as of now.
  To run it on Go release, go to "go0" and run "make" and "./mem"
  In the unicode package, there are many static variables which end up being put in the DATA section.  As GC scans the data section at runtime/mgc0.c:648, it treats the variables are pointers and some happen to "point" to the memory blocks in the heap.
  If we comment out the unicode package, GC works and the program runs fine.
4. The issue are more likely to crash applications which allocate memory in large chunks
because one "fake" pointer can hold a large piece of memory and it does not take a lot
of fake pointers to make the app run out of RAM. If allocated in small chunks, the
problem still persist, though much less severe, and often gets away unnoticed.
5. I am suspecting, the issue potentially exists in 64 bit as well.

Attachments:

  1. issue_909.zip (1645 bytes)

Comment 29 by zhigangc:

I spent some time digging into this issue and have found the following:
1. GC is Ineffectual because some memory blocks in the GO's heap can be mistakenly
marked as referenced, but in fact they are not.
2. The "fake" references are from the static variables defined in various go packages.
These variables are not pointers. However, as GC scans the data section for potential
references to the heap, they are treated as "pointers" and therefore the entire heap
blocks which these "pointers" happen to "reference" can never be reclaimed even when
they should be.
3. The attached test programs, mem.go can easily illustrate how these fake pointers
prevent GC from freeing used memory on both the tip (12661:426b1101b166) and r60.3 on
32-bit linux.
  To run the test programs, please unzip the attachment.
  To run it on Go 1, go to "go0" and run "go run mem.go". Make sure you are tip hash is 12661:426b1101b166, the most recent as of now.
  To run it on Go release, go to "go0" and run "make" and "./mem"
  In the unicode package, there are many static variables which end up being put in the DATA section.  As GC scans the data section at runtime/mgc0.c:648, it treats the variables are pointers and some happen to "point" to the memory blocks in the heap.
  If we comment out the unicode package, GC works and the program runs fine.
4. The issue are more likely to crash applications which allocate memory in large chunks
because one "fake" pointer can hold a large piece of memory and it does not take a lot
of fake pointers to make the app run out of RAM. If allocated in small chunks, the
problem still persist, though much less severe, and often gets away unnoticed.
5. I am suspecting, the issue potentially exists in 64 bit as well.

Attachments:

  1. issue_909.zip (1645 bytes)
@gopherbot

This comment has been minimized.

Show comment
Hide comment
@gopherbot

gopherbot Mar 12, 2012

Comment 30 by zhigangc:

To make it more convient, I just paste the code here:
package main
import (
        "runtime"
        //comment the following line and the program runs fine.
        "unicode"
)
func fs() []byte {
        //allocate 64 MB chunks
        r := make([]byte, 64*1024*1024)
        return r
}
func main() {
        //comment the following line and the program runs fine.
        println("addr:", &unicode.Scripts)
        var s []byte
        for i := 0; i < 100; i++ {
                println("")
                println(i, "---------------")
                s = fs()
                runtime.GC()
                var m runtime.MemStats
                runtime.ReadMemStats(&m)
                println(i, "MemStats.Alloc:", m.Alloc)
        }
        _ = s
}

Comment 30 by zhigangc:

To make it more convient, I just paste the code here:
package main
import (
        "runtime"
        //comment the following line and the program runs fine.
        "unicode"
)
func fs() []byte {
        //allocate 64 MB chunks
        r := make([]byte, 64*1024*1024)
        return r
}
func main() {
        //comment the following line and the program runs fine.
        println("addr:", &unicode.Scripts)
        var s []byte
        for i := 0; i < 100; i++ {
                println("")
                println(i, "---------------")
                s = fs()
                runtime.GC()
                var m runtime.MemStats
                runtime.ReadMemStats(&m)
                println(i, "MemStats.Alloc:", m.Alloc)
        }
        _ = s
}
@gopherbot

This comment has been minimized.

Show comment
Hide comment
@gopherbot

gopherbot Mar 12, 2012

Comment 31 by matthewrsiegel@comcast.net:

excellent investigation!
if there were a hack to prevent gc from bothering with static data...

Comment 31 by matthewrsiegel@comcast.net:

excellent investigation!
if there were a hack to prevent gc from bothering with static data...
@gopherbot

This comment has been minimized.

Show comment
Hide comment
@gopherbot

gopherbot Mar 13, 2012

Comment 32 by zhigangc:

I was able to prove that the bug exists on 64 bit linux as well with r60.3. The two fake
pointers prevents GC from reclaiming the two pieces of memory blocks allocated in the
first two iterations of the loop. However, it may not work for you because setting two
fake pointers is tricky.
My argument is, this is essentially a bug that can cause huge memory leaks to both 64
bit and 32 bit servers.
Moderators, should I create a separate issue?
package main
import (
        "runtime"
)
var fake_pointer1 uint64 = 0xf84402d000
var fake_pointer2 uint64 = 0xf84000d000
func fs() []byte {
        //allocate 64 MB chunks
        r := make([]byte, 64*1024*1024)
        return r
}
func main() {
        println("addr:", &fake_pointer1, &fake_pointer2)
        var s []byte
        for i := 0; i < 10; i++ {
                println("")
                println(i, "---------------")
                s = fs()
                runtime.GC()
                m := runtime.MemStats
                println(i, "MemStats.Alloc:", m.Alloc)
        }
        _ = s
}

Comment 32 by zhigangc:

I was able to prove that the bug exists on 64 bit linux as well with r60.3. The two fake
pointers prevents GC from reclaiming the two pieces of memory blocks allocated in the
first two iterations of the loop. However, it may not work for you because setting two
fake pointers is tricky.
My argument is, this is essentially a bug that can cause huge memory leaks to both 64
bit and 32 bit servers.
Moderators, should I create a separate issue?
package main
import (
        "runtime"
)
var fake_pointer1 uint64 = 0xf84402d000
var fake_pointer2 uint64 = 0xf84000d000
func fs() []byte {
        //allocate 64 MB chunks
        r := make([]byte, 64*1024*1024)
        return r
}
func main() {
        println("addr:", &fake_pointer1, &fake_pointer2)
        var s []byte
        for i := 0; i < 10; i++ {
                println("")
                println(i, "---------------")
                s = fs()
                runtime.GC()
                m := runtime.MemStats
                println(i, "MemStats.Alloc:", m.Alloc)
        }
        _ = s
}
@ianlancetaylor

This comment has been minimized.

Show comment
Hide comment
@ianlancetaylor

ianlancetaylor Mar 13, 2012

Contributor

Comment 33:

Yes, that is the problem.  Sorry the analysis never got attached.  rsc's patch in
comment #20 partially ameliorates it, but as he says the generate problem remains. 
There is no simple fix.
Contributor

ianlancetaylor commented Mar 13, 2012

Comment 33:

Yes, that is the problem.  Sorry the analysis never got attached.  rsc's patch in
comment #20 partially ameliorates it, but as he says the generate problem remains. 
There is no simple fix.
@gopherbot

This comment has been minimized.

Show comment
Hide comment
@gopherbot

gopherbot Apr 6, 2012

Comment 34 by ezyang@mit.edu:

Here's what I don't understand: Go appears to keep track of some simple type
information, namely, whether or not a region may have pointers, or doesn't have
pointers. Why can't this be added to static data?

Comment 34 by ezyang@mit.edu:

Here's what I don't understand: Go appears to keep track of some simple type
information, namely, whether or not a region may have pointers, or doesn't have
pointers. Why can't this be added to static data?
@rsc

This comment has been minimized.

Show comment
Hide comment
@rsc

rsc Apr 6, 2012

Contributor

Comment 35:

We do track that for static data too, and pure-data (no pointer)
blocks do not get
scanned by the garbage collector.  That's what I did in the CL described by
comment #20.
But that's not enough.  All it takes is one unfortunate collision to pin a
pointer in memory that should not be, and that might be the root of some
large tree of allocations.  The chance of collision increases the more
memory you allocate, since more and more 32-bit patterns point into
allocated data.
The collector is structured so that it can keep per-word information about
what is a pointer and what is not, but we have not hooked that up to the
C compiler and the Go compiler yet, so we can't take advantage of that.
It is a known, unfortunate issue.
The best workaround I can suggest for the short term is to use a 64-bit machine.
Sorry.
Russ
Contributor

rsc commented Apr 6, 2012

Comment 35:

We do track that for static data too, and pure-data (no pointer)
blocks do not get
scanned by the garbage collector.  That's what I did in the CL described by
comment #20.
But that's not enough.  All it takes is one unfortunate collision to pin a
pointer in memory that should not be, and that might be the root of some
large tree of allocations.  The chance of collision increases the more
memory you allocate, since more and more 32-bit patterns point into
allocated data.
The collector is structured so that it can keep per-word information about
what is a pointer and what is not, but we have not hooked that up to the
C compiler and the Go compiler yet, so we can't take advantage of that.
It is a known, unfortunate issue.
The best workaround I can suggest for the short term is to use a 64-bit machine.
Sorry.
Russ
@gopherbot

This comment has been minimized.

Show comment
Hide comment
@gopherbot

gopherbot Apr 7, 2012

Comment 36 by xanium4332:

Is it actually feasible for Go to have a non conservative garbage collector? I.e. are
there any language design choices which make a practical realisation impossible? I
really like the Go language, and I understand the reasons for the current choice of GC,
but realistically, due to design choices and restrictions (Cgo etc..), are we going to
be stuck with this situation for the foreseeable future?
I really hope not, but currently this issue is barring me from promoting the use of Go
in my workplace...
Regards,

Comment 36 by xanium4332:

Is it actually feasible for Go to have a non conservative garbage collector? I.e. are
there any language design choices which make a practical realisation impossible? I
really like the Go language, and I understand the reasons for the current choice of GC,
but realistically, due to design choices and restrictions (Cgo etc..), are we going to
be stuck with this situation for the foreseeable future?
I really hope not, but currently this issue is barring me from promoting the use of Go
in my workplace...
Regards,
@gopherbot

This comment has been minimized.

Show comment
Hide comment
@gopherbot

gopherbot Apr 7, 2012

Comment 37:

It is feasible for Go to have a GC that is less conservative (more precise) than the
current GC.
Cgo isn't a restriction in this respect. Go ignores memory belonging to C, and if C
refers to a Go object then the programming convention is that Go needs to refer to the
object as well.

Comment 37:

It is feasible for Go to have a GC that is less conservative (more precise) than the
current GC.
Cgo isn't a restriction in this respect. Go ignores memory belonging to C, and if C
refers to a Go object then the programming convention is that Go needs to refer to the
object as well.
@bradfitz

This comment has been minimized.

Show comment
Hide comment
@bradfitz

bradfitz May 29, 2012

Member

Comment 38:

Issue #3673 has been merged into this issue.

Member

bradfitz commented May 29, 2012

Comment 38:

Issue #3673 has been merged into this issue.

@rsc

This comment has been minimized.

Show comment
Hide comment
@rsc

rsc Sep 12, 2012

Contributor

Comment 39:

Issue #2703 has been merged into this issue.

Contributor

rsc commented Sep 12, 2012

Comment 39:

Issue #2703 has been merged into this issue.

@rsc

This comment has been minimized.

Show comment
Hide comment
@rsc

rsc Sep 12, 2012

Contributor

Comment 40:

Issue #3900 has been merged into this issue.

Contributor

rsc commented Sep 12, 2012

Comment 40:

Issue #3900 has been merged into this issue.

@rsc

This comment has been minimized.

Show comment
Hide comment
@rsc

rsc Sep 12, 2012

Contributor

Comment 41:

Labels changed: added go1.1.

Contributor

rsc commented Sep 12, 2012

Comment 41:

Labels changed: added go1.1.

@gopherbot

This comment has been minimized.

Show comment
Hide comment
@gopherbot

gopherbot Jun 22, 2013

Comment 65 by trent@wireover.com:

That allays my concerns about this issue. Thank you for the detailed explanation.
I recall another issue about Go memory management - something about Go requiring a large
contiguous chunk of RAM when a program launches, which on 32-bit systems can fail in
certain circumstances. I can't seem to find updated info on this issue - is it still a
problem with Go 1.1? Can you characterize this issue? 
I'd like to determine if Go can be stable WRT memory running as a 32-bit Windows desktop
app (understanding that Go is intended as a 64-bit server language). Are there still any
memory-management concerns with this?

Comment 65 by trent@wireover.com:

That allays my concerns about this issue. Thank you for the detailed explanation.
I recall another issue about Go memory management - something about Go requiring a large
contiguous chunk of RAM when a program launches, which on 32-bit systems can fail in
certain circumstances. I can't seem to find updated info on this issue - is it still a
problem with Go 1.1? Can you characterize this issue? 
I'd like to determine if Go can be stable WRT memory running as a 32-bit Windows desktop
app (understanding that Go is intended as a 64-bit server language). Are there still any
memory-management concerns with this?
@ianlancetaylor

This comment has been minimized.

Show comment
Hide comment
@ianlancetaylor

ianlancetaylor Jun 22, 2013

Contributor

Comment 66:

In both Go 1 and Go 1.1, in 32-bit mode, the memory allocator starts with a chunk of
768MB and then asks for additional chunks of 256MB as they are needed.  The Go 1.1
allocator is slightly cleverer but not really noticeably so.
Unfortunately I know very little about Go running in the Windows environment; I don't
know if there are any memory issues that arise on Windows but not elsewhere.
Contributor

ianlancetaylor commented Jun 22, 2013

Comment 66:

In both Go 1 and Go 1.1, in 32-bit mode, the memory allocator starts with a chunk of
768MB and then asks for additional chunks of 256MB as they are needed.  The Go 1.1
allocator is slightly cleverer but not really noticeably so.
Unfortunately I know very little about Go running in the Windows environment; I don't
know if there are any memory issues that arise on Windows but not elsewhere.
@alexbrainman

This comment has been minimized.

Show comment
Hide comment
@alexbrainman

alexbrainman Jun 24, 2013

Member

Comment 67:

trent@wireover.com,
> I recall another issue about Go memory management - something about Go requiring a
large contiguous chunk of RAM when a program launches, which on 32-bit systems can fail
in certain circumstances. I can't seem to find updated info on this issue - is it still
a problem with Go 1.1? Can you characterize this issue? 
I think you are talking about https://golang.org/issue/2323.
Looking at the issue again, Go used to reserve one large block of about 800MB memory.
Checking a recently built Go program (with vmmap), I can see it reserves 2 large blocks
of about 250MB and 500MB instead now. So, that should be an improvement here.
> ... Are there still any memory-management concerns with this?
There is also issue of Go not returning committed memory back to Windows
https://golang.org/issue/4960 and
https://golang.org/issue/5584. It is a big issue, as far as I am
concerned, if your program requires "a lot" of memory occasionally, because once it has
been allocated, it won't get released back to OS until your process will exit. So your
whole system will suffer. If your process is short lived, or you don't require excessive
amounts of memory you should be OK.
Alex
Member

alexbrainman commented Jun 24, 2013

Comment 67:

trent@wireover.com,
> I recall another issue about Go memory management - something about Go requiring a
large contiguous chunk of RAM when a program launches, which on 32-bit systems can fail
in certain circumstances. I can't seem to find updated info on this issue - is it still
a problem with Go 1.1? Can you characterize this issue? 
I think you are talking about https://golang.org/issue/2323.
Looking at the issue again, Go used to reserve one large block of about 800MB memory.
Checking a recently built Go program (with vmmap), I can see it reserves 2 large blocks
of about 250MB and 500MB instead now. So, that should be an improvement here.
> ... Are there still any memory-management concerns with this?
There is also issue of Go not returning committed memory back to Windows
https://golang.org/issue/4960 and
https://golang.org/issue/5584. It is a big issue, as far as I am
concerned, if your program requires "a lot" of memory occasionally, because once it has
been allocated, it won't get released back to OS until your process will exit. So your
whole system will suffer. If your process is short lived, or you don't require excessive
amounts of memory you should be OK.
Alex
@gopherbot

This comment has been minimized.

Show comment
Hide comment
@gopherbot

gopherbot Jun 25, 2013

Comment 68 by trent@wireover.com:

That's very useful info guys, thank you. Keep up the good work - Go has a bright future.

Comment 68 by trent@wireover.com:

That's very useful info guys, thank you. Keep up the good work - Go has a bright future.
@gopherbot

This comment has been minimized.

Show comment
Hide comment
@gopherbot

gopherbot Jul 12, 2013

Comment 69 by Cevian:

The make function also holds a static pointer to the last thing it alloced giving weird
result in memory statistics:
package main
import (
    "runtime"
)
func fs(sz int) []byte {
    //allocate sz MB chunks
    r := make([]byte, sz*1024*1024)
    return r
}
func printMemStats(i int) {
    var m runtime.MemStats
    runtime.GC()
    runtime.ReadMemStats(&m)
    println(i, "MemStats.Alloc:", m.Alloc/(1024*1024))
}
func clearMake() {
    r := make([]int, 1)
    _ = r
}
func main() {
    var s []byte
    var t []byte
    s = fs(64)
    printMemStats(1)
    s = nil
    printMemStats(2)
    clearMake()
    printMemStats(3)
    t = fs(32)
    printMemStats(4)
    s = fs(64)
    printMemStats(5)
    _ = s
    _ = t
}
Output:
1 MemStats.Alloc: 64
2 MemStats.Alloc: 64
3 MemStats.Alloc: 0
4 MemStats.Alloc: 32
5 MemStats.Alloc: 96
Expected:
1 MemStats.Alloc: 64
2 MemStats.Alloc: 0
3 MemStats.Alloc: 0
4 MemStats.Alloc: 32
5 MemStats.Alloc: 96

Comment 69 by Cevian:

The make function also holds a static pointer to the last thing it alloced giving weird
result in memory statistics:
package main
import (
    "runtime"
)
func fs(sz int) []byte {
    //allocate sz MB chunks
    r := make([]byte, sz*1024*1024)
    return r
}
func printMemStats(i int) {
    var m runtime.MemStats
    runtime.GC()
    runtime.ReadMemStats(&m)
    println(i, "MemStats.Alloc:", m.Alloc/(1024*1024))
}
func clearMake() {
    r := make([]int, 1)
    _ = r
}
func main() {
    var s []byte
    var t []byte
    s = fs(64)
    printMemStats(1)
    s = nil
    printMemStats(2)
    clearMake()
    printMemStats(3)
    t = fs(32)
    printMemStats(4)
    s = fs(64)
    printMemStats(5)
    _ = s
    _ = t
}
Output:
1 MemStats.Alloc: 64
2 MemStats.Alloc: 64
3 MemStats.Alloc: 0
4 MemStats.Alloc: 32
5 MemStats.Alloc: 96
Expected:
1 MemStats.Alloc: 64
2 MemStats.Alloc: 0
3 MemStats.Alloc: 0
4 MemStats.Alloc: 32
5 MemStats.Alloc: 96
@davecheney

This comment has been minimized.

Show comment
Hide comment
@davecheney

davecheney Jul 13, 2013

Contributor

Comment 70:

@Cevian, could you please open a new issue (issue #909 is too long) with your code sample.
Contributor

davecheney commented Jul 13, 2013

Comment 70:

@Cevian, could you please open a new issue (issue #909 is too long) with your code sample.
@rsc

This comment has been minimized.

Show comment
Hide comment
@rsc

rsc Jul 13, 2013

Contributor

Comment 71:

Labels changed: added restrict-addissuecomment-commit.

Contributor

rsc commented Jul 13, 2013

Comment 71:

Labels changed: added restrict-addissuecomment-commit.

@rsc

This comment has been minimized.

Show comment
Hide comment
@rsc

rsc Jul 30, 2013

Contributor

Comment 72:

Labels changed: added feature.

Contributor

rsc commented Jul 30, 2013

Comment 72:

Labels changed: added feature.

@robpike

This comment has been minimized.

Show comment
Hide comment
@robpike

robpike Aug 15, 2013

Contributor

Comment 73:

work is underway but it won't all be done by 1.2

Labels changed: added go1.3, removed go1.2maybe.

Contributor

robpike commented Aug 15, 2013

Comment 73:

work is underway but it won't all be done by 1.2

Labels changed: added go1.3, removed go1.2maybe.

@robpike

This comment has been minimized.

Show comment
Hide comment
@robpike

robpike Aug 20, 2013

Contributor

Comment 74:

Labels changed: removed go1.3.

Contributor

robpike commented Aug 20, 2013

Comment 74:

Labels changed: removed go1.3.

@rsc

This comment has been minimized.

Show comment
Hide comment
@rsc

rsc Nov 14, 2013

Contributor

Comment 75:

Issue #6756 has been merged into this issue.

Contributor

rsc commented Nov 14, 2013

Comment 75:

Issue #6756 has been merged into this issue.

@dvyukov

This comment has been minimized.

Show comment
Hide comment
@dvyukov

dvyukov Nov 14, 2013

Member

Comment 76:

We need some story for testing GC precision (both progress and regressions). Good test
cases are invaluable, because usually people do not notice moderate leaks and do not
dig. We need to look through the merged issues and commit them as tests.
Member

dvyukov commented Nov 14, 2013

Comment 76:

We need some story for testing GC precision (both progress and regressions). Good test
cases are invaluable, because usually people do not notice moderate leaks and do not
dig. We need to look through the merged issues and commit them as tests.
@rsc

This comment has been minimized.

Show comment
Hide comment
@rsc

rsc Nov 27, 2013

Contributor

Comment 77:

Labels changed: added go1.3maybe.

Contributor

rsc commented Nov 27, 2013

Comment 77:

Labels changed: added go1.3maybe.

@rsc

This comment has been minimized.

Show comment
Hide comment
@rsc

rsc Nov 27, 2013

Contributor

Comment 78:

Labels changed: removed feature.

Contributor

rsc commented Nov 27, 2013

Comment 78:

Labels changed: removed feature.

@rsc

This comment has been minimized.

Show comment
Hide comment
@rsc

rsc Dec 4, 2013

Contributor

Comment 79:

Labels changed: added release-none, removed go1.3maybe.

Contributor

rsc commented Dec 4, 2013

Comment 79:

Labels changed: added release-none, removed go1.3maybe.

@rsc

This comment has been minimized.

Show comment
Hide comment
@rsc

rsc Dec 4, 2013

Contributor

Comment 80:

Labels changed: added repo-main.

Contributor

rsc commented Dec 4, 2013

Comment 80:

Labels changed: added repo-main.

@randall77

This comment has been minimized.

Show comment
Hide comment
@randall77

randall77 Mar 5, 2014

Contributor

Comment 81:

Status update.  We've done a lot to improve the preciseness of the garbage collector for
1.3.  The major change is that scanning of Go stack frames is now completely precise. 
There are lots of minor changes, including scanning interfaces precisely, correct
context pointer scanning, and modifying reflect.Value to do its magic in a precise way.
We're not completely precise yet, but we're 99% there.  The major remaining piece is the
scanning of C stack frames.  A few minor pieces also remain, including scanning of some
runtime internal data structures.
Contributor

randall77 commented Mar 5, 2014

Comment 81:

Status update.  We've done a lot to improve the preciseness of the garbage collector for
1.3.  The major change is that scanning of Go stack frames is now completely precise. 
There are lots of minor changes, including scanning interfaces precisely, correct
context pointer scanning, and modifying reflect.Value to do its magic in a precise way.
We're not completely precise yet, but we're 99% there.  The major remaining piece is the
scanning of C stack frames.  A few minor pieces also remain, including scanning of some
runtime internal data structures.
@rsc

This comment has been minimized.

Show comment
Hide comment
@rsc

rsc Sep 18, 2014

Contributor

Comment 82:

We've done a lot over the last few months. C stack frames are gone, and the internal
data structures are described correctly now. The only piece I am aware of that is left
are a few C-declared data structures that the linker instructs the garbage collector to
scan conservatively. I think we can eliminate those for 1.4 and finally close this bug.
There is one other piece that I am not counting: if you use SWIG to allocate Go memory
from C++, that Go memory is scanned conservatively. That's a different problem (issue
6461) and not a concern for most Go programmers (since most don't use SWIG).

Labels changed: added release-go1.4, removed release-none.

Contributor

rsc commented Sep 18, 2014

Comment 82:

We've done a lot over the last few months. C stack frames are gone, and the internal
data structures are described correctly now. The only piece I am aware of that is left
are a few C-declared data structures that the linker instructs the garbage collector to
scan conservatively. I think we can eliminate those for 1.4 and finally close this bug.
There is one other piece that I am not counting: if you use SWIG to allocate Go memory
from C++, that Go memory is scanned conservatively. That's a different problem (issue
6461) and not a concern for most Go programmers (since most don't use SWIG).

Labels changed: added release-go1.4, removed release-none.

@gopherbot

This comment has been minimized.

Show comment
Hide comment
@gopherbot

gopherbot Sep 24, 2014

Comment 83:

CL https://golang.org/cl/149770043 mentions this issue.

Comment 83:

CL https://golang.org/cl/149770043 mentions this issue.
@rsc

This comment has been minimized.

Show comment
Hide comment
@rsc

rsc Sep 24, 2014

Contributor

Comment 84:

This issue was closed by revision 193daab.

Status changed to Fixed.

Contributor

rsc commented Sep 24, 2014

Comment 84:

This issue was closed by revision 193daab.

Status changed to Fixed.

@peterGo peterGo added fixed labels Sep 24, 2014

@adg adg referenced this issue Dec 8, 2014

Closed

memory leak on 8g #1210

@golang golang locked and limited conversation to collaborators Dec 8, 2014

@rsc rsc added this to the Go1.4 milestone Apr 14, 2015

@rsc rsc removed the release-go1.4 label Apr 14, 2015

wheatman added a commit to wheatman/go-akaros that referenced this issue Jun 25, 2018

cmd/cc, cmd/ld, runtime: disallow conservative data/bss objects
In linker, refuse to write conservative (array of pointers) as the
garbage collection type for any variable in the data/bss GC program.

In the linker, attach the Go type to an already-read C declaration
during dedup. This gives us Go types for C globals for free as long
as the cmd/dist-generated Go code contains the declaration.
(Most runtime C declarations have a corresponding Go declaration.
Both are bss declarations and so the linker dedups them.)

In cmd/dist, add a few more C files to the auto-Go-declaration list
in order to get Go type information for the C declarations into the linker.

In C compiler, mark all non-pointer-containing global declarations
and all string data as NOPTR. This allows them to exist in C files
without any corresponding Go declaration. Count C function pointers
as "non-pointer-containing", since we have no heap-allocated C functions.

In runtime, add NOPTR to the remaining pointer-containing declarations,
none of which refer to Go heap objects.

In runtime, also move os.Args and syscall.envs data into runtime-owned
variables. Otherwise, in programs that do not import os or syscall, the
runtime variables named os.Args and syscall.envs will be missing type
information.

I believe that this CL eliminates the final source of conservative GC scanning
in non-SWIG Go programs, and therefore...

Fixes #909.

LGTM=iant
R=iant
CC=golang-codereviews
https://golang.org/cl/149770043

wheatman added a commit to wheatman/go-akaros that referenced this issue Jun 26, 2018

cmd/cc, cmd/ld, runtime: disallow conservative data/bss objects
In linker, refuse to write conservative (array of pointers) as the
garbage collection type for any variable in the data/bss GC program.

In the linker, attach the Go type to an already-read C declaration
during dedup. This gives us Go types for C globals for free as long
as the cmd/dist-generated Go code contains the declaration.
(Most runtime C declarations have a corresponding Go declaration.
Both are bss declarations and so the linker dedups them.)

In cmd/dist, add a few more C files to the auto-Go-declaration list
in order to get Go type information for the C declarations into the linker.

In C compiler, mark all non-pointer-containing global declarations
and all string data as NOPTR. This allows them to exist in C files
without any corresponding Go declaration. Count C function pointers
as "non-pointer-containing", since we have no heap-allocated C functions.

In runtime, add NOPTR to the remaining pointer-containing declarations,
none of which refer to Go heap objects.

In runtime, also move os.Args and syscall.envs data into runtime-owned
variables. Otherwise, in programs that do not import os or syscall, the
runtime variables named os.Args and syscall.envs will be missing type
information.

I believe that this CL eliminates the final source of conservative GC scanning
in non-SWIG Go programs, and therefore...

Fixes #909.

LGTM=iant
R=iant
CC=golang-codereviews
https://golang.org/cl/149770043

wheatman added a commit to wheatman/go-akaros that referenced this issue Jul 9, 2018

cmd/cc, cmd/ld, runtime: disallow conservative data/bss objects
In linker, refuse to write conservative (array of pointers) as the
garbage collection type for any variable in the data/bss GC program.

In the linker, attach the Go type to an already-read C declaration
during dedup. This gives us Go types for C globals for free as long
as the cmd/dist-generated Go code contains the declaration.
(Most runtime C declarations have a corresponding Go declaration.
Both are bss declarations and so the linker dedups them.)

In cmd/dist, add a few more C files to the auto-Go-declaration list
in order to get Go type information for the C declarations into the linker.

In C compiler, mark all non-pointer-containing global declarations
and all string data as NOPTR. This allows them to exist in C files
without any corresponding Go declaration. Count C function pointers
as "non-pointer-containing", since we have no heap-allocated C functions.

In runtime, add NOPTR to the remaining pointer-containing declarations,
none of which refer to Go heap objects.

In runtime, also move os.Args and syscall.envs data into runtime-owned
variables. Otherwise, in programs that do not import os or syscall, the
runtime variables named os.Args and syscall.envs will be missing type
information.

I believe that this CL eliminates the final source of conservative GC scanning
in non-SWIG Go programs, and therefore...

Fixes #909.

LGTM=iant
R=iant
CC=golang-codereviews
https://golang.org/cl/149770043

This issue was closed.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.