Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: allow use of > 16 GB on 64-bit machines #2142

Closed
rsc opened this issue Aug 9, 2011 · 20 comments
Closed

runtime: allow use of > 16 GB on 64-bit machines #2142

rsc opened this issue Aug 9, 2011 · 20 comments
Assignees
Milestone

Comments

@rsc
Copy link
Contributor

@rsc rsc commented Aug 9, 2011

allocator should try to grow vm addr space reservation
@rsc
Copy link
Contributor Author

@rsc rsc commented Dec 9, 2011

Comment 1:

Labels changed: added priority-later, removed priority-medium.

@robpike
Copy link
Contributor

@robpike robpike commented Dec 12, 2011

Comment 2:

Issue #2177 has been merged into this issue.

@rsc
Copy link
Contributor Author

@rsc rsc commented Mar 26, 2012

Comment 4:

Issue #2385 has been merged into this issue.

@rsc
Copy link
Contributor Author

@rsc rsc commented Mar 26, 2012

@donovanhide
Copy link
Contributor

@donovanhide donovanhide commented Apr 16, 2012

Comment 6:

At some point I'm considering porting this project to go, but this issue is a blocker
for me. Ideally I'd like to be able to use as much RAM as a server has available,
typically 32 or 64GB in our current use cases.
https://github.com/mediastandardstrust/superfastmatch
@alberts
Copy link
Contributor

@alberts alberts commented Apr 16, 2012

Comment 7:

You can already do allocation outside of the size-limited Go heap by writing a bit of
stuff on top of a raw mmap syscall that uses uintptrs instead of syscall.Mmap's []byte
interface. You would then expose your big buffer as many smaller 2 GB []bytes. Not
ideal, but manageable. With the current GC (or any GC) you probably don't want many
gigabytes of small objects anyway.
The same can probably be done with Alloc (if that's the right name) on Windows.
@donovanhide
Copy link
Contributor

@donovanhide donovanhide commented Apr 16, 2012

Comment 8:

Hi,
thanks for the suggestion and leads! I'm currently using sparsetable from the sparsehash
project in the C++ version to manage up to 1<<32 pointers to variable length
blocks. The overhead per block is 2.36 bits plus 8 bytes for the pointer itself on
64-bit. It's effectively a huge hash table where the hashed value itself is discarded
with sparsetable being abused as a memory allocation manager! The hash table is slotted
by ranges to allow multi-threaded access. Obviously there is the overhead of the malloc
implementation as well (tried tcmalloc, faster-but eats memory!). 
Your suggestion would involve writing a custom memory allocator which does sound like a
more difficult proposition :) I wonder if there's any internal Google projects to do
with threadsafe/goroutinesafe sparse hash tables in Go? I suppose the requirement is
some kind of bitmapped memory allocation with GC. All memory allocators need to
deallocate memory at some point anyway, so the question is who is best to do it :)
Cheers,
Donovan.
@egonelbre
Copy link
Contributor

@egonelbre egonelbre commented Apr 28, 2012

Comment 9:

If you really need to be able to use TBs of memory, then there seems to be a temporary
fix as suggested by Russ (
https://groups.google.com/d/msg/golang-nuts/bzFaGbmCVMs/Fn8Fdv3-KIEJ ). Change the
memory limit by editing src/pkg/runtime/malloc.goc and changing '16LL<<30' to
something larger. (But this may cause a segv with gob.Encode or some other unknown
problems).
@kortschak
Copy link
Contributor

@kortschak kortschak commented Apr 29, 2012

Comment 10:

If you have a look at issue #2177, you'll see that Russ' suggestion does not work
unfortunately - even in the simplest of cases.
@kortschak
Copy link
Contributor

@kortschak kortschak commented Jun 10, 2012

Comment 11:

Further experimentation on this.
This change allows allocation and addressing of data beyond 16GB (in this case 64GB):
diff -r 920e9d1ffd1f src/pkg/runtime/malloc.goc
--- a/src/pkg/runtime/malloc.goc    Wed Mar 28 23:41:59 2012 +1100
+++ b/src/pkg/runtime/malloc.goc    Sun Jun 10 09:39:58 2012 +0930
@@ -305,7 +305,7 @@
        // allocate 15 GB before we get that far.
        //
        // If this fails we fall back to the 32 bit memory mechanism
-       arena_size = 16LL<<30;
+       arena_size = 16LL<<32;
        bitmap_size = arena_size / (sizeof(void*)*8/4);
        p = runtime·SysReserve((void*)(0x00f8ULL<<32), bitmap_size + arena_size);
    }
diff -r 920e9d1ffd1f src/pkg/runtime/malloc.h
--- a/src/pkg/runtime/malloc.h  Wed Mar 28 23:41:59 2012 +1100
+++ b/src/pkg/runtime/malloc.h  Sun Jun 10 09:39:58 2012 +0930
@@ -116,7 +116,7 @@
    // On 64-bit, we limit the arena to 16G, so 22 bits suffices.
    // On 32-bit, we don't bother limiting anything: 20 bits for 4G.
 #ifdef _64BIT
-   MHeapMap_Bits = 22,
+   MHeapMap_Bits = 24,
 #else
    MHeapMap_Bits = 20,
 #endif
It may well have bad interactions with garbage collection, but could be a work around in
the short term if that is not an issue.
@egonelbre
Copy link
Contributor

@egonelbre egonelbre commented Jun 10, 2012

Comment 12:

The change Dan suggested seems to work fine - GC was slow, but still released memory
when over 100GB of RAM was used. I used basic primitives - structs, interfaces, arrays,
maps, channels.
@rsc
Copy link
Contributor Author

@rsc rsc commented Jun 18, 2012

Comment 13:

Labels changed: added go1.1.

@rsc
Copy link
Contributor Author

@rsc rsc commented Jun 18, 2012

Comment 14:

Very nice, thank you for identifying that fix.
@davecheney
Copy link
Contributor

@davecheney davecheney commented Jul 8, 2012

Comment 15:

This may be useful, http://golang.org/cl/6343086.
@alberts
Copy link
Contributor

@alberts alberts commented Sep 6, 2012

Comment 16:

Here's a story from the trenches that could be considered.
We have been running Go services under systemd on Linux. We have also been using the
systemd support for cgroups to limit the memory these processes can allocate.
There was some discussion on systemd-devel:
http://lists.freedesktop.org/archives/systemd-devel/2012-March/004802.html
http://lists.freedesktop.org/archives/systemd-devel/2012-March/004811.html
http://lists.freedesktop.org/archives/systemd-devel/2012-March/004826.html
http://lists.freedesktop.org/archives/systemd-devel/2012-March/004810.html
http://lists.freedesktop.org/archives/systemd-devel/2012-March/004823.html
http://www.kernel.org/doc/Documentation/cgroups/memory.txt
As far as we understand memory.txt, cgroups memory limits accounts mapped files under
some conditions, so it doesn't work for limiting a Go process that should have a small
heap but be able to map terabytes of files. As far as I can tell, RSS limits aren't
implemented on Linux.
We see the following kinds of programs:
1. Go programs that don't map files
2. Go programs that call cgo libraries (that might leak)
3. Go programs that map files
4. Go programs that call cgo libraries and map files
Some programs must simply never hit their memory limits under normal operation. Some
programs that interact with connections from many other programs must potentially be
able to throttle or later reject work depending on how close to their memory limit they
are.
For pure Go programs that don't map files, you can use cgroups. You can even use both
the hard memory limit and a soft memory limit and using an eventfd (see memory.txt) or
reading your limits you can throttle back or cancel actions in your service to limit its
memory. Currently you need to keep both these limits a bit below 16 GB to avoid hitting
the heap limit.
Any kind of program that maps files is tricky. You can't use cgroups for these kinds of
programs in general. It can be useful to limit the Go heap, but depending on the
service, you might want to do different things when you go over the soft limit. You
don't necessarily want an infinite hard limit. Eventually the kernel's OOM killer might
kill you, but it might kill someone else first.
Go+leaking cgo+mapped files is the hardest. The best you can do here is probably to fix
the leaks and then just limit the Go part.
Finally, this leads me to my proposal. Once the 16 GB limit is lifted (which allows apps
to potentially grow huge heaps), it might be useful to be able to instruct the runtime
to still limit the heap to cater for cases where cgroups can't help you.
It's probably a bit sucky to do it via environment variables or flags understood by all
Go programs (kind of like the JVM -Xmx stuff), so people that want it can do their own
flags and call runtime functions.
I think a soft limit in the runtime is also useful (maybe with some kind of channel API,
like you can get with the cgroups eventfd thing) as it allows goroutines handling
transactions to temporarily pause work (maybe inside a select that combines a limit
checker, ticker, and other channels) when you go over the soft limit, but before the
runtime starts failing allocations due to hitting the hard limit.
P.S. the cgroups code for reading limits looks something like this. We haven't done
anything with the eventfds yet. This code assumes that the limits don't change while the
program is running.
func MemoryLimit() int64 {
    return memoryLimit
}
func MemorySoftLimit() int64 {
    return memorySoftLimit
}
func MemoryUsage() int64 {
    usageStr := readLine(memoryUsagePath)
    n, err := strconv.ParseUint(usageStr, 0, 64)
    if err != nil {
        return 0
    }
    return int64(n)
}
var memoryLimit int64
var memorySoftLimit int64
var memoryUsagePath string
func init() {
    var path string
    lines := readLines("/proc/self/cgroup")
    for _, line := range lines {
        if !strings.Contains(line, ":memory:") {
            continue
        }
        parts := strings.Split(line, ":")
        if len(parts) != 3 {
            continue
        }
        path = "/sys/fs/cgroup/memory" + parts[2]
    }
    if len(path) == 0 {
        return
    }
    softLimitStr := readLine(path + "/memory.soft_limit_in_bytes")
    n, err := strconv.ParseUint(softLimitStr, 0, 64)
    if err == nil {
        memorySoftLimit = int64(n)
    }
    limitStr := readLine(path + "/memory.limit_in_bytes")
    n, err = strconv.ParseUint(limitStr, 0, 64)
    if err == nil {
        memoryLimit = int64(n)
    }
    memoryUsagePath = path + "/memory.usage_in_bytes"
}
@rsc
Copy link
Contributor Author

@rsc rsc commented Sep 24, 2012

Comment 17:

Note that CL 6551067 introduces a MaxMem constant in malloc.h that bounds the size of
any single allocation via makeslice or makechan on 64-bit machines to 16 GB. If you
expand this amount, that constant needs expansion too.
@rsc
Copy link
Contributor Author

@rsc rsc commented Sep 24, 2012

Comment 18:

One constraint about the address choice is that I want to make sure that heap pointers
never appear in valid UTF-8 sequences, for the imprecise garbage collection of at least
stacks if not the whole heap. The current pointers are in the range 0xf8<<32 up to
(but not including) 0xfc<<32, and f8, f9, fa, fb never appear in UTF-8 sequences.
We can expand the current space to ff, so 32 GB, before hitting valid UTF-8.
An alternative would be to use 0x0080<<32 up to (but not including)
0x00C0<<32, because 80-bf never appear in UTF-8 following an ASCII byte like 00.
That would be 256 GB worth of address space.
@rsc
Copy link
Contributor Author

@rsc rsc commented Nov 12, 2012

Comment 19:

Actually the bytes are backward because this is little-endian. So we should use
0x00C0<<32 because 0xC0 and later are never followed by a 00 byte.
@rsc
Copy link
Contributor Author

@rsc rsc commented Nov 12, 2012

@rsc
Copy link
Contributor Author

@rsc rsc commented Nov 13, 2012

Comment 21:

This issue was closed by revision 9799a5a.

Status changed to Fixed.

@rsc rsc added fixed labels Nov 13, 2012
@rsc rsc self-assigned this Nov 13, 2012
@rsc rsc added this to the Go1.1 milestone Apr 14, 2015
@rsc rsc removed the go1.1 label Apr 14, 2015
@golang golang locked and limited conversation to collaborators Jun 24, 2016
This issue was closed.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
8 participants
You can’t perform that action at this time.