runtime: 512GB memory limitation

[ndaniels]

In Go 1.4, the compiler still has a hardcoded 128GB memory limit. While this is much better than the 16GB limit from 1.0, it's presenting a real limitation to using Go for big-data scientific work, such as my research, such as https://github.com/ndaniels/cablastp2. While it's trivial for me to change the allocation limit in go's source and rebuild my own compiler, this presents a problem for other users using my work; if they find out that they need to patch the Go compiler just to build my source code, they're not going to like it. It's already enough of a barrier for some users to require they have the Go compiler. While I could provide statically compiled binaries for a variety of platforms, I can't target every OS and kernel version, and it becomes a nuisance to update those binaries whenever I update my source code.

I understand Go's memory allocation philosophy, and I realize that (for the time being) it needs to be hardcoded in a Go binary. But, it would be a huge improvement if a compiler flag could be used to alter the allocation limit for a particular package being compiled. Even better would be a runtime flag (much like Java's Xmx flag for the heap).

[ianlancetaylor]

To be clear, I think you are talking about the value of MaxMem in runtime/malloc.go (tip) or runtime/malloc.h (Go 1.4), right?

[ndaniels]

Yes, exactly. Couldn't that be a default, but have it changeable via a compiler flag? I mean when calling 6g, not when compiling 6g.

[minux]

How about use ulimit -v as a mechanism to increase the max heap? If ulimit -v (RLIMIT_AS) is unlimited, we reserve for 128GB, however, if it's limited to another amount, we reserve that amount. so if you raise the ulimit -v for a Go program, it will be able to use more memory. Another benefit is that you can actually reduce the VM footprint of a Go process if needed.

[ndaniels]

I think an explicit command-line argument to the binary (handled by the runtime, perhaps; the GHC Haskell compiler, for example, does this with -rtsopts) is more intuitive and discoverable, and less likely to lead to surprising effects. That would be ideal; a reasonable fallback would be a compile-time flag. Certainly it's better than requiring users to patch the compiler source code!

[minux]

We can't add runtime command line flags to Go binaries, that's for sure. (Had we reserved a -goruntime or similar flag before Go 1 is released, we could probably do that, but now adding any universal runtime flag might have conflict with user command line parsing.) Using an environment variable, perhaps, but the runtime already takes many environment variables, we shouldn't add one more lightly. I think using ulimit to control VM usage is very natural and it also fixes a few other issues (where on certain systems due to large amount of VM space reserved, you can't fork a Go process and thus can't execute another program.) PS: changing MaxMem doesn't require patching the Go compiler. It's a runtime change.

[ndaniels]

I certainly understand that something like a "-goruntime" flag can't be added until Go 2.

I think there is definite interest in Go in data science, and the need for more than 128GB is going to be common enough. So, perhaps adding an environment variable is warranted.

How does one change MaxMem at runtime (of the compiler, I presume)? Up until 1.4 it certainly required a source change to malloc.(go,h); I recall patching it for 1.0 when 16GB didn't cut it.

[RLH]

If we want to increase this to say a Terabyte I don't see why a Go flag or
environment variable is needed, reflecting the limits of the underlying
OS/HW seems the correct approach. I don't have a sense of what, in terms
of allocation rate, pointer density, sharing between Goroutines, and
pointer mutation, would fill a Terabyte of Go memory.

On Tue, Apr 14, 2015 at 11:07 PM, Noah Daniels notifications@github.com
wrote:

I certainly understand that something like a "-goruntime" flag can't be
added until Go 2.

I think there is definite interest in Go in data science, and the need for
more than 128GB is going to be common enough. So, perhaps adding an
environment variable is warranted.

How does one change MaxMem at runtime (of the compiler, I presume)? Up
until 1.4 it certainly required a source change to malloc.(go,h); I recall
patching it for 1.0 when 16GB didn't cut it.

—
Reply to this email directly or view it on GitHub
#10460 (comment).

[rsc]

It is not trivial to just change MaxMem at run time, and it's probably too late in the cycle to introduce code allowing MaxMem to be a variable, although I agree it is probably the right long-term solution.

I will, however, bump MaxMem on 64-bit systems to 512 GB for Go 1.5. That should buy us some more headroom until we can fix things properly.

[gopherbot]

CL https://golang.org/cl/10819 mentions this issue.

[ndaniels]

"I don't have a sense of what, in terms
of allocation rate, pointer density, sharing between Goroutines, and
pointer mutation, would fill a Terabyte of Go memory."

The National Center for Biotechnology Information (NCBI) maintains a database of "non-redundant" (unique) protein sequences. That database is now over 60 million sequences, and for the past several years has been doubling every two years.

PubChem (a public database of chemical compounds) is over 300GB in size on disk. It has also been growing, though not as quickly.

If Go is going to be useful for big-data science, it needs to be able to use as much RAM as we can throw into a system. Terabyte-plus compute servers are common now. Now, I don't need a terabyte today, but I need more than 128GB. The right solution is for this not to be hardcoded.

In my case, what consumes all the RAM is a large lookup table of kmers (very short sequences) indexed into every sequence in the database I'm trying to compress. Lots of pointers, lots of string data, and we don't want to write things out to disk until we're done.

[davecheney]

@ndaniels understood. This is not a solution, just a stop gap, see the comment by @rsc from three days ago.

[pbnjay]

I wanted to raise this issue again, I know it's been a while but I've hit the 512GB limit on a system with 4TB. It took me a while to determine that Go's runtime was the issue, and not something else. Ideally, the out of memory error message could report that this is an internal Go limit reached and not what looks like a system OOM message.

I feel 512GB is perfectly reasonable as a default, but it would be great to document and provide a way for make.bash et al to update this limit. I've simply hardcoded _MHeapMap_TotalBits = 42 for now but it feels brittle.

FWIW, I'm also doing bioinformatics work like @ndaniels - Go is very well suited to it.

[rsc]

/cc @aclements to think about for Go 1.10.

[rajakhurram]

+1 @pbnjay
At the minimum, Golang should exit with a meaningful panic() when the system hits 512GB limit. This should really be released as a fix in 1.8 and should not wait for the next major release.
We are using memory mapped files in our system, and this means that our virtual address space can be huge - virtually unlimited. Please either provide a compile time flag for setting the limit or automatically extend it at runtime to honor "ulimit -v = unlimited" on linux systems.

[aclements]

At the minimum, Golang should exit with a meaningful panic() when the system hits 512GB limit.

This would require threading some error information around that we currently don't, but should be pretty easy.

We are using memory mapped files in our system, and this means that our virtual address space can be huge - virtually unlimited.

I'm not sure what problem you're getting at here. Memory mapped files have no effect on Go's heap arena limit since they don't live in the Go heap.

Please either provide a compile time flag for setting the limit or automatically extend it at runtime to honor "ulimit -v = unlimited" on linux systems.

I would love to eliminate this limit, but doing so requires a complete redesign of the Go heap. I've actually been thinking about how to do that, but it's not a simple matter of just extending it at runtime. :)

[gopherbot]

Change https://golang.org/cl/85887 mentions this issue: runtime: use sparse mappings for the heap

[gopherbot]

Change https://golang.org/cl/85889 mentions this issue: runtime: move comment about address space sizes to malloc.go

[gopherbot]

Change https://golang.org/cl/85885 mentions this issue: runtime: make span map sparse

[gopherbot]

Change https://golang.org/cl/85888 mentions this issue: runtime: remove non-reserved heap logic

[gopherbot]

Change https://golang.org/cl/85884 mentions this issue: runtime: abstract remaining mheap.spans access

[gopherbot]

Change https://golang.org/cl/85882 mentions this issue: runtime: fix various contiguous bitmap assumptions

[gopherbot]

Change https://golang.org/cl/85883 mentions this issue: runtime: make the heap bitmap sparse

[gopherbot]

Change https://golang.org/cl/85886 mentions this issue: runtime: eliminate most uses of mheap_.arena_*