runtime: 512GB memory limitation

[ndaniels]

In Go 1.4, the compiler still has a hardcoded 128GB memory limit. While this is much better than the 16GB limit from 1.0, it's presenting a real limitation to using Go for big-data scientific work, such as my research, such as https://github.com/ndaniels/cablastp2. While it's trivial for me to change the allocation limit in go's source and rebuild my own compiler, this presents a problem for other users using my work; if they find out that they need to patch the Go compiler just to build my source code, they're not going to like it. It's already enough of a barrier for some users to require they have the Go compiler. While I could provide statically compiled binaries for a variety of platforms, I can't target every OS and kernel version, and it becomes a nuisance to update those binaries whenever I update my source code.

I understand Go's memory allocation philosophy, and I realize that (for the time being) it needs to be hardcoded in a Go binary. But, it would be a huge improvement if a compiler flag could be used to alter the allocation limit for a particular package being compiled. Even better would be a runtime flag (much like Java's Xmx flag for the heap).

[ianlancetaylor]

To be clear, I think you are talking about the value of MaxMem in runtime/malloc.go (tip) or runtime/malloc.h (Go 1.4), right?

[ndaniels]

Yes, exactly. Couldn't that be a default, but have it changeable via a compiler flag? I mean when calling 6g, not when compiling 6g.

[minux]

How about use ulimit -v as a mechanism to increase the max heap? If ulimit -v (RLIMIT_AS) is unlimited, we reserve for 128GB, however, if it's limited to another amount, we reserve that amount. so if you raise the ulimit -v for a Go program, it will be able to use more memory. Another benefit is that you can actually reduce the VM footprint of a Go process if needed.

[ndaniels]

I think an explicit command-line argument to the binary (handled by the runtime, perhaps; the GHC Haskell compiler, for example, does this with -rtsopts) is more intuitive and discoverable, and less likely to lead to surprising effects. That would be ideal; a reasonable fallback would be a compile-time flag. Certainly it's better than requiring users to patch the compiler source code!

[minux]

We can't add runtime command line flags to Go binaries, that's for sure. (Had we reserved a -goruntime or similar flag before Go 1 is released, we could probably do that, but now adding any universal runtime flag might have conflict with user command line parsing.) Using an environment variable, perhaps, but the runtime already takes many environment variables, we shouldn't add one more lightly. I think using ulimit to control VM usage is very natural and it also fixes a few other issues (where on certain systems due to large amount of VM space reserved, you can't fork a Go process and thus can't execute another program.) PS: changing MaxMem doesn't require patching the Go compiler. It's a runtime change.

[ndaniels]

I certainly understand that something like a "-goruntime" flag can't be added until Go 2.

I think there is definite interest in Go in data science, and the need for more than 128GB is going to be common enough. So, perhaps adding an environment variable is warranted.

How does one change MaxMem at runtime (of the compiler, I presume)? Up until 1.4 it certainly required a source change to malloc.(go,h); I recall patching it for 1.0 when 16GB didn't cut it.

[RLH]

If we want to increase this to say a Terabyte I don't see why a Go flag or
environment variable is needed, reflecting the limits of the underlying
OS/HW seems the correct approach. I don't have a sense of what, in terms
of allocation rate, pointer density, sharing between Goroutines, and
pointer mutation, would fill a Terabyte of Go memory.

On Tue, Apr 14, 2015 at 11:07 PM, Noah Daniels notifications@github.com
wrote:

I certainly understand that something like a "-goruntime" flag can't be
added until Go 2.

I think there is definite interest in Go in data science, and the need for
more than 128GB is going to be common enough. So, perhaps adding an
environment variable is warranted.

How does one change MaxMem at runtime (of the compiler, I presume)? Up
until 1.4 it certainly required a source change to malloc.(go,h); I recall
patching it for 1.0 when 16GB didn't cut it.

—
Reply to this email directly or view it on GitHub
#10460 (comment).

[rsc]

It is not trivial to just change MaxMem at run time, and it's probably too late in the cycle to introduce code allowing MaxMem to be a variable, although I agree it is probably the right long-term solution.

I will, however, bump MaxMem on 64-bit systems to 512 GB for Go 1.5. That should buy us some more headroom until we can fix things properly.

[gopherbot]

CL https://golang.org/cl/10819 mentions this issue.

[ndaniels]

"I don't have a sense of what, in terms
of allocation rate, pointer density, sharing between Goroutines, and
pointer mutation, would fill a Terabyte of Go memory."

The National Center for Biotechnology Information (NCBI) maintains a database of "non-redundant" (unique) protein sequences. That database is now over 60 million sequences, and for the past several years has been doubling every two years.

PubChem (a public database of chemical compounds) is over 300GB in size on disk. It has also been growing, though not as quickly.

If Go is going to be useful for big-data science, it needs to be able to use as much RAM as we can throw into a system. Terabyte-plus compute servers are common now. Now, I don't need a terabyte today, but I need more than 128GB. The right solution is for this not to be hardcoded.

In my case, what consumes all the RAM is a large lookup table of kmers (very short sequences) indexed into every sequence in the database I'm trying to compress. Lots of pointers, lots of string data, and we don't want to write things out to disk until we're done.

[davecheney]

@ndaniels understood. This is not a solution, just a stop gap, see the comment by @rsc from three days ago.