-
Notifications
You must be signed in to change notification settings - Fork 17.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
runtime: high startup address space usage (RLIMIT_AS) on Linux AMD64 #38010
Comments
@pkramme Hi, would you mind to post your Apache setup for this too? I did not reproduce this on a fresh Apache. Maybe it is due to my configuration. |
@pkramme is this shared hosting environment cPanel by any chance? We also started getting reports of this same panic with our Go application, which exposes itself as a Going to downgrade to 1.13.9 for now. |
@aleksator No, it is not, it is a custom build setup. @ivzhh I'm not able to share the config, as it is proprietary. Theoretically, if we execute any code with 512MB memory limitation, the problem should become visible. I will try to produce something not based on fcgi as a reproducer, so that no apache2 setup is necessary. |
Tagging a proper person here: @alexzorin |
I think what @pkramme suggested about the 512MB memory limit is correct - specifically "Back in the day" (EL5-ish era), shared web hosting admins did not have access to the RSS cgroups controller (because of EL5's ancient kernel), and so controlling VSZ limits was the only choice available to them. In the long term, this has resulted in a lot of misguided admins keeping these VSZ limits around for no good reason. Anyway, the Apache-based reproducer is straightforward. (For some reason, a simple Go hello world wrapped in a bash
package main
import (
"net/http/cgi"
)
func main() {
if err := cgi.Serve(nil); err != nil {
panic(err)
}
}
RLimitMEM 536870912
I would prefer not to ask our customers to remove the rlimit (or else we'll be stuck shipping with Go 1.13 for all eternity). Would it be practical for the Go runtime to try work within whatever it sees by |
Adding that I'm also seeing this issue in a different memory-limited environment with a 512mb limit (a Grid Engine setup). Raising the memory limit to 950mb fixes the issue, but it's unclear to me why it should ever be an issue anyway - the program does not use that much memory during running. |
apologies for the "me too" post, but this has also prevented to migrate a little Go-based "script" of one of my colleagues at CERN from Go-1.13.x to the latest Go-1.14.x. |
Well, after reinvestigating this issue I stumbled over the proposal for the new page allocator which was introduced in go1.14: https://github.com/golang/proposal/blob/master/design/35112-scaling-the-page-allocator.md
So, this explains it. @aclements Is there a workaround for cases like this? |
cc @mknyszek |
As @pkramme points out, we were aware of this issue when the changes to the page allocator were proposed. As @alexzorin points out,
The short answer is no. The virtual memory mappings made to support structures in the page allocator significantly simplified the improvements made in the 1.14 release. Earlier on in the release cycle the amount of memory mapped was much larger which caused problems on certain platforms where the default @sbinet @mashedkeyboard @alexzorin @pkramme: In order to understand your situations better, could you elaborate on the reasons why your and/or your customers cannot set As a side note, (and to be totally clear, I'm not recommending this as an official workaround) compiling your code with |
This is our plan. It's going to be a challenge for XX,000 hosts between X,000 customers, so we are first planning to add telemetry to our 1.13 builds to see how many systems run the CGI under restricted virtual memory. |
Sure!
The Go software I am writing is supposed to run on a shared hosting server.
The technical foundation is a LAMP (linux apache2 mysql php/python/...)
stack. Inside a shared hosting LAMP stack, the Apache2 webserver is spawning
CGI/FastCGI software inside a restricted environment, which is heavily
controlled in access and resources by the provider, in order to prevent one
user taking up all the resources. On my shared hosting account one FastCGI
process is limited to 512MB "memory".
The important part is that in managed hosting, you simply cannot make that
change, because you do not control the environment. The only possibility for
me is to upgrade to another, more expensive hosting plan, so that I can use
1GB memory or more so that this allocation works.
|
We don't have much lever on how to configure the CGI environment. nonetheless, I've sent a ticket on raising the we'll see. (anyways, it's not a high profile CGI service, we won't miss supersymetry nor mini-blackholes, or loose the beam if we're stuck w/ Go-1.13.x b/c of that. :P) |
@mknyszek Is there any progress on this on your end? |
To put a conclusion on this from my end, we gathered some As long as Go continues to work within that limit, we're happy to live with it and will ask those other users to adapt. Thanks. |
any progress on it? |
Go 1.14 以上在 judge 限制环境中运行时会报错: `fatal error: failed to reserve page summary memory` 参考 golang/go#38010
Can't believe this remains unresolved. Are we supposed to downgrade golang to 1.13? |
What version of Go are you using (
go version
)?Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (
go env
)?go env
OutputWhat did you do?
I am trying to get an FastCGI server running behind an Apache2 webserver on a shared hosting system using the net/http/fcgi library. The webserver is limiting my software to 512MB memory.
This is the code: https://play.golang.org/p/Z-Gc6icOpw5
What did you expect to see?
I expect to see "This was generated by Go running as a FastCGI app" on the website generated by the FastCGI server.
What did you see instead?
I have modified the sysReserve() function in the runtime to include println() to print out the error code from mmap() and the requested memory size. This is a diff of
src/runtime/mem_linux.go
and my version:I kept the output in the following output in the hopes that it might be useful.
The application crashes with this trace:
The application works fine with golang 1.13.9.
I have no idea how to debug this further.
The text was updated successfully, but these errors were encountered: