Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: failures with "cannot map pages in arena address space" on freebsd-amd64-race since 2021-11-09 #49687

Closed
bcmills opened this issue Nov 19, 2021 · 10 comments
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. release-blocker
Milestone

Comments

@bcmills
Copy link
Member

bcmills commented Nov 19, 2021

fatal error: runtime: cannot map pages in arena address space

runtime stack:
runtime.throw({0x9f747c?, 0x0?})
	/tmp/workdir/go/src/runtime/panic.go:992 +0x71
runtime.sysMap(0x0, 0x40ccc9?, 0xc000611358?)
	/tmp/workdir/go/src/runtime/mem_bsd.go:76 +0xa5
runtime.(*mheap).grow(0xdd4320, 0xc000611200?)
	/tmp/workdir/go/src/runtime/mheap.go:1404 +0x225
runtime.(*mheap).allocSpan(0xdd4320, 0x0, 0x0, 0x0)
	/tmp/workdir/go/src/runtime/mheap.go:1170 +0x171
runtime.(*mheap).alloc.func1()
	/tmp/workdir/go/src/runtime/mheap.go:912 +0x65
runtime.systemstack()
	/tmp/workdir/go/src/runtime/asm_amd64.s:469 +0x49

greplogs --dashboard -md -l -e '^fatal error: runtime: cannot map pages in arena address space' --since=2021-01-01

2021-11-19T00:05:59-ae814b3-e8cda0a/freebsd-amd64-race
2021-11-09T00:08:42-b76863e-5344dca/freebsd-amd64-race

@bcmills
Copy link
Member Author

bcmills commented Nov 19, 2021

I suspect that this may be due to memory exhaustion; as noted in #49347 (comment), the freebsd-amd64-race builder has a higher CPU-to-RAM ratio than most of the other -race builders, which may cause its GOMAXPROCS to be fairly aggressive for the amount of available memory.

@bcmills bcmills added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Nov 19, 2021
@bcmills bcmills added this to the Go1.18 milestone Nov 19, 2021
@bcmills bcmills added okay-after-beta1 Used by release team to mark a release-blocker issue as okay to resolve either before or after beta1 release-blocker labels Nov 19, 2021
@mknyszek
Copy link
Contributor

mknyszek commented Dec 6, 2021

Have there been any new instances of this? I can't find any on the main dashboard (I haven't downloaded the subrepo logs).

@bcmills
Copy link
Member Author

bcmills commented Dec 6, 2021

None since Nov. 19, including the subrepos.

greplogs --dashboard -md -l -e '^fatal error: runtime: cannot map pages in arena address space' --since=2021-11-20

@bcmills
Copy link
Member Author

bcmills commented Dec 6, 2021

Actually, hang on. I think there's something wrong with my regexp.

@bcmills
Copy link
Member Author

bcmills commented Dec 6, 2021

Ah, it was a time-zone difference. There are indeed no occurrences since Nov. 19 UTC.

greplogs --dashboard -md -l -e 'cannot map pages in arena address space' --since=2021-11-01

2021-11-19T00:05:59-ae814b3-e8cda0a/freebsd-amd64-race

@mknyszek
Copy link
Contributor

mknyszek commented Dec 6, 2021

Since this was due to memory exhaustion (it looks like that to me too -- the OS won't let the runtime commit any more pages) and hasn't happened since Nov 19th, @bcmills what do you think about putting this in WaitingForInfo?

We can also try increasing decreasing the CPU-to-RAM ratio as you suggest, but I get the impression that this doesn't even get into any -race code. That is, unless I'm misunderstanding and the go command is actually built with -race. If that's the case, this may also be totally mitigated by https://go.devl/cl/333529 which imports a new TSAN runtime that uses a lot less memory (though I'm gonna guess that's coming in 1.19 at this point).

@cherrymui cherrymui removed the okay-after-beta1 Used by release team to mark a release-blocker issue as okay to resolve either before or after beta1 label Dec 14, 2021
@cherrymui
Copy link
Member

cherrymui commented Dec 15, 2021

Any update on this? Does it happen again? (pinging as this is a release blocker)

the go command is actually built with -race

I think you're right, the go command is not built with -race flag.

Perhaps other processes that are built with -race use more memory and cause the go command fail to map the address?

@aclements
Copy link
Member

aclements commented Jan 5, 2022

As far as I can tell this hasn't happened again (though I only have a partial image of the subrepo logs). This seems like just random memory exhaustion, especially given the builder config. I think we should just close this.

@aclements
Copy link
Member

aclements commented Jan 18, 2022

The error reporting could definitely be better here (like, what was errno?). I think we should just improve the error reporting and close this. If it recurs, then we can reopen with more information.

@gopherbot
Copy link

gopherbot commented Jan 19, 2022

Change https://golang.org/cl/379354 mentions this issue: runtime: print error if mmap fails

jproberts pushed a commit to jproberts/go that referenced this issue Jun 21, 2022
Fixes golang#49687

Change-Id: Ife7f64f4c98449eaff7327e09bc1fb67acee72c9
Reviewed-on: https://go-review.googlesource.com/c/go/+/379354
Trust: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Austin Clements <austin@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. release-blocker
Projects
None yet
Development

No branches or pull requests

5 participants