Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: linux-arm builder intermittently failing TestPhysicalMemoryUtilization #32010

josharian opened this issue May 13, 2019 · 7 comments


Copy link

@josharian josharian commented May 13, 2019

Sample from

--- FAIL: TestPhysicalMemoryUtilization (0.15s)
    malloc_test.go:175: expected "OK\n", but got "exceeded physical memory overuse threshold of 10%: 11.42%\n(alloc: 378579448, sys: 507150336, rel: 85319680, objs: 240)\n"
FAIL	runtime	73.024s

cc @aclements @mknyszek

Tentatively marking release-blocker, since we want the builders to be consistently green.

Copy link

@mknyszek mknyszek commented May 13, 2019

Thanks for pointing this out and creating an issue. I've seen this before and it should be resolved sooner rather than later.

The test likely needs to be tweaked or revamped because of the new scavenger, or maybe if #32012 is fixed it won't really be relevant anymore.

I'll update this issue when I understand better what to do about this test.

@mknyszek mknyszek self-assigned this May 13, 2019
Copy link

@mknyszek mknyszek commented May 14, 2019

OK so after thinking about this for a bit, and looking over the failures (which are all just failures due to rising slightly above the threshold), I think we should just raise the threshold.

The test is still useful, but the scavenging policy now is to retain a 10% overhead of RSS over the heap goal, which means that picking 10% as the test's threshold is probably not reasonable, especially since the scavenger isn't guaranteed to finish by the time the test actually checks what the utilization is. The test is still useful though, to ensure the scavenger is actually doing its job, but there's no guarantee anymore that it'll be under 10%. We could sleep to ensure the scavenger runs for long enough with a worst-case pace to ensure we get close to or below the threshold, but adding more time to all.bash seems like the wrong way to go.

Instead, increasing the threshold just accounts for any variability in the scavenger being able to complete its goal. It should be extremely likely for the test's utilization to be under 15%, especially with the fix to #32012, so I think we should just set it to that.


Copy link

@gopherbot gopherbot commented May 14, 2019

Change mentions this issue: runtime: increase physical memory utilization threshold in test

Copy link

@laboger laboger commented May 17, 2019

This problem has been happening intermittently on linux/ppc64 and linux/ppc64le.

Copy link

@bradfitz bradfitz commented May 19, 2019

This now happens on linux-amd64-longtest too:

Copy link

@shakeel shakeel commented May 20, 2019

Fails consistently on my local Linux workstation --- FAIL: TestPhysicalMemoryUtilization (0.01s) malloc_test.go:175: expected "OK\n", but got "exceeded physical memory overuse threshold of 10%: 18.15%\n(alloc: 378900688, sys: 536248320, rel: 88571904, objs: 240)\n" FAIL

Copy link

@mknyszek mknyszek commented May 20, 2019

Yeah I can reproduce in linux/amd64 also. I've overhauled the test now and it passes very consistently on linux/amd64 and linux/arm. Waiting on review now for

@gopherbot gopherbot closed this in 5a90306 May 20, 2019
@golang golang locked and limited conversation to collaborators May 19, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
6 participants
You can’t perform that action at this time.