Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: freebsd-amd64-race logs for x/tools marked as failing, but do not include error messages #49347

Open
bcmills opened this issue Nov 4, 2021 · 1 comment

Comments

@bcmills
Copy link
Member

@bcmills bcmills commented Nov 4, 2021

(Note that the below regexp is not specific to freebsd or to x/tools. It is just looking for logs that contain only non-error lines.)

greplogs --dashboard -md -l -e '(?m)\A[^ ]+ at.*\n\n(?:(?:[:][:] Running .+\n\n)|(?:go: downloading [^ ]+ v[^ ]+\n)|(?:ok\s+[^ ]+\s+[^ ]+s\n)|(?:[?]\s+[^ ]+\s+\[no test files\]\n))+\z'

2021-11-04T07:05:31-84e69e7-2622235/freebsd-amd64-race
2021-11-03T23:49:25-84e69e7-fa19146/freebsd-amd64-race
2021-11-03T19:32:33-3b1a674-74f99d0/freebsd-amd64-race
2021-11-03T00:49:50-714668c-a4b2c57/freebsd-amd64-race
2021-11-02T23:44:58-714668c-a0f373c/freebsd-amd64-race
2021-11-02T16:40:04-058ed05-8c8baad/freebsd-amd64-race
2021-11-02T00:12:17-ee08195-81fea0b/freebsd-amd64-race
2021-10-28T16:54:58-c8ad2e1-6bd0e7f/freebsd-amd64-race
2021-10-26T01:18:46-f916b54-49fda9e/freebsd-amd64-race
2021-10-25T14:51:39-fc8b4ca-1ed060c/freebsd-amd64-race
2021-10-17T02:23:16-98f6e03-fa7d11a/freebsd-amd64-race
2021-10-16T16:27:40-98f6e03-640a49b/freebsd-amd64-race
2021-10-16T14:50:21-98f6e03-680caf1/freebsd-amd64-race
2021-10-15T20:34:15-98f6e03-1b072b3/freebsd-amd64-race
2021-10-15T14:09:01-98f6e03-1cbec68/freebsd-amd64-race
2021-10-15T02:45:22-288d0b3-2ac3bdf/freebsd-amd64-race
2021-10-15T02:02:50-ce04ca3-3da0ff8/freebsd-amd64-race
2021-10-15T01:35:56-ce04ca3-0c45ed0/freebsd-amd64-race
2021-10-14T23:32:09-3a269dc-9e8ed86/freebsd-amd64-race
2021-10-14T18:44:37-e69ba9d-a37bebc/freebsd-amd64-race

CC @golang/release

@bcmills
Copy link
Member Author

@bcmills bcmills commented Nov 11, 2021

I think the process may be being killed due to memory exhaustion.

The freebsd-amd64-race builder appears to be running on an e2-highcpu-16 builder, while the normal linux-amd64-race builder is running on an e2-standard-4. (https://play.golang.org/p/5LXUgwHEZPn)

According to https://cloud.google.com/compute/vm-instance-pricing, e2-standard-4 has 4 CPUs and 16GB memory, while e2-highcpu-16 has 16 CPUs and 16GB.

Since the default GOMAXPROCS is based on the CPU count, and the default test parallelism is based on GOMAXPROCS, I suspect that the increased CPU-to-RAM ratio on the highcpu host is causing it to run too many gopls tests in parallel and run out of RAM.

@findleyr, @golang/release: should we address this by adjusting the host shape, or by adding some kind of additional limiting to the test, or explicitly setting GOMAXPROCS on this builder (which would possibly reduce its testing throughput), or something else entirely?

Loading

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants