x/build: frequent "communication error to buildlet" failures on plan9-amd64-0intro

[bcmills]

#!watchflakes
post <- builder == "plan9-amd64-0intro" && `communication error to buildlet`

greplogs --dashboard -md -l -e 'communication error to buildlet' --since=2021-11-01

2021-11-22T23:51:43-83bfed9/plan9-amd64-0intro
2021-11-22T23:09:35-9678f79/plan9-amd64-0intro
2021-11-22T20:34:40-f13fcd9/plan9-amd64-0intro
2021-11-22T16:53:57-cd0bf38/plan9-arm
2021-11-22T04:27:29-e30ebaa/plan9-amd64-0intro
2021-11-20T00:32:49-57aba32/plan9-386-0intro
2021-11-19T21:59:14-5e774b0/plan9-amd64-0intro
2021-11-19T21:41:33-ba9f0f6/plan9-amd64-0intro
2021-11-18T16:18:50-feb330d/plan9-amd64-0intro
2021-11-17T19:51:32-9a33945/plan9-amd64-0intro
2021-11-17T19:51:32-9a33945/plan9-arm
2021-11-17T19:18:24-aa34ea2/plan9-amd64-0intro
2021-11-17T19:18:24-aa34ea2/plan9-arm
2021-11-17T18:14:23-9bdbed1/plan9-arm
2021-11-16T14:02:36-bddb79f/plan9-arm
2021-11-12T18:10:48-0c6a6cd/plan9-arm
2021-11-10T21:32:50-f410786/plan9-arm
2021-11-10T17:15:54-8a3be15/plan9-arm
2021-11-09T20:08:48-f48115c/plan9-arm
2021-11-07T04:57:22-9e6ad46/plan9-amd64-0intro
2021-11-05T21:13:38-091948a/plan9-arm
2021-11-05T17:46:27-4f543b5/plan9-arm
2021-11-05T17:39:43-0bc98b3/plan9-arm
2021-11-05T17:23:06-37951d8/plan9-arm
2021-11-05T16:35:00-f249fa2/plan9-arm
2021-11-04T18:22:03-b2149ac/plan9-386-0intro
2021-11-04T18:22:03-b2149ac/plan9-amd64-0intro
2021-11-04T17:07:48-5772877/plan9-amd64-0intro
2021-11-04T14:54:46-00d6d20/plan9-amd64-0intro
2021-11-04T14:17:18-901bf29/plan9-arm
2021-11-04T07:05:31-2622235/plan9-amd64-0intro
2021-11-04T02:57:53-2cf85b1/plan9-amd64-0intro
2021-11-04T02:57:48-5fd0c49/plan9-amd64-0intro
2021-11-04T00:15:18-be0cd9e/plan9-arm
2021-11-02T03:54:24-3c61cb3/plan9-arm
2021-11-01T22:55:50-02e5913/plan9-arm
2021-11-01T15:55:20-d2b5121/plan9-arm

CC @millerresearch @0intro

@golang/release, what provokes this error message in the coordinator? (Is this a failing keepalive of some sort?)

[dmitshur]

This looks relevant to:

// If a build fails multiple times due to communication
// problems with the buildlet, assume something's wrong with
// the buildlet or machine and fail the build, rather than
// looping forever. This promotes the err (communication
// error) to a remoteErr (an error that occurred remotely and
// is terminal).
if rerr := st.repeatedCommunicationError(err); rerr != nil {
	remoteErr = rerr
	err = nil
	doneMsg = "communication error to buildlet (promoted to terminal error): " + rerr.Error()
	fmt.Fprintf(st, "\n%s\n", doneMsg)
}

The current implementation of repeatedCommunicationError is implemented only for Plan 9, and doesn't allow for any retries yet:

// For now, only do this for plan9, which is flaky (Issue 31261)
if strings.HasPrefix(st.Name, "plan9-") && execErr == errBuildletsGone {
	// TODO: give it two tries at least later (store state
	// somewhere; global map?). But for now we're going to
	// only give it one try.
	return fmt.Errorf("network error promoted to terminal error: %v", execErr)
}

So this is related to issues #31261 and #13026.

[millerresearch]

When I observe this on the plan9-arm builders, it's because the builder machine has crashed or rebooted for some reason unrelated to the go tests (eg power flicker, file server disk full). I often use the retrybuilds utility to clear and restart the failed build, and it's fine.

[bcmills]

The rate of these for the plan9-arm builder seems to have gone down over the past couple of weeks, but the plan9-amd64-0intro builder appears to be particularly unreliable.

@0intro, could you please investigate?

greplogs --dashboard -md -l -e 'communication error to buildlet' --since=2021-11-24

2021-12-08T23:38:20-d6c4583/plan9-amd64-0intro
2021-12-08T23:00:01-f5ddd94/plan9-amd64-0intro
2021-12-08T21:21:44-9e29dd4/plan9-amd64-0intro
2021-12-08T18:30:00-61011de/plan9-amd64-0intro
2021-12-08T18:06:06-7b7efd7/plan9-amd64-0intro
2021-12-08T17:55:13-c1c303f/plan9-amd64-0intro
2021-12-08T17:48:45-85a8e17/plan9-amd64-0intro
2021-12-08T17:34:34-6b60911/plan9-amd64-0intro
2021-12-08T17:24:46-ac7e950/plan9-amd64-0intro
2021-12-08T17:04:27-c759ec2/plan9-amd64-0intro
2021-12-08T16:50:04-f5b5939/plan9-amd64-0intro
2021-12-08T15:45:29-46db6aa/plan9-amd64-0intro
2021-12-08T15:34:19-3042ba3/plan9-amd64-0intro
2021-12-08T15:15:40-08025a9/plan9-amd64-0intro
2021-12-08T04:14:00-a19e72c/plan9-amd64-0intro
2021-12-08T01:23:09-016e6eb/plan9-amd64-0intro
2021-12-08T01:17:33-34573ae/plan9-amd64-0intro
2021-12-07T22:33:39-0a15e78/plan9-amd64-0intro
2021-12-07T22:33:33-a3ae45e/plan9-amd64-0intro
2021-12-07T21:31:42-dc50d69/plan9-amd64-0intro
2021-12-07T21:16:00-daf9018/plan9-386-0intro
2021-12-07T21:16:00-daf9018/plan9-amd64-0intro
2021-12-07T21:15:51-2ebe081/plan9-amd64-0intro
2021-12-07T02:31:12-e07b02f/plan9-amd64-0intro
2021-12-04T01:07:28-fa88ba1/plan9-386-0intro
2021-12-03T18:27:20-c4a8550/plan9-amd64-0intro
2021-12-03T18:09:19-deb988a/plan9-amd64-0intro
2021-11-30T18:09:02-931d80e/plan9-amd64-0intro
2021-11-29T19:24:34-68da368/plan9-arm
2021-11-27T23:29:50-0fa53e4/plan9-amd64-0intro
2021-11-27T23:27:52-9f2a075/plan9-386-0intro
2021-11-27T19:49:32-a142d65/plan9-amd64-0intro
2021-11-27T01:14:21-fad67f8/plan9-amd64-0intro
2021-11-26T22:36:21-1d47a11/plan9-amd64-0intro

[0intro]

I've recently reworked the plan9-386 and plan9-amd64 builders so they reboot automatically after running the tests.
This should prevent the builders from remaining stuck in case of failure.

I'll investigate the issues with the plan9-amd64 builder.

[bcmills]

@0intro, any progress on the plan9-amd64 builder? It is currently showing completely broken on the build dashboard.

greplogs --dashboard -md -l -e 'communication error to buildlet' --since=2021-12-10

Details

2022-01-03T23:45:12-95b240b/plan9-amd64-0intro
2022-01-03T23:45:06-a893d0f/plan9-amd64-0intro
2022-01-02T14:27:43-c886143/plan9-amd64-0intro
2022-01-02T14:27:32-d181885/plan9-amd64-0intro
2021-12-29T20:20:32-6178d25/plan9-amd64-0intro
2021-12-29T19:23:11-8a306e2/plan9-amd64-0intro
2021-12-29T04:10:07-91e7821/plan9-amd64-0intro
2021-12-29T01:44:55-a78532a/plan9-amd64-0intro
2021-12-23T20:03:38-b357b05/plan9-amd64-0intro
2021-12-23T19:58:13-0e4b878/plan9-amd64-0intro
2021-12-23T18:58:36-af3b8cf/plan9-amd64-0intro
2021-12-23T17:27:50-ed766b6/plan9-amd64-0intro
2021-12-22T23:31:40-d2ce939/plan9-amd64-0intro
2021-12-22T20:47:09-8cfcee1/plan9-amd64-0intro
2021-12-22T18:43:55-0f3becf/plan9-amd64-0intro
2021-12-22T18:38:41-d7b035f/plan9-amd64-0intro
2021-12-22T00:11:21-90fb5a4/plan9-amd64-0intro
2021-12-21T23:09:59-ebac50e/plan9-amd64-0intro
2021-12-21T22:43:13-3fb17cf/plan9-amd64-0intro
2021-12-21T22:26:48-60f2c12/plan9-amd64-0intro
2021-12-21T19:17:52-a1ce550/plan9-amd64-0intro
2021-12-21T18:43:00-9502339/plan9-amd64-0intro
2021-12-21T03:55:43-2d1d548/plan9-amd64-0intro
2021-12-21T03:54:36-b5e0654/plan9-amd64-0intro
2021-12-21T01:10:17-e087949/plan9-amd64-0intro
2021-12-20T23:46:23-1555062/plan9-amd64-0intro
2021-12-20T23:02:08-59d04d1/plan9-amd64-0intro
2021-12-19T20:16:45-87b2a54/plan9-amd64-0intro
2021-12-17T16:05:31-c5fee93/plan9-amd64-0intro
2021-12-17T03:59:09-5299390/plan9-amd64-0intro
2021-12-17T03:28:06-3c49583/plan9-amd64-0intro
2021-12-17T03:28:05-2bdf34f/plan9-amd64-0intro
2021-12-17T02:00:14-55a25aa/plan9-amd64-0intro
2021-12-17T01:59:32-63077bf/plan9-amd64-0intro
2021-12-17T00:14:53-2e6e9df/plan9-amd64-0intro
2021-12-17T00:14:51-c3561dd/plan9-amd64-0intro
2021-12-16T23:52:17-0c24038/plan9-amd64-0intro
2021-12-16T22:41:31-1387b5e/plan9-amd64-0intro
2021-12-16T22:39:00-b1a53ec/plan9-amd64-0intro
2021-12-16T22:15:57-d01cd8f/plan9-amd64-0intro
2021-12-16T22:15:22-d93677a/plan9-amd64-0intro
2021-12-16T21:02:37-ea26ce7/plan9-amd64-0intro
2021-12-16T15:44:40-1d3a5b4/plan9-amd64-0intro
2021-12-16T00:34:10-7f23145/plan9-amd64-0intro
2021-12-16T00:33:58-d107aa2/plan9-amd64-0intro
2021-12-15T23:51:57-6e7c691/plan9-amd64-0intro
2021-12-15T22:33:52-c9ffcca/plan9-amd64-0intro
2021-12-15T22:32:47-58e8e26/plan9-amd64-0intro
2021-12-15T20:26:19-07ed86c/plan9-amd64-0intro
2021-12-15T20:26:19-07ed86c/plan9-arm
2021-12-15T20:26:10-4cda05d/plan9-amd64-0intro
2021-12-15T20:26:03-bc0aba9/plan9-amd64-0intro
2021-12-15T20:19:05-83fc097/plan9-amd64-0intro
2021-12-15T20:18:30-d5fefbb/plan9-amd64-0intro
2021-12-15T19:56:22-567b177/plan9-amd64-0intro
2021-12-15T14:48:19-b1c7703/plan9-amd64-0intro
2021-12-15T13:31:58-b5c0dba/plan9-amd64-0intro
2021-12-15T00:33:55-9d0ca26/plan9-amd64-0intro
2021-12-15T00:14:57-c948823/plan9-amd64-0intro
2021-12-14T23:57:51-dc5a8f9/plan9-amd64-0intro
2021-12-14T23:55:40-38c067d/plan9-amd64-0intro
2021-12-14T23:20:01-8108444/plan9-amd64-0intro
2021-12-14T22:42:47-1540239/plan9-amd64-0intro
2021-12-14T21:23:43-fc8ae98/plan9-amd64-0intro
2021-12-14T20:25:03-0f05ed3/plan9-amd64-0intro
2021-12-14T19:53:20-d407a8c/plan9-amd64-0intro
2021-12-14T19:10:22-265fbaa/plan9-amd64-0intro
2021-12-14T17:43:51-becaeea/plan9-amd64-0intro
2021-12-14T17:22:06-46ba32a/plan9-amd64-0intro
2021-12-14T16:19:51-c1f012a/plan9-amd64-0intro
2021-12-14T01:48:22-1afa432/plan9-amd64-0intro
2021-12-14T01:18:43-006d4e6/plan9-amd64-0intro
2021-12-13T22:45:26-5b9207f/plan9-amd64-0intro
2021-12-13T22:42:08-67917c3/plan9-amd64-0intro
2021-12-13T21:29:11-9e85dc5/plan9-amd64-0intro
2021-12-13T19:40:29-7bdbc73/plan9-amd64-0intro
2021-12-13T19:08:31-3e8aa5d/plan9-amd64-0intro
2021-12-13T18:55:52-f909f81/plan9-amd64-0intro
2021-12-13T18:45:54-2580d0e/plan9-amd64-0intro
2021-12-13T18:41:35-083ef54/plan9-amd64-0intro
2021-12-13T16:44:13-acc65b4/plan9-amd64-0intro
2021-12-13T16:42:31-4b3d8d1/plan9-amd64-0intro
2021-12-13T16:42:23-b55cbbb/plan9-amd64-0intro
2021-12-13T16:09:22-d198a36/plan9-amd64-0intro
2021-12-13T15:48:26-f4ca598/plan9-amd64-0intro
2021-12-13T15:47:32-6f42be7/plan9-amd64-0intro
2021-12-13T15:47:14-36db10f/plan9-amd64-0intro
2021-12-13T06:35:06-9bfe09d/plan9-amd64-0intro
2021-12-12T16:09:13-49b7c9c/plan9-amd64-0intro
2021-12-12T14:36:21-8692bac/plan9-amd64-0intro
2021-12-12T14:36:20-cc795a0/plan9-amd64-0intro
2021-12-12T14:36:17-6b89773/plan9-amd64-0intro
2021-12-12T06:14:07-9c6e8f6/plan9-amd64-0intro
2021-12-12T02:02:17-1c1998e/plan9-amd64-0intro
2021-12-10T19:26:50-766f89b/plan9-amd64-0intro
2021-12-10T18:20:08-c473ca0/plan9-amd64-0intro
2021-12-10T18:02:44-13d15d1/plan9-amd64-0intro

[bcmills]

Still ongoing — essentially every run for plan9-amd64-0intro for the main Go repo.

greplogs --dashboard -md -l -e 'communication error to buildlet' --since=2022-01-06

Details

2022-01-27T05:30:27-a991d9d/plan9-amd64-0intro
2022-01-27T00:03:31-f4aa021/plan9-amd64-0intro
2022-01-26T23:43:39-db48840/plan9-amd64-0intro
2022-01-26T22:33:26-ef0b09c/plan9-amd64-0intro
2022-01-26T22:09:36-ca6a5c0/plan9-amd64-0intro
2022-01-26T20:51:54-c8b0dce/plan9-amd64-0intro
2022-01-26T19:33:44-8cfbb58/plan9-amd64-0intro
2022-01-26T19:17:30-946a866/plan9-amd64-0intro
2022-01-26T17:58:00-719e989/plan9-amd64-0intro
2022-01-26T17:25:25-55589e7/plan9-amd64-0intro
2022-01-26T17:03:32-0f4bd92/plan9-amd64-0intro
2022-01-26T16:15:53-a9eedc0/plan9-amd64-0intro
2022-01-26T16:07:38-5b1b80b/plan9-amd64-0intro
2022-01-26T15:51:52-827babf/plan9-amd64-0intro
2022-01-25T22:56:45-6eb58cd/plan9-amd64-0intro
2022-01-25T22:04:10-38729cf/plan9-amd64-0intro
2022-01-25T21:57:38-b66bc0a/plan9-amd64-0intro
2022-01-25T20:14:15-078ddec/plan9-amd64-0intro
2022-01-25T00:39:08-16d6a52/plan9-amd64-0intro
2022-01-24T22:28:27-84eefdc/plan9-amd64-0intro
2022-01-24T21:27:33-671e115/plan9-amd64-0intro
2022-01-24T21:27:32-fe85c24/plan9-amd64-0intro
2022-01-24T21:27:28-2abfa30/plan9-amd64-0intro
2022-01-24T21:27:25-7520c08/plan9-amd64-0intro
2022-01-24T21:27:22-9dfd458/plan9-amd64-0intro
2022-01-24T21:27:20-cdd9e93/plan9-amd64-0intro
2022-01-24T21:27:18-4284d45/plan9-amd64-0intro
2022-01-24T21:27:13-0328b4f/plan9-amd64-0intro
2022-01-24T19:28:08-b850f36/plan9-amd64-0intro
2022-01-24T19:02:43-48ec6df/plan9-amd64-0intro
2022-01-24T17:50:23-97e740e/plan9-amd64-0intro
2022-01-24T17:07:30-f88c3b9/plan9-amd64-0intro
2022-01-24T16:55:59-f9df4ea/plan9-amd64-0intro
2022-01-24T14:54:34-19d819d/plan9-amd64-0intro
2022-01-24T12:26:25-0ef6dd7/plan9-amd64-0intro
2022-01-21T23:16:33-b7fa0f9/plan9-amd64-0intro
2022-01-21T21:58:14-35b0db7/plan9-amd64-0intro
2022-01-21T16:59:19-9eba5ff/plan9-amd64-0intro
2022-01-21T01:14:28-d15481b/plan9-amd64-0intro
2022-01-21T00:39:55-32636cd/plan9-amd64-0intro
2022-01-20T19:24:26-2c2e081/plan9-amd64-0intro
2022-01-20T19:10:29-65535bf/plan9-amd64-0intro
2022-01-20T16:31:11-59122f8/plan9-amd64-0intro
2022-01-20T14:59:17-9284279/plan9-amd64-0intro
2022-01-20T01:03:13-e7d5857/plan9-amd64-0intro
2022-01-19T22:56:09-e4ab8b0/plan9-amd64-0intro
2022-01-19T21:46:55-bb7fb8a/plan9-amd64-0intro
2022-01-19T21:14:18-c1296af/plan9-amd64-0intro
2022-01-19T20:54:49-1efc581/plan9-amd64-0intro
2022-01-19T20:47:21-a4d3c73/plan9-amd64-0intro
2022-01-19T20:07:20-d1640d8/plan9-amd64-0intro
2022-01-19T16:33:42-efbecc7/plan9-amd64-0intro
2022-01-19T16:33:11-985d97e/plan9-amd64-0intro
2022-01-19T16:33:11-985d97e/plan9-arm
2022-01-19T16:33:05-3e45eb3/plan9-amd64-0intro
2022-01-19T16:23:21-bec2cc3/plan9-amd64-0intro
2022-01-19T15:45:58-d93ff73/plan9-amd64-0intro
2022-01-19T15:34:05-ca33b34/plan9-amd64-0intro
2022-01-19T14:29:17-2a061fd/plan9-amd64-0intro
2022-01-19T04:00:05-4042194/plan9-amd64-0intro
2022-01-18T23:59:40-50869f3/plan9-amd64-0intro
2022-01-18T23:48:55-fa4df65/plan9-amd64-0intro
2022-01-18T23:19:04-626f13d/plan9-amd64-0intro
2022-01-18T21:43:02-cf5d73e/plan9-amd64-0intro
2022-01-18T21:38:42-71888fe/plan9-amd64-0intro
2022-01-18T19:03:00-46f138f/plan9-amd64-0intro
2022-01-18T18:56:51-75bcdd5/plan9-amd64-0intro
2022-01-18T18:16:14-ef4be98/plan9-amd64-0intro
2022-01-18T18:06:50-d54f663/plan9-amd64-0intro
2022-01-18T14:56:19-5b3ebc8/plan9-amd64-0intro
2022-01-17T09:23:25-897b3da/plan9-amd64-0intro
2022-01-14T21:54:39-3b5eec9/plan9-amd64-0intro
2022-01-14T20:50:36-a99c38d/plan9-amd64-0intro
2022-01-14T20:50:32-07b995e/plan9-amd64-0intro
2022-01-14T18:36:49-4f0c32d/plan9-amd64-0intro
2022-01-14T18:36:41-1302f93/plan9-amd64-0intro
2022-01-14T18:20:45-e4a6b84/plan9-amd64-0intro
2022-01-14T17:58:14-9c6ecc4/plan9-amd64-0intro
2022-01-14T17:49:58-17b2fb1/plan9-amd64-0intro
2022-01-14T16:41:18-b41185c/plan9-amd64-0intro
2022-01-13T23:35:37-e550c30/plan9-amd64-0intro
2022-01-13T23:04:22-4f6f68e/plan9-amd64-0intro
2022-01-13T22:58:24-899d19a/plan9-amd64-0intro
2022-01-13T21:37:29-1a8b4e0/plan9-amd64-0intro
2022-01-13T20:43:56-6891d07/plan9-amd64-0intro
2022-01-13T19:35:14-3ff12a0/plan9-amd64-0intro
2022-01-13T19:25:50-ce01afe/plan9-amd64-0intro
2022-01-13T18:06:33-4fa6e33/plan9-amd64-0intro
2022-01-13T00:53:53-2423912/plan9-amd64-0intro
2022-01-12T19:47:40-f005df8/plan9-amd64-0intro
2022-01-12T19:03:12-a4b6fc7/plan9-amd64-0intro
2022-01-12T19:03:11-deb4580/plan9-amd64-0intro
2022-01-12T19:03:10-68b3d36/plan9-amd64-0intro
2022-01-12T16:50:38-83bfdb6/plan9-amd64-0intro
2022-01-12T15:53:47-6e8b7e4/plan9-amd64-0intro
2022-01-12T01:49:33-3d3f5d9/plan9-amd64-0intro
2022-01-12T01:41:46-3df4472/plan9-amd64-0intro
2022-01-12T00:12:10-9bce089/plan9-amd64-0intro
2022-01-11T22:50:23-1ee70da/plan9-amd64-0intro
2022-01-11T21:56:11-13c912d/plan9-amd64-0intro
2022-01-11T18:17:05-a20724d/plan9-amd64-0intro
2022-01-11T16:08:43-1cc3c73/plan9-amd64-0intro
2022-01-11T15:28:40-c7fa661/plan9-amd64-0intro
2022-01-10T22:49:07-4ceb5a9/plan9-arm
2022-01-06T14:02:44-f300fc2/plan9-amd64-0intro
2022-01-06T00:26:47-b5bfaf4/plan9-amd64-0intro

[0intro]

The plan9-amd64 builder was running out of memory while executing GOMAXPROCS=2 go test runtime -cpu=1,2,4 -quick.
I think it should be better now.

@bcmills Thank you for your time tracking the various Plan 9 issues. It's very appreciated.

[gopherbot]

Change https://go.dev/cl/408698 mentions this issue: dashboard: add known issues for plan9-386, plan9-386-0intro, and plan9-amd64-0intro