Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime,cmd/compile: frequent memory corruption on NetBSD and OpenBSD since 2021-10-11 #49209

Closed
bcmills opened this issue Oct 28, 2021 · 45 comments
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-NetBSD
Milestone

Comments

@bcmills bcmills changed the title runtime,cmd/compile: frequent failures with "freeIndex is not valid" since 2021-10-18 runtime,cmd/compile: frequent failures with "freeIndex is not valid" on NetBSD since 2021-10-18 Oct 28, 2021
@bcmills
Copy link
Member Author

bcmills commented Oct 28, 2021

CC @bsiegert @coypoop

@bcmills
Copy link
Member Author

bcmills commented Oct 28, 2021

This appears to be a regression in Go 1.18.

Since NetBSD is not a first-class port, this doesn't necessarily block the 1.18 release — however, if the regression remains at the time of the release it at least needs a clear writeup in the release notes. (That part, at least, is a release-blocker.)

@bcmills bcmills added NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. release-blocker labels Oct 28, 2021
@bcmills bcmills added this to the Go1.18 milestone Oct 28, 2021
@bcmills
Copy link
Member Author

bcmills commented Oct 28, 2021

@jeremyfaller, any idea who on the Go team might have insight into what broke on the 18th specifically?

@coypoop
Copy link
Contributor

coypoop commented Oct 29, 2021

I'm not sure this error is entirely a regression, as I've seen this with pre-Go-1.18 on NetBSD. But perhaps something is making it much more frequent.

@jeremyfaller
Copy link
Contributor

jeremyfaller commented Oct 29, 2021

Did a little digging on Oct17-18, and didn't see much that looked suspicious.

pinging @mknyszek

@bcmills
Copy link
Member Author

bcmills commented Nov 2, 2021

There have been a lot of other failures since then that look like arbitrary memory corruption in the runtime.
Whatever this is, it looks like it's got all of the NetBSD builders in pretty bad shape.
(Silver lining: the failure rate is high enough that this should be feasible to bisect!)

greplogs --dashboard -md -l -e '(?ms)\Anetbsd.*fatal error:' --since=2021-10-01

2021-11-02T15:15:09-4ff95d6-da7173a/netbsd-amd64-9_0
2021-11-02T06:25:39-30b2efe-088bb4b/netbsd-amd64-9_0
2021-11-02T06:19:33-4633b2d/netbsd-amd64-9_0
2021-11-02T05:43:05-a97c527/netbsd-amd64-9_0
2021-11-02T05:43:05-835dae6-a97c527/netbsd-386-9_0
2021-11-02T05:35:11-6f327f7/netbsd-386-9_0
2021-11-02T03:09:01-a45457d/netbsd-amd64-9_0
2021-11-02T00:12:17-4ff95d6-81fea0b/netbsd-386-9_0
2021-11-01T22:55:50-02e5913/netbsd-amd64-9_0
2021-11-01T21:27:26-4ff95d6-631b567/netbsd-386-9_0
2021-11-01T21:27:26-2c98350-631b567/netbsd-amd64-9_0
2021-11-01T19:56:50-ee08195-5f1bcd1/netbsd-386-9_0
2021-11-01T18:53:06-9967326-b5a5b7b/netbsd-amd64-9_0
2021-11-01T16:31:02-2bcf1c0/netbsd-386-9_0
2021-11-01T15:55:25-e2e910e/netbsd-386-9_0
2021-11-01T15:20:08-b7d65fb-82f902a/netbsd-amd64-9_0
2021-11-01T14:32:47-30b2efe-4a84298/netbsd-386-9_0
2021-11-01T13:54:03-42daa65-4a84298/netbsd-386-9_0
2021-11-01T13:12:37-611d5d6-4056934/netbsd-amd64-9_0
2021-11-01T13:12:37-4ff95d6-4056934/netbsd-386-9_0
2021-11-01T02:47:30-fde4cc2/netbsd-386-9_0
2021-11-01T02:47:30-fde4cc2/netbsd-amd64-9_0
2021-10-31T18:39:05-12ab535-89c5270/netbsd-amd64-9_0
2021-10-31T17:52:41-4ff95d6-3fa9ee4/netbsd-386-9_0
2021-10-30T18:30:34-ba495a6-6113dac/netbsd-386-9_0
2021-10-30T18:30:34-6113dac/netbsd-amd64-9_0
2021-10-30T18:30:34-351c04c-6113dac/netbsd-386-9_0
2021-10-30T16:47:17-30b5d63/netbsd-386-9_0
2021-10-30T16:46:47-d1dceaf/netbsd-386-9_0
2021-10-30T16:45:25-e39b854/netbsd-386-9_0
2021-10-30T16:08:13-b3129d9-5d6d9f5/netbsd-386-9_0
2021-10-30T00:47:26-a6c6f4b-d19c5bd/netbsd-amd64-9_0
2021-10-30T00:47:26-a6c6f4b-5d6d9f5/netbsd-386-9_0
2021-10-30T00:47:26-a6c6f4b-5d6d9f5/netbsd-amd64-9_0
2021-10-30T00:47:26-a6c6f4b-4a84298/netbsd-386-9_0
2021-10-29T23:15:00-a2be0cd-8dfb447/netbsd-amd64-9_0
2021-10-29T22:29:31-c96bc14-c812b97/netbsd-386-9_0
2021-10-29T22:27:54-71e6ab8/netbsd-amd64-9_0
2021-10-29T22:27:26-3571ab5/netbsd-386-9_0
2021-10-29T21:19:39-a2be0cd-d19c5bd/netbsd-amd64-9_0
2021-10-29T21:19:39-a2be0cd-994049a/netbsd-amd64-9_0
2021-10-29T19:42:51-540bb53-4a84298/netbsd-amd64-9_0
2021-10-29T19:42:43-4ff95d6-d19c5bd/netbsd-386-9_0
2021-10-29T19:30:03-4592933/netbsd-386-9_0
2021-10-29T18:35:01-8e112a7/netbsd-amd64-9_0
2021-10-29T18:23:03-413672f/netbsd-386-9_0
2021-10-29T17:44:15-d8fc7f7/netbsd-386-9_0
2021-10-29T17:44:15-d8fc7f7/netbsd-amd64-9_0
2021-10-29T04:21:07-d6a9af8-f288526/netbsd-386-9_0
2021-10-29T04:21:07-d6a9af8-f288526/netbsd-amd64-9_0
2021-10-29T02:12:05-d6a9af8-d3d8852/netbsd-amd64-9_0
2021-10-28T22:38:00-af05d8b/netbsd-amd64-9_0
2021-10-28T20:43:39-b954024/netbsd-386-9_0
2021-10-28T20:43:39-6944b10-b954024/netbsd-amd64-9_0
2021-10-28T20:27:44-5e21728/netbsd-386-9_0
2021-10-28T19:04:41-4e1c44d-18b9702/netbsd-386-9_0
2021-10-28T18:48:25-e7eb6f6-9004433/netbsd-386-9_0
2021-10-28T18:17:57-f229e70/netbsd-386-9_0
2021-10-28T18:01:34-03971e3-18b9702/netbsd-386-9_0
2021-10-28T16:54:58-c8ad2e1-6bd0e7f/netbsd-amd64-9_0
2021-10-28T15:31:34-8de2a7f-8c9c148/netbsd-amd64-9_0
2021-10-28T03:44:12-103d89b-2ff1074/netbsd-amd64-9_0
2021-10-28T03:35:34-b8f928b/netbsd-amd64-9_0
2021-10-28T01:15:26-b2fe2eb/netbsd-386-9_0
2021-10-28T01:15:26-103d89b-b2fe2eb/netbsd-386-9_0
2021-10-27T21:37:54-7b0b504-749f6e9/netbsd-amd64-9_0
2021-10-27T21:34:46-de1abf7/netbsd-386-9_0
2021-10-27T21:34:46-de1abf7/netbsd-amd64-9_0
2021-10-27T20:29:07-51be206/netbsd-386-9_0
2021-10-27T20:25:06-bbc0595/netbsd-amd64-9_0
2021-10-27T20:24:34-56dcf97/netbsd-386-9_0
2021-10-27T20:03:17-7b0b504-68bd512/netbsd-386-9_0
2021-10-27T19:33:43-7b0b504-11b64b4/netbsd-amd64-9_0
2021-10-27T19:33:07-1b02aa2-514ebae/netbsd-386-9_0
2021-10-27T18:08:49-08f2e89-bb49eb3/netbsd-amd64-9_0
2021-10-27T17:22:43-714a425-11b64b4/netbsd-386-9_0
2021-10-27T16:59:43-c0ac39c/netbsd-amd64-9_0
2021-10-27T16:39:27-94870a3-4f73fd0/netbsd-386-9_0
2021-10-27T13:12:49-e1e2965-cfb5321/netbsd-amd64-9_0
2021-10-27T13:12:49-d418f37-cfb5321/netbsd-386-9_0
2021-10-27T06:23:35-5786a54/netbsd-386-9_0
2021-10-27T05:33:58-ca5f65d/netbsd-386-9_0
2021-10-26T22:24:36-591e12a-80be4a4/netbsd-386-9_0
2021-10-26T22:05:53-80be4a4/netbsd-amd64-9_0
2021-10-26T22:00:36-26dbf47-86f6bf1/netbsd-amd64-9_0
2021-10-26T21:32:57-86f6bf1/netbsd-amd64-9_0
2021-10-26T21:17:38-03971e3-091db63/netbsd-386-9_0
2021-10-26T18:40:06-9626607-11b64b4/netbsd-386-9_0
2021-10-26T18:40:06-9626607-11b64b4/netbsd-amd64-9_0
2021-10-26T18:33:39-244f92e-11b64b4/netbsd-amd64-9_0
2021-10-26T17:10:45-2c98350-76cef81/netbsd-386-9_0
2021-10-26T15:46:18-c4ead46-1b2362b/netbsd-386-9_0
2021-10-26T15:20:53-903c757-1b2362b/netbsd-amd64-9_0
2021-10-26T11:58:05-1e2820a/netbsd-386-9_0
2021-10-26T11:58:05-1e2820a/netbsd-amd64-9_0
2021-10-26T02:02:46-903c757-11b64b4/netbsd-amd64-9_0
2021-10-26T01:22:47-adfb85b/netbsd-386-9_0
2021-10-26T01:22:47-adfb85b/netbsd-amd64-9_0
2021-10-26T00:38:52-22652b4-0ec1b62/netbsd-386-9_0
2021-10-26T00:30:24-22652b4-b36b001/netbsd-amd64-9_0
2021-10-25T22:17:34-fd2f4b5/netbsd-386-9_0
2021-10-25T20:40:31-5a40697-2ac3bdf/netbsd-amd64-9_0
2021-10-19T18:56:08-07e5527/netbsd-386-9_0
2021-10-19T08:44:22-98f6e03-1e49210/netbsd-386-9_0
2021-10-19T07:45:46-ee92daa/netbsd-amd64-9_0
2021-10-19T07:45:46-98f6e03-ee92daa/netbsd-386-9_0
2021-10-19T07:45:46-98f6e03-ee92daa/netbsd-amd64-9_0
2021-10-18T21:57:36-98f6e03-3befaf0/netbsd-amd64-9_0
2021-10-18T21:52:05-98f6e03-425db64/netbsd-386-9_0
2021-10-18T21:52:05-425db64/netbsd-amd64-9_0
2021-10-18T20:39:22-267abbe/netbsd-amd64-9_0
2021-10-18T19:46:27-394a1ad/netbsd-386-9_0
2021-10-14T07:18:59-1349c6e/netbsd-amd64-9_0
2021-10-12T21:14:34-03971e3-c580180/netbsd-amd64-9_0
2021-10-11T15:48:29-3b03d30-2478b6f/netbsd-amd64-9_0

@bcmills bcmills changed the title runtime,cmd/compile: frequent failures with "freeIndex is not valid" on NetBSD since 2021-10-18 runtime,cmd/compile: frequent memory corruption on NetBSD since 2021-10-11 Nov 2, 2021
@mknyszek
Copy link
Contributor

mknyszek commented Nov 8, 2021

On it.

@mknyszek mknyszek self-assigned this Nov 8, 2021
@mknyszek
Copy link
Contributor

mknyszek commented Nov 9, 2021

goswarm $ GOROOT=$HOME/toolchain ./goswarm -match 'freeIndex is not valid' netbsd-386-9_0 go/src/all.bash
2021/11/09 00:09:59 Created instance user-mknyszek-netbsd-386-9_0-1...
2021/11/09 00:10:02 Created instance user-mknyszek-netbsd-386-9_0-2...
2021/11/09 00:10:03 Created instance user-mknyszek-netbsd-386-9_0-3...
2021/11/09 00:10:03 Created instance user-mknyszek-netbsd-386-9_0-4...
2021/11/09 00:10:05 Created instance user-mknyszek-netbsd-386-9_0-5...
2021/11/09 00:10:05 Created instance user-mknyszek-netbsd-386-9_0-6...
2021/11/09 00:10:07 Created instance user-mknyszek-netbsd-386-9_0-7...
2021/11/09 00:10:13 Pushed to user-mknyszek-netbsd-386-9_0-1.
2021/11/09 00:10:13 Running command on user-mknyszek-netbsd-386-9_0-1.
2021/11/09 00:10:15 Created instance user-mknyszek-netbsd-386-9_0-8...
2021/11/09 00:10:15 Created instance user-mknyszek-netbsd-386-9_0-9...
2021/11/09 00:10:15 Pushed to user-mknyszek-netbsd-386-9_0-2.
2021/11/09 00:10:15 Running command on user-mknyszek-netbsd-386-9_0-2.
2021/11/09 00:10:17 Pushed to user-mknyszek-netbsd-386-9_0-3.
2021/11/09 00:10:17 Running command on user-mknyszek-netbsd-386-9_0-3.
2021/11/09 00:10:17 Pushed to user-mknyszek-netbsd-386-9_0-4.
2021/11/09 00:10:17 Running command on user-mknyszek-netbsd-386-9_0-4.
2021/11/09 00:10:19 Pushed to user-mknyszek-netbsd-386-9_0-6.
2021/11/09 00:10:19 Running command on user-mknyszek-netbsd-386-9_0-6.
2021/11/09 00:10:19 Pushed to user-mknyszek-netbsd-386-9_0-5.
2021/11/09 00:10:19 Running command on user-mknyszek-netbsd-386-9_0-5.
2021/11/09 00:10:20 Pushed to user-mknyszek-netbsd-386-9_0-7.
2021/11/09 00:10:20 Running command on user-mknyszek-netbsd-386-9_0-7.
2021/11/09 00:10:28 Pushed to user-mknyszek-netbsd-386-9_0-8.
2021/11/09 00:10:28 Running command on user-mknyszek-netbsd-386-9_0-8.
2021/11/09 00:10:28 Pushed to user-mknyszek-netbsd-386-9_0-9.
2021/11/09 00:10:28 Running command on user-mknyszek-netbsd-386-9_0-9.
2021/11/09 00:10:29 Created instance user-mknyszek-netbsd-386-9_0-10...
2021/11/09 00:10:42 Pushed to user-mknyszek-netbsd-386-9_0-10.
2021/11/09 00:10:42 Running command on user-mknyszek-netbsd-386-9_0-10.
2021/11/09 00:21:43 Running command on user-mknyszek-netbsd-386-9_0-2.
2021/11/09 00:31:28 Running command on user-mknyszek-netbsd-386-9_0-1.
2021/11/09 00:31:29 Running command on user-mknyszek-netbsd-386-9_0-6.
2021/11/09 00:31:33 Running command on user-mknyszek-netbsd-386-9_0-8.
2021/11/09 00:31:54 Running command on user-mknyszek-netbsd-386-9_0-7.
2021/11/09 00:31:58 Running command on user-mknyszek-netbsd-386-9_0-3.
2021/11/09 00:31:58 Running command on user-mknyszek-netbsd-386-9_0-4.
2021/11/09 00:32:38 Running command on user-mknyszek-netbsd-386-9_0-10.
2021/11/09 00:33:05 Running command on user-mknyszek-netbsd-386-9_0-9.
2021/11/09 00:40:36 Running command on user-mknyszek-netbsd-386-9_0-5.
2021/11/09 00:42:10 Running command on user-mknyszek-netbsd-386-9_0-2.
2021/11/09 00:52:34 Running command on user-mknyszek-netbsd-386-9_0-1.
2021/11/09 00:52:37 Running command on user-mknyszek-netbsd-386-9_0-6.
2021/11/09 00:53:15 Running command on user-mknyszek-netbsd-386-9_0-7.
2021/11/09 00:53:18 Running command on user-mknyszek-netbsd-386-9_0-3.
2021/11/09 00:53:36 Running command on user-mknyszek-netbsd-386-9_0-4.

This is more difficult to reproduce than I anticipated...

@mknyszek
Copy link
Contributor

mknyszek commented Nov 9, 2021

My gomote swarm ran all night and I got nowhere, unfortunately.

@bcmills
Copy link
Member Author

bcmills commented Nov 9, 2021

Looking more carefully at the fatal error failures, I notice that some of them are actually on the release branches.

For example, the 2021-10-28T18:48:25-e7eb6f6-9004433/netbsd-386-9_0 failure is on release-branch.go1.16.

@golang/release, was there a change to the builder images on or around Oct. 11 that could explain these failures? (If so, is there an older image we can restore, or a platform expert who can, say, try bisecting the kernel?)

@bcmills
Copy link
Member Author

bcmills commented Nov 9, 2021

@mknyszek, the freeIndex is not valid failure mode seems to be rarer than some of the other failure modes. Would it make sense to cast a wider net for fatal errors that lead to core files?

@mknyszek
Copy link
Contributor

mknyszek commented Nov 9, 2021

Ah OK. I'll focus on other failure modes.

@bcmills
Copy link
Member Author

bcmills commented Nov 9, 2021

Going back further in the history seems to show:

greplogs --dashboard -md -l -e '(?ms)\Anetbsd-(?:amd64|386).*fatal error:' --since=2021-01-01

2021-11-09T01:45:54-fafc446-5344dca/netbsd-386-9_0
2021-11-08T22:05:00-3a5865c-67e2294/netbsd-amd64-9_0
2021-11-08T21:52:47-a5321bf-955f9f5/netbsd-386-9_0
2021-11-08T21:45:43-cc49178/netbsd-386-9_0
2021-11-08T21:45:43-cc49178/netbsd-amd64-9_0
2021-11-08T21:45:43-39ade5b-cc49178/netbsd-386-9_0
2021-11-08T21:20:25-a5321bf-23991f5/netbsd-386-9_0
2021-11-08T21:20:16-0edd7b8-f58c78a/netbsd-386-9_0
2021-11-08T21:19:47-17472e0-f58c78a/netbsd-amd64-9_0
2021-11-08T20:46:37-c8e8de6-f58c78a/netbsd-amd64-9_0
2021-11-08T20:32:21-b766c28-830b393/netbsd-386-9_0
2021-11-08T20:32:21-b766c28-23991f5/netbsd-386-9_0
2021-11-08T18:39:25-2cdcc60-f58c78a/netbsd-386-9_0
2021-11-08T18:39:25-2cdcc60-23991f5/netbsd-amd64-9_0
2021-11-08T18:06:16-7ae8057-5e64755/netbsd-386-9_0
2021-11-08T18:06:16-5e64755/netbsd-386-9_0
2021-11-08T18:06:16-39ade5b-5e64755/netbsd-amd64-9_0
2021-11-08T17:07:45-6635138-7bda349/netbsd-386-9_0
2021-11-08T16:15:01-30b2efe-759eaa2/netbsd-386-9_0
2021-11-08T14:49:56-18b340f-ab31dbc/netbsd-386-9_0
2021-11-07T04:56:11-85493d5/netbsd-386-9_0
2021-11-06T19:41:15-036812b-61d789d/netbsd-386-9_0
2021-11-06T16:43:43-39ade5b-3544082/netbsd-386-9_0
2021-11-06T16:43:43-0c60b7c-3544082/netbsd-amd64-9_0
2021-11-06T13:10:06-1f47c86-4f083c7/netbsd-386-9_0
2021-11-06T10:24:44-0c60b7c-f19e400/netbsd-amd64-9_0
2021-11-06T00:29:44-a66bbe2-b74f2ef/netbsd-386-9_0
2021-11-06T00:29:44-a07c284-b74f2ef/netbsd-amd64-9_0
2021-11-06T00:29:44-0c60b7c-b74f2ef/netbsd-amd64-9_0
2021-11-05T23:32:57-b8b8e7f-09e8de7/netbsd-386-9_0
2021-11-05T22:57:08-a66bbe2-09e8de7/netbsd-amd64-9_0
2021-11-05T22:54:47-ba79c1e/netbsd-386-9_0
2021-11-05T22:54:47-ba79c1e/netbsd-amd64-9_0
2021-11-05T22:30:17-03971e3-b07c41d/netbsd-386-9_0
2021-11-05T22:27:21-8f923a4/netbsd-amd64-9_0
2021-11-05T22:26:07-4ab7496-d3a80c7/netbsd-386-9_0
2021-11-05T22:26:07-39ade5b-d3a80c7/netbsd-amd64-9_0
2021-11-05T22:23:52-35a5881/netbsd-386-9_0
2021-11-05T21:48:25-a66bbe2-6b223e8/netbsd-amd64-9_0
2021-11-05T21:48:25-4ab7496-6b223e8/netbsd-386-9_0
2021-11-05T21:27:34-7aed6dd/netbsd-386-9_0
2021-11-05T21:13:38-39ade5b-091948a/netbsd-amd64-9_0
2021-11-05T20:06:27-a7b6526-7be227c/netbsd-386-9_0
2021-11-05T20:06:27-a07c284-7be227c/netbsd-386-9_0
2021-11-05T20:06:27-7be227c/netbsd-amd64-9_0
2021-11-05T19:48:29-c353f1b/netbsd-386-9_0
2021-11-05T19:48:29-c353f1b/netbsd-amd64-9_0
2021-11-05T19:14:22-a07c284-fa16efb/netbsd-amd64-9_0
2021-11-05T19:01:13-93bab8a/netbsd-386-9_0
2021-11-05T18:20:07-a66bbe2-53bab19/netbsd-386-9_0
2021-11-05T18:20:07-a07c284-53bab19/netbsd-386-9_0
2021-11-05T17:52:30-df18377/netbsd-amd64-9_0
2021-11-05T17:47:28-6f32d20/netbsd-386-9_0
2021-11-05T17:47:28-6f32d20/netbsd-amd64-9_0
2021-11-05T17:39:43-0bc98b3/netbsd-amd64-9_0
2021-11-05T17:17:30-a66bbe2-62c6ff4/netbsd-amd64-9_0
2021-11-05T17:17:30-62c6ff4/netbsd-amd64-9_0
2021-11-05T17:17:30-39ade5b-62c6ff4/netbsd-386-9_0
2021-11-05T17:08:36-1c6f3cc-3796df1/netbsd-386-9_0
2021-11-05T16:54:01-a66bbe2-3796df1/netbsd-386-9_0
2021-11-05T16:54:01-a07c284-3796df1/netbsd-386-9_0
2021-11-05T16:51:14-c58417b/netbsd-amd64-9_0
2021-11-05T07:00:05-ce13745-6fefb7f/netbsd-386-9_0
2021-11-05T05:30:39-b68c02e/netbsd-amd64-9_0
2021-11-05T05:29:10-3b5add5/netbsd-amd64-9_0
2021-11-05T04:20:33-089bfa5-0a5ca24/netbsd-386-9_0
2021-11-05T00:52:06-3839b60/netbsd-386-9_0
2021-11-04T23:35:26-ce13745-256a8fc/netbsd-amd64-9_0
2021-11-04T23:35:26-b76863e-256a8fc/netbsd-386-9_0
2021-11-04T21:53:05-ce13745-76c48e9/netbsd-386-9_0
2021-11-04T21:53:05-37ea4aa-76c48e9/netbsd-amd64-9_0
2021-11-04T21:42:40-37ea4aa-23991f5/netbsd-386-9_0
2021-11-04T21:42:40-37ea4aa-156abe5/netbsd-386-9_0
2021-11-04T21:41:49-39ade5b-156abe5/netbsd-386-9_0
2021-11-04T20:42:35-1f9dce7/netbsd-386-9_0
2021-11-04T20:31:02-39ade5b-978e39e/netbsd-386-9_0
2021-11-04T20:24:01-99699d1/netbsd-386-9_0
2021-11-04T20:01:11-fc5e8cd/netbsd-amd64-9_0
2021-11-04T19:34:33-5af93a2/netbsd-amd64-9_0
2021-11-04T18:22:03-84e69e7-b2149ac/netbsd-386-9_0
2021-11-04T18:22:03-39ade5b-b2149ac/netbsd-386-9_0
2021-11-04T17:07:48-5772877/netbsd-386-9_0
2021-11-04T17:07:48-5772877/netbsd-amd64-9_0
2021-11-04T16:36:19-84e69e7-f934b83/netbsd-386-9_0
2021-11-04T15:43:59-84e69e7-6ba68a0/netbsd-386-9_0
2021-11-04T14:54:46-4a448f8-00d6d20/netbsd-386-9_0
2021-11-04T14:24:17-d7068e7-f58c78a/netbsd-386-9_0
2021-11-04T14:17:18-7861aae-901bf29/netbsd-386-9_0
2021-11-04T14:11:01-84e69e7-a3f7be9/netbsd-386-9_0
2021-11-04T13:55:28-4a4e1f2-23991f5/netbsd-386-9_0
2021-11-04T13:55:28-39ade5b-23991f5/netbsd-386-9_0
2021-11-04T13:55:28-39ade5b-23991f5/netbsd-amd64-9_0
2021-11-04T13:55:28-30b2efe-23991f5/netbsd-386-9_0
2021-11-04T13:55:28-089bfa5-23991f5/netbsd-386-9_0
2021-11-04T13:55:24-4a4e1f2-f58c78a/netbsd-amd64-9_0
2021-11-04T13:55:24-036812b-f58c78a/netbsd-386-9_0
2021-11-04T07:05:31-84e69e7-2622235/netbsd-386-9_0
2021-11-04T07:05:31-84e69e7-2622235/netbsd-amd64-9_0
2021-11-04T07:05:31-39ade5b-2622235/netbsd-386-9_0
2021-11-04T02:57:53-4a4e1f2-2cf85b1/netbsd-386-9_0
2021-11-04T02:57:53-2cf85b1/netbsd-amd64-9_0
2021-11-04T02:57:48-5fd0c49/netbsd-386-9_0
2021-11-04T02:57:48-5fd0c49/netbsd-amd64-9_0
2021-11-04T02:12:52-8da7c0f-bd49a99/netbsd-amd64-9_0
2021-11-04T00:46:41-e72d715/netbsd-386-9_0
2021-11-04T00:46:41-84e69e7-e72d715/netbsd-amd64-9_0
2021-11-04T00:36:13-84e69e7-1292e21/netbsd-386-9_0
2021-11-04T00:29:42-a419f2f/netbsd-amd64-9_0
2021-11-04T00:29:42-39ade5b-a419f2f/netbsd-386-9_0
2021-11-04T00:19:43-9cf6711/netbsd-amd64-9_0
2021-11-04T00:19:43-39ade5b-9cf6711/netbsd-386-9_0
2021-11-03T20:30:17-68536fa-7f2463c/netbsd-386-9_0
2021-11-03T19:32:33-74f99d0/netbsd-386-9_0
2021-11-03T19:32:33-74f99d0/netbsd-amd64-9_0
2021-11-03T18:37:22-3b1a674-cfd016d/netbsd-amd64-9_0
2021-11-03T18:37:22-3a5865c-cfd016d/netbsd-amd64-9_0
2021-11-03T18:37:22-2c98350-cfd016d/netbsd-amd64-9_0
2021-11-03T16:57:50-ffed863-88407a8/netbsd-amd64-9_0
2021-11-03T16:57:50-3b1a674-88407a8/netbsd-amd64-9_0
2021-11-03T16:57:44-3a5865c-b212ba6/netbsd-386-9_0
2021-11-03T15:51:25-3b1a674-6458b2e/netbsd-386-9_0
2021-11-03T15:47:47-c143661/netbsd-386-9_0
2021-11-03T05:00:00-39ade5b-519c0a2/netbsd-386-9_0
2021-11-03T00:07:03-d6f7203/netbsd-386-9_0
2021-11-03T00:07:03-d6f7203/netbsd-amd64-9_0
2021-11-02T23:44:58-a0f373c/netbsd-386-9_0
2021-11-02T23:05:17-eb63cbd/netbsd-386-9_0
2021-11-02T23:05:17-714668c-eb63cbd/netbsd-amd64-9_0
2021-11-02T22:20:40-39ade5b-2157498/netbsd-386-9_0
2021-11-02T22:20:40-2157498/netbsd-386-9_0
2021-11-02T21:21:51-9aacde2-631b567/netbsd-amd64-9_0
2021-11-02T21:21:51-9aacde2-42e6b5b/netbsd-386-9_0
2021-11-02T21:18:39-60fd3ed/netbsd-386-9_0
2021-11-02T20:59:34-6561d8c-79024cf/netbsd-386-9_0
2021-11-02T20:59:34-39ade5b-79024cf/netbsd-386-9_0
2021-11-02T20:33:56-b76863e-b29182b/netbsd-amd64-9_0
2021-11-02T19:37:42-bb4add0-1011e26/netbsd-386-9_0
2021-11-02T19:37:42-bb4add0-1011e26/netbsd-amd64-9_0
2021-11-02T18:35:35-c702f91/netbsd-386-9_0
2021-11-02T18:35:35-c702f91/netbsd-amd64-9_0
2021-11-02T18:35:29-a07c284-433ba58/netbsd-amd64-9_0
2021-11-02T18:35:29-4a4e1f2-433ba58/netbsd-386-9_0
2021-11-02T18:24:18-bb4add0-f7a95d2/netbsd-386-9_0
2021-11-02T18:22:55-bb4add0-f9cb33c/netbsd-amd64-9_0
2021-11-02T18:19:06-1ba8fdb-631b567/netbsd-amd64-9_0
2021-11-02T18:01:20-39ade5b-629ffeb/netbsd-386-9_0
2021-11-02T18:01:20-058ed05-629ffeb/netbsd-amd64-9_0
2021-11-02T17:53:48-7548327/netbsd-amd64-9_0
2021-11-02T17:31:50-39ade5b-4e7dd9f/netbsd-386-9_0
2021-11-02T17:01:01-af8aafd/netbsd-386-9_0
2021-11-02T16:57:03-c406380/netbsd-amd64-9_0
2021-11-02T16:29:07-d4902dd-f9cb33c/netbsd-386-9_0
2021-11-02T16:19:52-c96bc14-58fb05a/netbsd-386-9_0
2021-11-02T16:19:52-058ed05-58fb05a/netbsd-386-9_0
2021-11-02T16:12:28-f801da7/netbsd-amd64-9_0
2021-11-02T16:12:23-599de4b/netbsd-amd64-9_0
2021-11-02T15:54:27-058ed05-f9cb33c/netbsd-386-9_0
2021-11-02T15:54:27-058ed05-631b567/netbsd-amd64-9_0
2021-11-02T15:15:09-4ff95d6-da7173a/netbsd-amd64-9_0
2021-11-02T06:25:39-30b2efe-088bb4b/netbsd-amd64-9_0
2021-11-02T06:19:33-4633b2d/netbsd-amd64-9_0
2021-11-02T05:43:05-a97c527/netbsd-amd64-9_0
2021-11-02T05:43:05-835dae6-a97c527/netbsd-386-9_0
2021-11-02T05:35:11-6f327f7/netbsd-386-9_0
2021-11-02T03:55:19-39ade5b-6f1e9a9/netbsd-386-9_0
2021-11-02T03:09:01-a45457d/netbsd-amd64-9_0
2021-11-02T00:12:17-4ff95d6-81fea0b/netbsd-386-9_0
2021-11-01T22:55:50-02e5913/netbsd-amd64-9_0
2021-11-01T21:27:26-4ff95d6-631b567/netbsd-386-9_0
2021-11-01T21:27:26-39ade5b-631b567/netbsd-386-9_0
2021-11-01T21:27:26-2c98350-631b567/netbsd-amd64-9_0
2021-11-01T21:27:26-1f47c86-631b567/netbsd-386-9_0
2021-11-01T19:56:50-ee08195-5f1bcd1/netbsd-386-9_0
2021-11-01T18:53:06-9967326-b5a5b7b/netbsd-amd64-9_0
2021-11-01T16:31:02-39ade5b-2bcf1c0/netbsd-amd64-9_0
2021-11-01T16:31:02-2bcf1c0/netbsd-386-9_0
2021-11-01T15:55:25-e2e910e/netbsd-386-9_0
2021-11-01T15:20:08-b7d65fb-82f902a/netbsd-amd64-9_0
2021-11-01T14:32:47-30b2efe-4a84298/netbsd-386-9_0
2021-11-01T13:54:03-42daa65-4a84298/netbsd-386-9_0
2021-11-01T13:12:37-611d5d6-4056934/netbsd-amd64-9_0
2021-11-01T13:12:37-4ff95d6-4056934/netbsd-386-9_0
2021-11-01T02:47:30-fde4cc2/netbsd-386-9_0
2021-11-01T02:47:30-fde4cc2/netbsd-amd64-9_0
2021-10-31T18:39:05-39ade5b-89c5270/netbsd-386-9_0
2021-10-31T18:39:05-12ab535-89c5270/netbsd-amd64-9_0
2021-10-31T18:13:09-39ade5b-fd09e88/netbsd-amd64-9_0
2021-10-31T17:52:41-4ff95d6-3fa9ee4/netbsd-386-9_0
2021-10-31T08:29:02-39ade5b-8e3d5f0/netbsd-amd64-9_0
2021-10-30T18:30:34-ba495a6-6113dac/netbsd-386-9_0
2021-10-30T18:30:34-6113dac/netbsd-amd64-9_0
2021-10-30T18:30:34-351c04c-6113dac/netbsd-386-9_0
2021-10-30T16:47:17-30b5d63/netbsd-386-9_0
2021-10-30T16:46:47-d1dceaf/netbsd-386-9_0
2021-10-30T16:45:25-e39b854/netbsd-386-9_0
2021-10-30T16:08:13-b3129d9-5d6d9f5/netbsd-386-9_0
2021-10-30T00:47:26-a6c6f4b-d19c5bd/netbsd-amd64-9_0
2021-10-30T00:47:26-a6c6f4b-5d6d9f5/netbsd-386-9_0
2021-10-30T00:47:26-a6c6f4b-5d6d9f5/netbsd-amd64-9_0
2021-10-30T00:47:26-a6c6f4b-4a84298/netbsd-386-9_0
2021-10-29T23:15:00-a2be0cd-8dfb447/netbsd-amd64-9_0
2021-10-29T22:29:31-c96bc14-c812b97/netbsd-386-9_0
2021-10-29T22:29:31-b76863e-c812b97/netbsd-amd64-9_0
2021-10-29T22:29:31-39ade5b-c812b97/netbsd-386-9_0
2021-10-29T22:27:54-71e6ab8/netbsd-amd64-9_0
2021-10-29T22:27:26-3571ab5/netbsd-386-9_0
2021-10-29T21:19:39-a2be0cd-d19c5bd/netbsd-amd64-9_0
2021-10-29T21:19:39-a2be0cd-994049a/netbsd-amd64-9_0
2021-10-29T19:42:51-540bb53-4a84298/netbsd-amd64-9_0
2021-10-29T19:42:43-4ff95d6-d19c5bd/netbsd-386-9_0
2021-10-29T19:42:43-089bfa5-d19c5bd/netbsd-amd64-9_0
2021-10-29T19:30:03-4592933/netbsd-386-9_0
2021-10-29T18:35:01-8e112a7/netbsd-amd64-9_0
2021-10-29T18:23:03-413672f/netbsd-386-9_0
2021-10-29T17:44:15-d8fc7f7/netbsd-386-9_0
2021-10-29T17:44:15-d8fc7f7/netbsd-amd64-9_0
2021-10-29T04:21:07-d6a9af8-f288526/netbsd-386-9_0
2021-10-29T04:21:07-d6a9af8-f288526/netbsd-amd64-9_0
2021-10-29T02:16:47-089bfa5-2c7cdec/netbsd-386-9_0
2021-10-29T02:12:05-d6a9af8-d3d8852/netbsd-amd64-9_0
2021-10-29T02:12:05-036812b-d3d8852/netbsd-386-9_0
2021-10-28T22:38:00-af05d8b/netbsd-amd64-9_0
2021-10-28T21:17:17-089bfa5-f6f024f/netbsd-386-9_0
2021-10-28T20:43:39-b954024/netbsd-386-9_0
2021-10-28T20:43:39-6944b10-b954024/netbsd-amd64-9_0
2021-10-28T20:27:44-5e21728/netbsd-386-9_0
2021-10-28T19:04:41-4e1c44d-18b9702/netbsd-386-9_0
2021-10-28T18:48:25-e7eb6f6-9004433/netbsd-386-9_0
2021-10-28T18:17:57-f229e70/netbsd-386-9_0
2021-10-28T18:01:38-a07c284-9004433/netbsd-amd64-9_0
2021-10-28T18:01:38-39ade5b-9004433/netbsd-386-9_0
2021-10-28T18:01:34-03971e3-18b9702/netbsd-386-9_0
2021-10-28T16:54:58-c8ad2e1-6bd0e7f/netbsd-amd64-9_0
2021-10-28T15:35:25-39ade5b-278b9b3/netbsd-amd64-9_0
2021-10-28T15:31:34-8de2a7f-8c9c148/netbsd-amd64-9_0
2021-10-28T15:31:34-1f47c86-8c9c148/netbsd-386-9_0
2021-10-28T15:31:34-089bfa5-8c9c148/netbsd-amd64-9_0
2021-10-28T15:08:31-26ed8fd-5c98bcb/netbsd-amd64-9_0
2021-10-28T03:44:12-103d89b-2ff1074/netbsd-amd64-9_0
2021-10-28T03:35:34-b8f928b/netbsd-amd64-9_0
2021-10-28T02:35:22-39ade5b-056dfe6/netbsd-amd64-9_0
2021-10-28T01:15:26-b2fe2eb/netbsd-386-9_0
2021-10-28T01:15:26-103d89b-b2fe2eb/netbsd-386-9_0
2021-10-27T21:37:54-7b0b504-749f6e9/netbsd-amd64-9_0
2021-10-27T21:34:46-de1abf7/netbsd-386-9_0
2021-10-27T21:34:46-de1abf7/netbsd-amd64-9_0
2021-10-27T20:29:07-51be206/netbsd-386-9_0
2021-10-27T20:25:06-bbc0595/netbsd-amd64-9_0
2021-10-27T20:24:34-56dcf97/netbsd-386-9_0
2021-10-27T20:03:17-7b0b504-68bd512/netbsd-386-9_0
2021-10-27T19:34:49-a66eb64-ffd2284/netbsd-386-9_0
2021-10-27T19:33:43-7b0b504-11b64b4/netbsd-amd64-9_0
2021-10-27T19:33:07-1b02aa2-514ebae/netbsd-386-9_0
2021-10-27T18:08:49-08f2e89-bb49eb3/netbsd-amd64-9_0
2021-10-27T17:22:43-714a425-11b64b4/netbsd-386-9_0
2021-10-27T16:59:43-c0ac39c/netbsd-amd64-9_0
2021-10-27T16:39:27-94870a3-4f73fd0/netbsd-386-9_0
2021-10-27T16:39:27-39ade5b-4f73fd0/netbsd-386-9_0
2021-10-27T13:12:49-e1e2965-cfb5321/netbsd-amd64-9_0
2021-10-27T13:12:49-d418f37-cfb5321/netbsd-386-9_0
2021-10-27T08:50:27-39ade5b-bdefb77/netbsd-386-9_0
2021-10-27T06:23:35-5786a54/netbsd-386-9_0
2021-10-27T05:33:58-ca5f65d/netbsd-386-9_0
2021-10-26T22:24:36-591e12a-80be4a4/netbsd-386-9_0
2021-10-26T22:05:53-80be4a4/netbsd-amd64-9_0
2021-10-26T22:05:53-036812b-80be4a4/netbsd-amd64-9_0
2021-10-26T22:00:36-26dbf47-86f6bf1/netbsd-amd64-9_0
2021-10-26T21:32:57-86f6bf1/netbsd-amd64-9_0
2021-10-26T21:17:38-03971e3-091db63/netbsd-386-9_0
2021-10-26T20:41:32-03fcf44-3a0cd11/netbsd-amd64-9_0
2021-10-26T18:40:06-9626607-11b64b4/netbsd-386-9_0
2021-10-26T18:40:06-9626607-11b64b4/netbsd-amd64-9_0
2021-10-26T18:33:39-244f92e-11b64b4/netbsd-amd64-9_0
2021-10-26T17:10:45-2c98350-76cef81/netbsd-386-9_0
2021-10-26T15:46:18-c4ead46-1b2362b/netbsd-386-9_0
2021-10-26T15:20:53-903c757-1b2362b/netbsd-amd64-9_0
2021-10-26T15:20:53-39ade5b-1b2362b/netbsd-386-9_0
2021-10-26T15:20:53-036812b-1b2362b/netbsd-amd64-9_0
2021-10-26T14:24:17-39ade5b-283d8a3/netbsd-386-9_0
2021-10-26T14:05:47-39ade5b-a2b8c18/netbsd-386-9_0
2021-10-26T11:58:05-1e2820a/netbsd-386-9_0
2021-10-26T11:58:05-1e2820a/netbsd-amd64-9_0
2021-10-26T02:02:46-903c757-11b64b4/netbsd-amd64-9_0
2021-10-26T01:22:47-adfb85b/netbsd-386-9_0
2021-10-26T01:22:47-adfb85b/netbsd-amd64-9_0
2021-10-26T00:38:52-22652b4-0ec1b62/netbsd-386-9_0
2021-10-26T00:30:24-22652b4-b36b001/netbsd-amd64-9_0
2021-10-25T22:17:34-fd2f4b5/netbsd-386-9_0
2021-10-25T20:40:31-5a40697-2ac3bdf/netbsd-amd64-9_0
2021-10-19T18:56:08-07e5527/netbsd-386-9_0
2021-10-19T08:44:22-98f6e03-1e49210/netbsd-386-9_0
2021-10-19T07:45:46-ee92daa/netbsd-amd64-9_0
2021-10-19T07:45:46-98f6e03-ee92daa/netbsd-386-9_0
2021-10-19T07:45:46-98f6e03-ee92daa/netbsd-amd64-9_0
2021-10-18T21:57:36-98f6e03-3befaf0/netbsd-amd64-9_0
2021-10-18T21:52:05-98f6e03-425db64/netbsd-386-9_0
2021-10-18T21:52:05-425db64/netbsd-amd64-9_0
2021-10-18T20:39:22-267abbe/netbsd-amd64-9_0
2021-10-18T19:46:27-a07c284-394a1ad/netbsd-amd64-9_0
2021-10-18T19:46:27-394a1ad/netbsd-386-9_0
2021-10-14T07:18:59-1349c6e/netbsd-amd64-9_0
2021-10-12T21:14:34-03971e3-c580180/netbsd-amd64-9_0
2021-10-11T15:48:29-3b03d30-2478b6f/netbsd-amd64-9_0
2021-10-07T19:49:45-39ade5b-c580180/netbsd-386-9_0
2021-08-16T20:21:13-8d20661/netbsd-386-9_0
2021-06-21T16:58:36-16e82be/netbsd-amd64-9_0
2021-05-11T14:51:35-2db0265-04cd717/netbsd-386-9_0
2021-04-20T16:44:18-109d758/netbsd-386-9_0
2021-04-09T12:56:04-519f223/netbsd-amd64-9_0
2021-04-01T00:51:26-ec721d9/netbsd-386-9_0
2021-03-10T09:01:05-cf59850/netbsd-amd64-9_0
2021-02-19T15:31:06-fa18f22/netbsd-amd64-9_0
2021-02-19T15:30:59-01eb70e/netbsd-amd64-9_0
2021-02-19T15:30:52-49add6a/netbsd-amd64-9_0
2021-02-19T14:46:21-fce2a94/netbsd-amd64-9_0
2021-02-19T14:46:21-3c2d99c-fce2a94/netbsd-amd64-9_0
2021-02-19T08:48:55-7764ee5/netbsd-amd64-9_0
2021-02-19T01:22:14-87f425d/netbsd-amd64-9_0
2021-02-19T01:22:14-5f4716e-87f425d/netbsd-amd64-9_0
2021-02-19T01:19:01-4da0188/netbsd-amd64-9_0
2021-02-19T00:41:47-5f4716e-5f2e24e/netbsd-amd64-9_0
2021-02-19T00:41:47-5f2e24e/netbsd-amd64-9_0
2021-02-19T00:41:41-ee7038f/netbsd-amd64-9_0
2021-02-19T00:41:36-40765ff/netbsd-amd64-9_0
2021-02-19T00:41:28-b445d6e/netbsd-amd64-9_0
2021-02-19T00:41:17-b110a43/netbsd-amd64-9_0
2021-02-19T00:41:09-474d5f4/netbsd-amd64-9_0
2021-02-19T00:41:03-c7c6c11/netbsd-amd64-9_0
2021-02-19T00:40:56-3527caa/netbsd-amd64-9_0
2021-02-19T00:40:42-f6c4b4b/netbsd-amd64-9_0
2021-02-19T00:40:30-ac024a0/netbsd-amd64-9_0
2021-02-19T00:40:22-a3b97e7/netbsd-amd64-9_0
2021-02-19T00:40:16-985d087/netbsd-amd64-9_0
2021-02-19T00:40:00-0ca0551/netbsd-amd64-9_0
2021-02-19T00:05:03-0c63312/netbsd-amd64-9_0
2021-02-19T00:04:56-bb6efb9/netbsd-amd64-9_0
2021-02-19T00:04:49-a1222b7/netbsd-amd64-9_0
2021-02-19T00:04:42-1c659f2/netbsd-amd64-9_0
2021-02-19T00:04:30-b6379f1/netbsd-amd64-9_0
2021-02-19T00:04:22-09e059a/netbsd-amd64-9_0
2021-02-19T00:04:14-b19e7b5/netbsd-amd64-9_0
2021-02-19T00:04:07-5421c37/netbsd-amd64-9_0
2021-02-19T00:03:59-91cc484/netbsd-amd64-9_0
2021-02-19T00:03:52-38672d3/netbsd-amd64-9_0
2021-02-19T00:03:44-a1e9148/netbsd-amd64-9_0
2021-02-19T00:03:36-75e273f/netbsd-amd64-9_0
2021-02-19T00:03:30-76ab626/netbsd-amd64-9_0
2021-02-19T00:03:22-ece954d/netbsd-amd64-9_0
2021-02-19T00:03:14-a54f7fc/netbsd-amd64-9_0
2021-02-19T00:02:58-5ecd9e3/netbsd-amd64-9_0
2021-02-19T00:02:49-54da3ab/netbsd-amd64-9_0
2021-02-19T00:02:40-fbe74db/netbsd-amd64-9_0

The freeIndex is not valid failures have followed a similar trajectory to the memory-corruption failures in general, but have a much crisper starting point: 2021-10-18.

greplogs --dashboard -md -l -e 'freeIndex is not valid' --since=2021-01-01

2021-11-08T20:32:21-b766c28-830b393/netbsd-386-9_0
2021-11-05T23:32:57-b8b8e7f-09e8de7/netbsd-386-9_0
2021-11-05T22:26:07-4ab7496-d3a80c7/netbsd-386-9_0
2021-11-05T22:23:52-35a5881/netbsd-386-9_0
2021-11-05T19:48:29-c353f1b/netbsd-amd64-9_0
2021-11-05T18:20:07-a66bbe2-53bab19/netbsd-386-9_0
2021-11-05T17:08:36-1c6f3cc-3796df1/netbsd-386-9_0
2021-11-05T16:54:01-a66bbe2-3796df1/netbsd-386-9_0
2021-11-04T21:42:40-37ea4aa-23991f5/netbsd-386-9_0
2021-11-04T21:42:40-37ea4aa-156abe5/netbsd-386-9_0
2021-11-04T18:22:03-84e69e7-b2149ac/netbsd-386-9_0
2021-11-04T15:43:59-84e69e7-6ba68a0/netbsd-386-9_0
2021-11-04T14:24:17-d7068e7-f58c78a/netbsd-386-9_0
2021-11-04T14:11:01-84e69e7-a3f7be9/netbsd-386-9_0
2021-11-04T00:36:13-84e69e7-1292e21/netbsd-386-9_0
2021-11-04T00:19:43-9cf6711/netbsd-amd64-9_0
2021-11-03T00:07:03-d6f7203/netbsd-386-9_0
2021-11-02T23:05:17-714668c-eb63cbd/netbsd-amd64-9_0
2021-11-02T20:59:34-6561d8c-79024cf/netbsd-386-9_0
2021-11-02T19:37:42-bb4add0-1011e26/netbsd-386-9_0
2021-11-02T19:37:42-1011e26/openbsd-386-68
2021-11-02T18:24:18-bb4add0-f7a95d2/netbsd-386-9_0
2021-11-02T15:54:27-058ed05-f9cb33c/netbsd-386-9_0
2021-11-01T21:27:26-4ff95d6-631b567/netbsd-386-9_0
2021-11-01T15:20:08-b7d65fb-82f902a/netbsd-amd64-9_0
2021-11-01T13:54:03-42daa65-4a84298/netbsd-386-9_0
2021-11-01T02:47:30-fde4cc2/netbsd-amd64-9_0
2021-10-30T18:30:34-351c04c-6113dac/netbsd-386-9_0
2021-10-30T00:47:26-a6c6f4b-5d6d9f5/netbsd-amd64-9_0
2021-10-30T00:47:26-a6c6f4b-4a84298/netbsd-386-9_0
2021-10-29T19:30:03-4592933/netbsd-386-9_0
2021-10-29T17:44:15-d8fc7f7/netbsd-386-9_0
2021-10-29T04:21:07-d6a9af8-f288526/netbsd-386-9_0
2021-10-29T04:21:07-d6a9af8-f288526/netbsd-amd64-9_0
2021-10-29T02:12:05-d6a9af8-d3d8852/netbsd-amd64-9_0
2021-10-29T02:12:05-036812b-d3d8852/netbsd-386-9_0
2021-10-28T22:38:00-af05d8b/netbsd-amd64-9_0
2021-10-28T19:04:41-4e1c44d-18b9702/netbsd-386-9_0
2021-10-28T18:17:57-f229e70/netbsd-386-9_0
2021-10-28T18:01:34-03971e3-18b9702/netbsd-386-9_0
2021-10-28T01:15:26-103d89b-b2fe2eb/netbsd-386-9_0
2021-10-27T20:03:17-7b0b504-68bd512/netbsd-386-9_0
2021-10-27T16:39:27-94870a3-4f73fd0/netbsd-386-9_0
2021-10-27T13:12:49-d418f37-cfb5321/netbsd-386-9_0
2021-10-27T06:23:35-5786a54/netbsd-386-9_0
2021-10-27T05:33:58-ca5f65d/netbsd-386-9_0
2021-10-26T22:24:36-591e12a-80be4a4/netbsd-386-9_0
2021-10-26T22:05:53-80be4a4/netbsd-amd64-9_0
2021-10-26T18:40:06-9626607-11b64b4/netbsd-386-9_0
2021-10-26T15:46:18-c4ead46-1b2362b/netbsd-386-9_0
2021-10-19T07:45:46-98f6e03-ee92daa/netbsd-386-9_0
2021-10-18T21:52:05-98f6e03-425db64/netbsd-386-9_0
2021-10-18T21:52:05-425db64/netbsd-amd64-9_0
2021-04-01T15:50:43-45ca9ef/linux-arm-scaleway
2021-04-01T01:26:29-5f646f0/linux-arm-scaleway
2021-04-01T00:51:26-ec721d9/linux-arm-scaleway
2021-04-01T00:51:24-1f29e69/linux-arm-scaleway
2021-04-01T00:51:23-3304b22/linux-arm-scaleway
2021-03-31T20:21:57-5d6581d/linux-arm-scaleway

@mknyszek
Copy link
Contributor

mknyszek commented Nov 16, 2021

Got a core dump... for a go1.4 failure. I feel like I'm cursed.

Building Go cmd/dist using /tmp/workdir/go1.4. (devel +0b3b511 Tue Dec 5 18:44:31 2017 +0000 netbsd/386)
Building Go toolchain1 using /tmp/workdir/go1.4.
fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x1f8 pc=0x80553c7]

runtime stack:
runtime.throw(0x8428440, 0x2a)
	/home/bradfitz/go-netbsd-386-bootstrap/src/runtime/panic.go:616 +0x6b fp=0x1893bec8 sp=0x1893bebc pc=0x806e24b
runtime.sigpanic()
	/home/bradfitz/go-netbsd-386-bootstrap/src/runtime/signal_unix.go:366 +0x230 fp=0x1893beec sp=0x1893bec8 pc=0x8081480
runtime.unlock(0x86290b0)
	/home/bradfitz/go-netbsd-386-bootstrap/src/runtime/lock_sema.go:107 +0x97 fp=0x1893bf08 sp=0x1893beec pc=0x80553c7
runtime.goschedImpl(0x18960fc0)
	/home/bradfitz/go-netbsd-386-bootstrap/src/runtime/proc.go:2606 +0xcb fp=0x1893bf18 sp=0x1893bf08 pc=0x8074c7b
runtime.gopreempt_m(0x18960fc0)
	/home/bradfitz/go-netbsd-386-bootstrap/src/runtime/proc.go:2636 +0x26 fp=0x1893bf20 sp=0x1893bf18 pc=0x8074d36
runtime.newstack()
	/home/bradfitz/go-netbsd-386-bootstrap/src/runtime/stack.go:1039 +0x213 fp=0x1893bfd8 sp=0x1893bf20 pc=0x8084393
runtime.morestack()
	/home/bradfitz/go-netbsd-386-bootstrap/src/runtime/asm_386.s:536 +0x70 fp=0x1893bfdc sp=0x1893bfd8 pc=0x8092b50

@mknyszek
Copy link
Contributor

mknyszek commented Nov 16, 2021

Got 2 crashes in one go: a freeIndex crash (!) and a scanstack - bad status crash. I also got a whole bunch of cores.

@mknyszek
Copy link
Contributor

mknyszek commented Nov 16, 2021

OK so I can't seem to find the core for the scanstack issue, but the freeIndex issue is kind of interesting. First let me say that I'm pretty confident nothing is off here -- all the pointers line up with what's in the stack trace, nothing's inconsistent so far.

So, the check that's failing has code that looks like

 871     s = c.alloc[spc]
 872     shouldhelpgc = false
 873     freeIndex := s.nextFreeIndex()
 874     if freeIndex == s.nelems {
 875         // The span is full.
 876         if uintptr(s.allocCount) != s.nelems {
 877             println("runtime: s.allocCount=", s.allocCount, "s.nelems=", s.nelems)
 878             throw("s.allocCount != s.nelems && freeIndex == s.nelems")
 879         }
 880         c.refill(spc)
 881         shouldhelpgc = true
 882         s = c.alloc[spc]                                                                     
 883
 884         freeIndex = s.nextFreeIndex()
 885     }
 886
 887     if freeIndex >= s.nelems {
 888         throw("freeIndex is not valid")
 889     }

I accessed the relevant mspan in the core dump by: finding the crashing goroutine by goid, looking up its p, accessing that p's mcache, and then picking out the span class from the backtrace (note that this is reliable because that value never changes for the duration of nextFree, also it lines up with the size passed to mallocgc: span class 0x49 (size class 36) for size 0x600). Note also that s.nextFreeIndex always writes to mspan.freeindex before returning.

When I looked at s.nelems and s.freeindex, I found the values 5 and 0 respectively. That should not be causing the check to fire.

If all this makes sense, then I think that means what we have here is some kind of stack corruption. Specifically, the value of freeIndex on the stack is getting corrupted. Unfortunately I can't figure out how to coax gdb into letting me inspect an off-thread stack and get into that frame to print locals. That would confirm it for sure.

@bcmills
Copy link
Member Author

bcmills commented Nov 17, 2021

Looks like this also affects OpenBSD:

greplogs --dashboard -md -l -e '(?ms)\Aopenbsd-.*freeIndex is not valid'

2021-11-17T04:31:22-f384c70/openbsd-386-70
2021-11-15T23:55:09-9e13a88/openbsd-386-70
2021-11-10T19:35:55-03971e3-b954f58/openbsd-386-70
2021-11-02T19:37:42-1011e26/openbsd-386-68

@bcmills bcmills changed the title runtime,cmd/compile: frequent memory corruption on NetBSD since 2021-10-11 runtime,cmd/compile: frequent memory corruption on NetBSD and OpenBSD since 2021-10-11 Nov 17, 2021
@toothrot toothrot added the okay-after-beta1 Used by release team to mark a release-blocker issue as okay to resolve either before or after beta1 label Nov 19, 2021
@cherrymui
Copy link
Member

cherrymui commented Dec 15, 2021

From #34988 it seems we can conclude that it is a kernel issue on NetBSD? Is there any conclusion for OpenBSD?

Do we still need to keep this issue open? Thanks.

@gopherbot
Copy link

gopherbot commented Dec 15, 2021

Change https://golang.org/cl/372355 mentions this issue: doc/go1.18: document OpenBSD & NetBSD memory corruption

@bcmills
Copy link
Member Author

bcmills commented Dec 15, 2021

Do we still need to keep this issue open?

From my perspective, we do still need to keep the issue open, to the extent that we need to:

  • document the architectures on which Go is known not to work, and/or
  • configure the Go project's builders to work around the kernel issue (e.g. by avoiding running on the affected AMD CPUs), and/or
  • explicitly drop support for the affected ports until such time as the port maintainers can fix or work around the upstream bugs.

@bcmills
Copy link
Member Author

bcmills commented Dec 15, 2021

Hmm. Actually, I think it suffices to mark this as a duplicate of #34988, and do the above things for that bug. 👍

@prattmic
Copy link
Member

prattmic commented Dec 15, 2021

Is there any conclusion for OpenBSD?

Not yet, as the C reproducer doesn't currently work on OpenBSD. We probably need to go back to David's Go reproducer and see if that works, and if so try to get it working in C again.

@cagedmantis
Copy link
Contributor

cagedmantis commented Jan 7, 2022

This is not a bug in Go. The failing builders will be annotated with a known issue until it is resolved. Because of this, it is no longer a release blocker.

@gopherbot
Copy link

gopherbot commented Jan 11, 2022

Change https://golang.org/cl/377474 mentions this issue: dashboard: clean up builders affected by memory corruption

gopherbot pushed a commit to golang/build that referenced this issue Jan 12, 2022
Remove freebsd 12.2, which is replaced by 12.3 with the XSAVE fix.
Move freebsd 11.* to N2 machines, which are not affected.
Remove openbsd and netbsd e2/n1/n2/n2d-specific configurations now that
we have mostly understood the nature of that problem. Keep one around so
that the runtime team can create gomotes. Move the "official" builder to
the n2 cpu that works.

For golang/go#49967, golang/go#49209, golang/go#40561.
Fixes golang/go#50496.

Change-Id: If6989317f06cbec95d5addb19d9e968aecfa3f8a
Reviewed-on: https://go-review.googlesource.com/c/build/+/377474
Trust: Heschi Kreinick <heschi@google.com>
Run-TryBot: Heschi Kreinick <heschi@google.com>
Reviewed-by: Carlos Amedee <carlos@golang.org>
Reviewed-by: Alex Rakoczy <alex@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
gopherbot pushed a commit that referenced this issue Jan 14, 2022
Knowing whether test failures are correlated with specific CPU models on
has proven useful on several issues. Log it for prior to testing so it
is always available.

internal/sysinfo provides the CPU model, but it is not available in the
bootstrap toolchain, so we can't access this unconditionally in
cmd/dist. Instead use a build-tagged file, as the final version of
cmd/dist will use the final toolchain.

The addition of new data to the beginning of cmd/dist output will break
x/build/cmd/coordinator's banner parsing, leaving extra lines in the log
output, though information will not be lost.
https://golang.org/cl/372538 fixes up the coordinator and should be
submitted and deployed before this CL is submitted.

For #46272.
For #49209.
For #50146.

Change-Id: I515d2ec58e4c0034b76bf624ecaab38f16146074
Reviewed-on: https://go-review.googlesource.com/c/go/+/371474
Trust: Benny Siegert <bsiegert@gmail.com>
Reviewed-by: Benny Siegert <bsiegert@gmail.com>
Trust: Michael Pratt <mpratt@google.com>
Run-TryBot: Michael Pratt <mpratt@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Austin Clements <austin@google.com>
@gopherbot
Copy link

gopherbot commented Jan 14, 2022

Change https://golang.org/cl/378589 mentions this issue: cmd/dist: log CPU model when testing

@ianlancetaylor
Copy link
Contributor

ianlancetaylor commented Jan 29, 2022

If our conclusion is that this is not a bug in Go, we should consider moving this to the 1.19 milestone.

@bcmills
Copy link
Member Author

bcmills commented Jan 29, 2022

The issue is documented and the builders are configured to work around it. As far as I am concerned it would be appropriate to close this issue.

@ianlancetaylor
Copy link
Contributor

ianlancetaylor commented Jan 29, 2022

SGTM. Thanks.

gopherbot pushed a commit that referenced this issue Mar 8, 2022
Knowing whether test failures are correlated with specific CPU models on
has proven useful on several issues. Log it for prior to testing so it
is always available.

internal/sysinfo provides the CPU model, but it is not available in the
bootstrap toolchain, so we can't access this in cmd/dist. Instead use a
separate binary which cmd/dist will only build once testing begins.

The addition of new data to the beginning of cmd/dist output will break
x/build/cmd/coordinator's banner parsing, leaving extra lines in the log
output, though information will not be lost.
https://golang.org/cl/372538 fixes up the coordinator and should be
submitted and deployed before this CL is submitted.

This is a redo of CL 371474. It switches back to the original approach
of using a separate binary, as the bootstap toolchain won't allow
cmd/dist to import internal packages.

For #46272.
For #49209.
For #50146.

Change-Id: I906bbda987902a2120c5183290a4e89a2440de58
Reviewed-on: https://go-review.googlesource.com/c/go/+/378589
Reviewed-by: Austin Clements <austin@google.com>
Trust: Michael Pratt <mpratt@google.com>
Run-TryBot: Michael Pratt <mpratt@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
jproberts pushed a commit to jproberts/go that referenced this issue Jun 21, 2022
Knowing whether test failures are correlated with specific CPU models on
has proven useful on several issues. Log it for prior to testing so it
is always available.

internal/sysinfo provides the CPU model, but it is not available in the
bootstrap toolchain, so we can't access this unconditionally in
cmd/dist. Instead use a build-tagged file, as the final version of
cmd/dist will use the final toolchain.

The addition of new data to the beginning of cmd/dist output will break
x/build/cmd/coordinator's banner parsing, leaving extra lines in the log
output, though information will not be lost.
https://golang.org/cl/372538 fixes up the coordinator and should be
submitted and deployed before this CL is submitted.

For golang#46272.
For golang#49209.
For golang#50146.

Change-Id: I515d2ec58e4c0034b76bf624ecaab38f16146074
Reviewed-on: https://go-review.googlesource.com/c/go/+/371474
Trust: Benny Siegert <bsiegert@gmail.com>
Reviewed-by: Benny Siegert <bsiegert@gmail.com>
Trust: Michael Pratt <mpratt@google.com>
Run-TryBot: Michael Pratt <mpratt@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Austin Clements <austin@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-NetBSD
Projects
None yet
Development

No branches or pull requests