Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime,cmd/compile: frequent memory corruption on NetBSD and OpenBSD since 2021-10-11 #49209

Open
bcmills opened this issue Oct 28, 2021 · 20 comments

Comments

@bcmills bcmills changed the title runtime,cmd/compile: frequent failures with "freeIndex is not valid" since 2021-10-18 runtime,cmd/compile: frequent failures with "freeIndex is not valid" on NetBSD since 2021-10-18 Oct 28, 2021
@bcmills
Copy link
Member Author

@bcmills bcmills commented Oct 28, 2021

Loading

@bcmills
Copy link
Member Author

@bcmills bcmills commented Oct 28, 2021

This appears to be a regression in Go 1.18.

Since NetBSD is not a first-class port, this doesn't necessarily block the 1.18 release — however, if the regression remains at the time of the release it at least needs a clear writeup in the release notes. (That part, at least, is a release-blocker.)

Loading

@bcmills
Copy link
Member Author

@bcmills bcmills commented Oct 28, 2021

@jeremyfaller, any idea who on the Go team might have insight into what broke on the 18th specifically?

Loading

@coypoop
Copy link
Contributor

@coypoop coypoop commented Oct 29, 2021

I'm not sure this error is entirely a regression, as I've seen this with pre-Go-1.18 on NetBSD. But perhaps something is making it much more frequent.

Loading

@jeremyfaller
Copy link
Contributor

@jeremyfaller jeremyfaller commented Oct 29, 2021

Did a little digging on Oct17-18, and didn't see much that looked suspicious.

pinging @mknyszek

Loading

@bcmills
Copy link
Member Author

@bcmills bcmills commented Nov 2, 2021

There have been a lot of other failures since then that look like arbitrary memory corruption in the runtime.
Whatever this is, it looks like it's got all of the NetBSD builders in pretty bad shape.
(Silver lining: the failure rate is high enough that this should be feasible to bisect!)

greplogs --dashboard -md -l -e '(?ms)\Anetbsd.*fatal error:' --since=2021-10-01

2021-11-02T15:15:09-4ff95d6-da7173a/netbsd-amd64-9_0
2021-11-02T06:25:39-30b2efe-088bb4b/netbsd-amd64-9_0
2021-11-02T06:19:33-4633b2d/netbsd-amd64-9_0
2021-11-02T05:43:05-a97c527/netbsd-amd64-9_0
2021-11-02T05:43:05-835dae6-a97c527/netbsd-386-9_0
2021-11-02T05:35:11-6f327f7/netbsd-386-9_0
2021-11-02T03:09:01-a45457d/netbsd-amd64-9_0
2021-11-02T00:12:17-4ff95d6-81fea0b/netbsd-386-9_0
2021-11-01T22:55:50-02e5913/netbsd-amd64-9_0
2021-11-01T21:27:26-4ff95d6-631b567/netbsd-386-9_0
2021-11-01T21:27:26-2c98350-631b567/netbsd-amd64-9_0
2021-11-01T19:56:50-ee08195-5f1bcd1/netbsd-386-9_0
2021-11-01T18:53:06-9967326-b5a5b7b/netbsd-amd64-9_0
2021-11-01T16:31:02-2bcf1c0/netbsd-386-9_0
2021-11-01T15:55:25-e2e910e/netbsd-386-9_0
2021-11-01T15:20:08-b7d65fb-82f902a/netbsd-amd64-9_0
2021-11-01T14:32:47-30b2efe-4a84298/netbsd-386-9_0
2021-11-01T13:54:03-42daa65-4a84298/netbsd-386-9_0
2021-11-01T13:12:37-611d5d6-4056934/netbsd-amd64-9_0
2021-11-01T13:12:37-4ff95d6-4056934/netbsd-386-9_0
2021-11-01T02:47:30-fde4cc2/netbsd-386-9_0
2021-11-01T02:47:30-fde4cc2/netbsd-amd64-9_0
2021-10-31T18:39:05-12ab535-89c5270/netbsd-amd64-9_0
2021-10-31T17:52:41-4ff95d6-3fa9ee4/netbsd-386-9_0
2021-10-30T18:30:34-ba495a6-6113dac/netbsd-386-9_0
2021-10-30T18:30:34-6113dac/netbsd-amd64-9_0
2021-10-30T18:30:34-351c04c-6113dac/netbsd-386-9_0
2021-10-30T16:47:17-30b5d63/netbsd-386-9_0
2021-10-30T16:46:47-d1dceaf/netbsd-386-9_0
2021-10-30T16:45:25-e39b854/netbsd-386-9_0
2021-10-30T16:08:13-b3129d9-5d6d9f5/netbsd-386-9_0
2021-10-30T00:47:26-a6c6f4b-d19c5bd/netbsd-amd64-9_0
2021-10-30T00:47:26-a6c6f4b-5d6d9f5/netbsd-386-9_0
2021-10-30T00:47:26-a6c6f4b-5d6d9f5/netbsd-amd64-9_0
2021-10-30T00:47:26-a6c6f4b-4a84298/netbsd-386-9_0
2021-10-29T23:15:00-a2be0cd-8dfb447/netbsd-amd64-9_0
2021-10-29T22:29:31-c96bc14-c812b97/netbsd-386-9_0
2021-10-29T22:27:54-71e6ab8/netbsd-amd64-9_0
2021-10-29T22:27:26-3571ab5/netbsd-386-9_0
2021-10-29T21:19:39-a2be0cd-d19c5bd/netbsd-amd64-9_0
2021-10-29T21:19:39-a2be0cd-994049a/netbsd-amd64-9_0
2021-10-29T19:42:51-540bb53-4a84298/netbsd-amd64-9_0
2021-10-29T19:42:43-4ff95d6-d19c5bd/netbsd-386-9_0
2021-10-29T19:30:03-4592933/netbsd-386-9_0
2021-10-29T18:35:01-8e112a7/netbsd-amd64-9_0
2021-10-29T18:23:03-413672f/netbsd-386-9_0
2021-10-29T17:44:15-d8fc7f7/netbsd-386-9_0
2021-10-29T17:44:15-d8fc7f7/netbsd-amd64-9_0
2021-10-29T04:21:07-d6a9af8-f288526/netbsd-386-9_0
2021-10-29T04:21:07-d6a9af8-f288526/netbsd-amd64-9_0
2021-10-29T02:12:05-d6a9af8-d3d8852/netbsd-amd64-9_0
2021-10-28T22:38:00-af05d8b/netbsd-amd64-9_0
2021-10-28T20:43:39-b954024/netbsd-386-9_0
2021-10-28T20:43:39-6944b10-b954024/netbsd-amd64-9_0
2021-10-28T20:27:44-5e21728/netbsd-386-9_0
2021-10-28T19:04:41-4e1c44d-18b9702/netbsd-386-9_0
2021-10-28T18:48:25-e7eb6f6-9004433/netbsd-386-9_0
2021-10-28T18:17:57-f229e70/netbsd-386-9_0
2021-10-28T18:01:34-03971e3-18b9702/netbsd-386-9_0
2021-10-28T16:54:58-c8ad2e1-6bd0e7f/netbsd-amd64-9_0
2021-10-28T15:31:34-8de2a7f-8c9c148/netbsd-amd64-9_0
2021-10-28T03:44:12-103d89b-2ff1074/netbsd-amd64-9_0
2021-10-28T03:35:34-b8f928b/netbsd-amd64-9_0
2021-10-28T01:15:26-b2fe2eb/netbsd-386-9_0
2021-10-28T01:15:26-103d89b-b2fe2eb/netbsd-386-9_0
2021-10-27T21:37:54-7b0b504-749f6e9/netbsd-amd64-9_0
2021-10-27T21:34:46-de1abf7/netbsd-386-9_0
2021-10-27T21:34:46-de1abf7/netbsd-amd64-9_0
2021-10-27T20:29:07-51be206/netbsd-386-9_0
2021-10-27T20:25:06-bbc0595/netbsd-amd64-9_0
2021-10-27T20:24:34-56dcf97/netbsd-386-9_0
2021-10-27T20:03:17-7b0b504-68bd512/netbsd-386-9_0
2021-10-27T19:33:43-7b0b504-11b64b4/netbsd-amd64-9_0
2021-10-27T19:33:07-1b02aa2-514ebae/netbsd-386-9_0
2021-10-27T18:08:49-08f2e89-bb49eb3/netbsd-amd64-9_0
2021-10-27T17:22:43-714a425-11b64b4/netbsd-386-9_0
2021-10-27T16:59:43-c0ac39c/netbsd-amd64-9_0
2021-10-27T16:39:27-94870a3-4f73fd0/netbsd-386-9_0
2021-10-27T13:12:49-e1e2965-cfb5321/netbsd-amd64-9_0
2021-10-27T13:12:49-d418f37-cfb5321/netbsd-386-9_0
2021-10-27T06:23:35-5786a54/netbsd-386-9_0
2021-10-27T05:33:58-ca5f65d/netbsd-386-9_0
2021-10-26T22:24:36-591e12a-80be4a4/netbsd-386-9_0
2021-10-26T22:05:53-80be4a4/netbsd-amd64-9_0
2021-10-26T22:00:36-26dbf47-86f6bf1/netbsd-amd64-9_0
2021-10-26T21:32:57-86f6bf1/netbsd-amd64-9_0
2021-10-26T21:17:38-03971e3-091db63/netbsd-386-9_0
2021-10-26T18:40:06-9626607-11b64b4/netbsd-386-9_0
2021-10-26T18:40:06-9626607-11b64b4/netbsd-amd64-9_0
2021-10-26T18:33:39-244f92e-11b64b4/netbsd-amd64-9_0
2021-10-26T17:10:45-2c98350-76cef81/netbsd-386-9_0
2021-10-26T15:46:18-c4ead46-1b2362b/netbsd-386-9_0
2021-10-26T15:20:53-903c757-1b2362b/netbsd-amd64-9_0
2021-10-26T11:58:05-1e2820a/netbsd-386-9_0
2021-10-26T11:58:05-1e2820a/netbsd-amd64-9_0
2021-10-26T02:02:46-903c757-11b64b4/netbsd-amd64-9_0
2021-10-26T01:22:47-adfb85b/netbsd-386-9_0
2021-10-26T01:22:47-adfb85b/netbsd-amd64-9_0
2021-10-26T00:38:52-22652b4-0ec1b62/netbsd-386-9_0
2021-10-26T00:30:24-22652b4-b36b001/netbsd-amd64-9_0
2021-10-25T22:17:34-fd2f4b5/netbsd-386-9_0
2021-10-25T20:40:31-5a40697-2ac3bdf/netbsd-amd64-9_0
2021-10-19T18:56:08-07e5527/netbsd-386-9_0
2021-10-19T08:44:22-98f6e03-1e49210/netbsd-386-9_0
2021-10-19T07:45:46-ee92daa/netbsd-amd64-9_0
2021-10-19T07:45:46-98f6e03-ee92daa/netbsd-386-9_0
2021-10-19T07:45:46-98f6e03-ee92daa/netbsd-amd64-9_0
2021-10-18T21:57:36-98f6e03-3befaf0/netbsd-amd64-9_0
2021-10-18T21:52:05-98f6e03-425db64/netbsd-386-9_0
2021-10-18T21:52:05-425db64/netbsd-amd64-9_0
2021-10-18T20:39:22-267abbe/netbsd-amd64-9_0
2021-10-18T19:46:27-394a1ad/netbsd-386-9_0
2021-10-14T07:18:59-1349c6e/netbsd-amd64-9_0
2021-10-12T21:14:34-03971e3-c580180/netbsd-amd64-9_0
2021-10-11T15:48:29-3b03d30-2478b6f/netbsd-amd64-9_0

Loading

@bcmills bcmills changed the title runtime,cmd/compile: frequent failures with "freeIndex is not valid" on NetBSD since 2021-10-18 runtime,cmd/compile: frequent memory corruption on NetBSD since 2021-10-11 Nov 2, 2021
@mknyszek
Copy link
Contributor

@mknyszek mknyszek commented Nov 8, 2021

On it.

Loading

@mknyszek mknyszek self-assigned this Nov 8, 2021
@mknyszek
Copy link
Contributor

@mknyszek mknyszek commented Nov 9, 2021

goswarm $ GOROOT=$HOME/toolchain ./goswarm -match 'freeIndex is not valid' netbsd-386-9_0 go/src/all.bash
2021/11/09 00:09:59 Created instance user-mknyszek-netbsd-386-9_0-1...
2021/11/09 00:10:02 Created instance user-mknyszek-netbsd-386-9_0-2...
2021/11/09 00:10:03 Created instance user-mknyszek-netbsd-386-9_0-3...
2021/11/09 00:10:03 Created instance user-mknyszek-netbsd-386-9_0-4...
2021/11/09 00:10:05 Created instance user-mknyszek-netbsd-386-9_0-5...
2021/11/09 00:10:05 Created instance user-mknyszek-netbsd-386-9_0-6...
2021/11/09 00:10:07 Created instance user-mknyszek-netbsd-386-9_0-7...
2021/11/09 00:10:13 Pushed to user-mknyszek-netbsd-386-9_0-1.
2021/11/09 00:10:13 Running command on user-mknyszek-netbsd-386-9_0-1.
2021/11/09 00:10:15 Created instance user-mknyszek-netbsd-386-9_0-8...
2021/11/09 00:10:15 Created instance user-mknyszek-netbsd-386-9_0-9...
2021/11/09 00:10:15 Pushed to user-mknyszek-netbsd-386-9_0-2.
2021/11/09 00:10:15 Running command on user-mknyszek-netbsd-386-9_0-2.
2021/11/09 00:10:17 Pushed to user-mknyszek-netbsd-386-9_0-3.
2021/11/09 00:10:17 Running command on user-mknyszek-netbsd-386-9_0-3.
2021/11/09 00:10:17 Pushed to user-mknyszek-netbsd-386-9_0-4.
2021/11/09 00:10:17 Running command on user-mknyszek-netbsd-386-9_0-4.
2021/11/09 00:10:19 Pushed to user-mknyszek-netbsd-386-9_0-6.
2021/11/09 00:10:19 Running command on user-mknyszek-netbsd-386-9_0-6.
2021/11/09 00:10:19 Pushed to user-mknyszek-netbsd-386-9_0-5.
2021/11/09 00:10:19 Running command on user-mknyszek-netbsd-386-9_0-5.
2021/11/09 00:10:20 Pushed to user-mknyszek-netbsd-386-9_0-7.
2021/11/09 00:10:20 Running command on user-mknyszek-netbsd-386-9_0-7.
2021/11/09 00:10:28 Pushed to user-mknyszek-netbsd-386-9_0-8.
2021/11/09 00:10:28 Running command on user-mknyszek-netbsd-386-9_0-8.
2021/11/09 00:10:28 Pushed to user-mknyszek-netbsd-386-9_0-9.
2021/11/09 00:10:28 Running command on user-mknyszek-netbsd-386-9_0-9.
2021/11/09 00:10:29 Created instance user-mknyszek-netbsd-386-9_0-10...
2021/11/09 00:10:42 Pushed to user-mknyszek-netbsd-386-9_0-10.
2021/11/09 00:10:42 Running command on user-mknyszek-netbsd-386-9_0-10.
2021/11/09 00:21:43 Running command on user-mknyszek-netbsd-386-9_0-2.
2021/11/09 00:31:28 Running command on user-mknyszek-netbsd-386-9_0-1.
2021/11/09 00:31:29 Running command on user-mknyszek-netbsd-386-9_0-6.
2021/11/09 00:31:33 Running command on user-mknyszek-netbsd-386-9_0-8.
2021/11/09 00:31:54 Running command on user-mknyszek-netbsd-386-9_0-7.
2021/11/09 00:31:58 Running command on user-mknyszek-netbsd-386-9_0-3.
2021/11/09 00:31:58 Running command on user-mknyszek-netbsd-386-9_0-4.
2021/11/09 00:32:38 Running command on user-mknyszek-netbsd-386-9_0-10.
2021/11/09 00:33:05 Running command on user-mknyszek-netbsd-386-9_0-9.
2021/11/09 00:40:36 Running command on user-mknyszek-netbsd-386-9_0-5.
2021/11/09 00:42:10 Running command on user-mknyszek-netbsd-386-9_0-2.
2021/11/09 00:52:34 Running command on user-mknyszek-netbsd-386-9_0-1.
2021/11/09 00:52:37 Running command on user-mknyszek-netbsd-386-9_0-6.
2021/11/09 00:53:15 Running command on user-mknyszek-netbsd-386-9_0-7.
2021/11/09 00:53:18 Running command on user-mknyszek-netbsd-386-9_0-3.
2021/11/09 00:53:36 Running command on user-mknyszek-netbsd-386-9_0-4.

This is more difficult to reproduce than I anticipated...

Loading

@mknyszek
Copy link
Contributor

@mknyszek mknyszek commented Nov 9, 2021

My gomote swarm ran all night and I got nowhere, unfortunately.

Loading

@bcmills
Copy link
Member Author

@bcmills bcmills commented Nov 9, 2021

Looking more carefully at the fatal error failures, I notice that some of them are actually on the release branches.

For example, the 2021-10-28T18:48:25-e7eb6f6-9004433/netbsd-386-9_0 failure is on release-branch.go1.16.

@golang/release, was there a change to the builder images on or around Oct. 11 that could explain these failures? (If so, is there an older image we can restore, or a platform expert who can, say, try bisecting the kernel?)

Loading

@bcmills
Copy link
Member Author

@bcmills bcmills commented Nov 9, 2021

@mknyszek, the freeIndex is not valid failure mode seems to be rarer than some of the other failure modes. Would it make sense to cast a wider net for fatal errors that lead to core files?

Loading

@mknyszek
Copy link
Contributor

@mknyszek mknyszek commented Nov 9, 2021

Ah OK. I'll focus on other failure modes.

Loading

@bcmills
Copy link
Member Author

@bcmills bcmills commented Nov 9, 2021

Going back further in the history seems to show:

  • There was an interval with an extremely high netbsd failure rate around 2021-02-19; looks like it was #44269.

  • Then there were a few intermittent failures with memory-corruption signatures — maybe once a month, mostly involving systemstack and/or wirep (maybe related to #49259?) — from 2021-03-10 until the onset of the unlock2 crashes (#49453), which started around 2021-09-22 on OpenBSD and first appeared 2021-10-07 on NetBSD.

  • Then a large uptick in failures in mid-October — maybe corresponding to a builder upgrade, or maybe even just the opening of the tree in October?

greplogs --dashboard -md -l -e '(?ms)\Anetbsd-(?:amd64|386).*fatal error:' --since=2021-01-01

2021-11-09T01:45:54-fafc446-5344dca/netbsd-386-9_0
2021-11-08T22:05:00-3a5865c-67e2294/netbsd-amd64-9_0
2021-11-08T21:52:47-a5321bf-955f9f5/netbsd-386-9_0
2021-11-08T21:45:43-cc49178/netbsd-386-9_0
2021-11-08T21:45:43-cc49178/netbsd-amd64-9_0
2021-11-08T21:45:43-39ade5b-cc49178/netbsd-386-9_0
2021-11-08T21:20:25-a5321bf-23991f5/netbsd-386-9_0
2021-11-08T21:20:16-0edd7b8-f58c78a/netbsd-386-9_0
2021-11-08T21:19:47-17472e0-f58c78a/netbsd-amd64-9_0
2021-11-08T20:46:37-c8e8de6-f58c78a/netbsd-amd64-9_0
2021-11-08T20:32:21-b766c28-830b393/netbsd-386-9_0
2021-11-08T20:32:21-b766c28-23991f5/netbsd-386-9_0
2021-11-08T18:39:25-2cdcc60-f58c78a/netbsd-386-9_0
2021-11-08T18:39:25-2cdcc60-23991f5/netbsd-amd64-9_0
2021-11-08T18:06:16-7ae8057-5e64755/netbsd-386-9_0
2021-11-08T18:06:16-5e64755/netbsd-386-9_0
2021-11-08T18:06:16-39ade5b-5e64755/netbsd-amd64-9_0
2021-11-08T17:07:45-6635138-7bda349/netbsd-386-9_0
2021-11-08T16:15:01-30b2efe-759eaa2/netbsd-386-9_0
2021-11-08T14:49:56-18b340f-ab31dbc/netbsd-386-9_0
2021-11-07T04:56:11-85493d5/netbsd-386-9_0
2021-11-06T19:41:15-036812b-61d789d/netbsd-386-9_0
2021-11-06T16:43:43-39ade5b-3544082/netbsd-386-9_0
2021-11-06T16:43:43-0c60b7c-3544082/netbsd-amd64-9_0
2021-11-06T13:10:06-1f47c86-4f083c7/netbsd-386-9_0
2021-11-06T10:24:44-0c60b7c-f19e400/netbsd-amd64-9_0
2021-11-06T00:29:44-a66bbe2-b74f2ef/netbsd-386-9_0
2021-11-06T00:29:44-a07c284-b74f2ef/netbsd-amd64-9_0
2021-11-06T00:29:44-0c60b7c-b74f2ef/netbsd-amd64-9_0
2021-11-05T23:32:57-b8b8e7f-09e8de7/netbsd-386-9_0
2021-11-05T22:57:08-a66bbe2-09e8de7/netbsd-amd64-9_0
2021-11-05T22:54:47-ba79c1e/netbsd-386-9_0
2021-11-05T22:54:47-ba79c1e/netbsd-amd64-9_0
2021-11-05T22:30:17-03971e3-b07c41d/netbsd-386-9_0
2021-11-05T22:27:21-8f923a4/netbsd-amd64-9_0
2021-11-05T22:26:07-4ab7496-d3a80c7/netbsd-386-9_0
2021-11-05T22:26:07-39ade5b-d3a80c7/netbsd-amd64-9_0
2021-11-05T22:23:52-35a5881/netbsd-386-9_0
2021-11-05T21:48:25-a66bbe2-6b223e8/netbsd-amd64-9_0
2021-11-05T21:48:25-4ab7496-6b223e8/netbsd-386-9_0
2021-11-05T21:27:34-7aed6dd/netbsd-386-9_0
2021-11-05T21:13:38-39ade5b-091948a/netbsd-amd64-9_0
2021-11-05T20:06:27-a7b6526-7be227c/netbsd-386-9_0
2021-11-05T20:06:27-a07c284-7be227c/netbsd-386-9_0
2021-11-05T20:06:27-7be227c/netbsd-amd64-9_0
2021-11-05T19:48:29-c353f1b/netbsd-386-9_0
2021-11-05T19:48:29-c353f1b/netbsd-amd64-9_0
2021-11-05T19:14:22-a07c284-fa16efb/netbsd-amd64-9_0
2021-11-05T19:01:13-93bab8a/netbsd-386-9_0
2021-11-05T18:20:07-a66bbe2-53bab19/netbsd-386-9_0
2021-11-05T18:20:07-a07c284-53bab19/netbsd-386-9_0
2021-11-05T17:52:30-df18377/netbsd-amd64-9_0
2021-11-05T17:47:28-6f32d20/netbsd-386-9_0
2021-11-05T17:47:28-6f32d20/netbsd-amd64-9_0
2021-11-05T17:39:43-0bc98b3/netbsd-amd64-9_0
2021-11-05T17:17:30-a66bbe2-62c6ff4/netbsd-amd64-9_0
2021-11-05T17:17:30-62c6ff4/netbsd-amd64-9_0
2021-11-05T17:17:30-39ade5b-62c6ff4/netbsd-386-9_0
2021-11-05T17:08:36-1c6f3cc-3796df1/netbsd-386-9_0
2021-11-05T16:54:01-a66bbe2-3796df1/netbsd-386-9_0
2021-11-05T16:54:01-a07c284-3796df1/netbsd-386-9_0
2021-11-05T16:51:14-c58417b/netbsd-amd64-9_0
2021-11-05T07:00:05-ce13745-6fefb7f/netbsd-386-9_0
2021-11-05T05:30:39-b68c02e/netbsd-amd64-9_0
2021-11-05T05:29:10-3b5add5/netbsd-amd64-9_0
2021-11-05T04:20:33-089bfa5-0a5ca24/netbsd-386-9_0
2021-11-05T00:52:06-3839b60/netbsd-386-9_0
2021-11-04T23:35:26-ce13745-256a8fc/netbsd-amd64-9_0
2021-11-04T23:35:26-b76863e-256a8fc/netbsd-386-9_0
2021-11-04T21:53:05-ce13745-76c48e9/netbsd-386-9_0
2021-11-04T21:53:05-37ea4aa-76c48e9/netbsd-amd64-9_0
2021-11-04T21:42:40-37ea4aa-23991f5/netbsd-386-9_0
2021-11-04T21:42:40-37ea4aa-156abe5/netbsd-386-9_0
2021-11-04T21:41:49-39ade5b-156abe5/netbsd-386-9_0
2021-11-04T20:42:35-1f9dce7/netbsd-386-9_0
2021-11-04T20:31:02-39ade5b-978e39e/netbsd-386-9_0
2021-11-04T20:24:01-99699d1/netbsd-386-9_0
2021-11-04T20:01:11-fc5e8cd/netbsd-amd64-9_0
2021-11-04T19:34:33-5af93a2/netbsd-amd64-9_0
2021-11-04T18:22:03-84e69e7-b2149ac/netbsd-386-9_0
2021-11-04T18:22:03-39ade5b-b2149ac/netbsd-386-9_0
2021-11-04T17:07:48-5772877/netbsd-386-9_0
2021-11-04T17:07:48-5772877/netbsd-amd64-9_0
2021-11-04T16:36:19-84e69e7-f934b83/netbsd-386-9_0
2021-11-04T15:43:59-84e69e7-6ba68a0/netbsd-386-9_0
2021-11-04T14:54:46-4a448f8-00d6d20/netbsd-386-9_0
2021-11-04T14:24:17-d7068e7-f58c78a/netbsd-386-9_0
2021-11-04T14:17:18-7861aae-901bf29/netbsd-386-9_0
2021-11-04T14:11:01-84e69e7-a3f7be9/netbsd-386-9_0
2021-11-04T13:55:28-4a4e1f2-23991f5/netbsd-386-9_0
2021-11-04T13:55:28-39ade5b-23991f5/netbsd-386-9_0
2021-11-04T13:55:28-39ade5b-23991f5/netbsd-amd64-9_0
2021-11-04T13:55:28-30b2efe-23991f5/netbsd-386-9_0
2021-11-04T13:55:28-089bfa5-23991f5/netbsd-386-9_0
2021-11-04T13:55:24-4a4e1f2-f58c78a/netbsd-amd64-9_0
2021-11-04T13:55:24-036812b-f58c78a/netbsd-386-9_0
2021-11-04T07:05:31-84e69e7-2622235/netbsd-386-9_0
2021-11-04T07:05:31-84e69e7-2622235/netbsd-amd64-9_0
2021-11-04T07:05:31-39ade5b-2622235/netbsd-386-9_0
2021-11-04T02:57:53-4a4e1f2-2cf85b1/netbsd-386-9_0
2021-11-04T02:57:53-2cf85b1/netbsd-amd64-9_0
2021-11-04T02:57:48-5fd0c49/netbsd-386-9_0
2021-11-04T02:57:48-5fd0c49/netbsd-amd64-9_0
2021-11-04T02:12:52-8da7c0f-bd49a99/netbsd-amd64-9_0
2021-11-04T00:46:41-e72d715/netbsd-386-9_0
2021-11-04T00:46:41-84e69e7-e72d715/netbsd-amd64-9_0
2021-11-04T00:36:13-84e69e7-1292e21/netbsd-386-9_0
2021-11-04T00:29:42-a419f2f/netbsd-amd64-9_0
2021-11-04T00:29:42-39ade5b-a419f2f/netbsd-386-9_0
2021-11-04T00:19:43-9cf6711/netbsd-amd64-9_0
2021-11-04T00:19:43-39ade5b-9cf6711/netbsd-386-9_0
2021-11-03T20:30:17-68536fa-7f2463c/netbsd-386-9_0
2021-11-03T19:32:33-74f99d0/netbsd-386-9_0
2021-11-03T19:32:33-74f99d0/netbsd-amd64-9_0
2021-11-03T18:37:22-3b1a674-cfd016d/netbsd-amd64-9_0
2021-11-03T18:37:22-3a5865c-cfd016d/netbsd-amd64-9_0
2021-11-03T18:37:22-2c98350-cfd016d/netbsd-amd64-9_0
2021-11-03T16:57:50-ffed863-88407a8/netbsd-amd64-9_0
2021-11-03T16:57:50-3b1a674-88407a8/netbsd-amd64-9_0
2021-11-03T16:57:44-3a5865c-b212ba6/netbsd-386-9_0
2021-11-03T15:51:25-3b1a674-6458b2e/netbsd-386-9_0
2021-11-03T15:47:47-c143661/netbsd-386-9_0
2021-11-03T05:00:00-39ade5b-519c0a2/netbsd-386-9_0
2021-11-03T00:07:03-d6f7203/netbsd-386-9_0
2021-11-03T00:07:03-d6f7203/netbsd-amd64-9_0
2021-11-02T23:44:58-a0f373c/netbsd-386-9_0
2021-11-02T23:05:17-eb63cbd/netbsd-386-9_0
2021-11-02T23:05:17-714668c-eb63cbd/netbsd-amd64-9_0
2021-11-02T22:20:40-39ade5b-2157498/netbsd-386-9_0
2021-11-02T22:20:40-2157498/netbsd-386-9_0
2021-11-02T21:21:51-9aacde2-631b567/netbsd-amd64-9_0
2021-11-02T21:21:51-9aacde2-42e6b5b/netbsd-386-9_0
2021-11-02T21:18:39-60fd3ed/netbsd-386-9_0
2021-11-02T20:59:34-6561d8c-79024cf/netbsd-386-9_0
2021-11-02T20:59:34-39ade5b-79024cf/netbsd-386-9_0
2021-11-02T20:33:56-b76863e-b29182b/netbsd-amd64-9_0
2021-11-02T19:37:42-bb4add0-1011e26/netbsd-386-9_0
2021-11-02T19:37:42-bb4add0-1011e26/netbsd-amd64-9_0
2021-11-02T18:35:35-c702f91/netbsd-386-9_0
2021-11-02T18:35:35-c702f91/netbsd-amd64-9_0
2021-11-02T18:35:29-a07c284-433ba58/netbsd-amd64-9_0
2021-11-02T18:35:29-4a4e1f2-433ba58/netbsd-386-9_0
2021-11-02T18:24:18-bb4add0-f7a95d2/netbsd-386-9_0
2021-11-02T18:22:55-bb4add0-f9cb33c/netbsd-amd64-9_0
2021-11-02T18:19:06-1ba8fdb-631b567/netbsd-amd64-9_0
2021-11-02T18:01:20-39ade5b-629ffeb/netbsd-386-9_0
2021-11-02T18:01:20-058ed05-629ffeb/netbsd-amd64-9_0
2021-11-02T17:53:48-7548327/netbsd-amd64-9_0
2021-11-02T17:31:50-39ade5b-4e7dd9f/netbsd-386-9_0
2021-11-02T17:01:01-af8aafd/netbsd-386-9_0
2021-11-02T16:57:03-c406380/netbsd-amd64-9_0
2021-11-02T16:29:07-d4902dd-f9cb33c/netbsd-386-9_0
2021-11-02T16:19:52-c96bc14-58fb05a/netbsd-386-9_0
2021-11-02T16:19:52-058ed05-58fb05a/netbsd-386-9_0
2021-11-02T16:12:28-f801da7/netbsd-amd64-9_0
2021-11-02T16:12:23-599de4b/netbsd-amd64-9_0
2021-11-02T15:54:27-058ed05-f9cb33c/netbsd-386-9_0
2021-11-02T15:54:27-058ed05-631b567/netbsd-amd64-9_0
2021-11-02T15:15:09-4ff95d6-da7173a/netbsd-amd64-9_0
2021-11-02T06:25:39-30b2efe-088bb4b/netbsd-amd64-9_0
2021-11-02T06:19:33-4633b2d/netbsd-amd64-9_0
2021-11-02T05:43:05-a97c527/netbsd-amd64-9_0
2021-11-02T05:43:05-835dae6-a97c527/netbsd-386-9_0
2021-11-02T05:35:11-6f327f7/netbsd-386-9_0
2021-11-02T03:55:19-39ade5b-6f1e9a9/netbsd-386-9_0
2021-11-02T03:09:01-a45457d/netbsd-amd64-9_0
2021-11-02T00:12:17-4ff95d6-81fea0b/netbsd-386-9_0
2021-11-01T22:55:50-02e5913/netbsd-amd64-9_0
2021-11-01T21:27:26-4ff95d6-631b567/netbsd-386-9_0
2021-11-01T21:27:26-39ade5b-631b567/netbsd-386-9_0
2021-11-01T21:27:26-2c98350-631b567/netbsd-amd64-9_0
2021-11-01T21:27:26-1f47c86-631b567/netbsd-386-9_0
2021-11-01T19:56:50-ee08195-5f1bcd1/netbsd-386-9_0
2021-11-01T18:53:06-9967326-b5a5b7b/netbsd-amd64-9_0
2021-11-01T16:31:02-39ade5b-2bcf1c0/netbsd-amd64-9_0
2021-11-01T16:31:02-2bcf1c0/netbsd-386-9_0
2021-11-01T15:55:25-e2e910e/netbsd-386-9_0
2021-11-01T15:20:08-b7d65fb-82f902a/netbsd-amd64-9_0
2021-11-01T14:32:47-30b2efe-4a84298/netbsd-386-9_0
2021-11-01T13:54:03-42daa65-4a84298/netbsd-386-9_0
2021-11-01T13:12:37-611d5d6-4056934/netbsd-amd64-9_0
2021-11-01T13:12:37-4ff95d6-4056934/netbsd-386-9_0
2021-11-01T02:47:30-fde4cc2/netbsd-386-9_0
2021-11-01T02:47:30-fde4cc2/netbsd-amd64-9_0
2021-10-31T18:39:05-39ade5b-89c5270/netbsd-386-9_0
2021-10-31T18:39:05-12ab535-89c5270/netbsd-amd64-9_0
2021-10-31T18:13:09-39ade5b-fd09e88/netbsd-amd64-9_0
2021-10-31T17:52:41-4ff95d6-3fa9ee4/netbsd-386-9_0
2021-10-31T08:29:02-39ade5b-8e3d5f0/netbsd-amd64-9_0
2021-10-30T18:30:34-ba495a6-6113dac/netbsd-386-9_0
2021-10-30T18:30:34-6113dac/netbsd-amd64-9_0
2021-10-30T18:30:34-351c04c-6113dac/netbsd-386-9_0
2021-10-30T16:47:17-30b5d63/netbsd-386-9_0
2021-10-30T16:46:47-d1dceaf/netbsd-386-9_0
2021-10-30T16:45:25-e39b854/netbsd-386-9_0
2021-10-30T16:08:13-b3129d9-5d6d9f5/netbsd-386-9_0
2021-10-30T00:47:26-a6c6f4b-d19c5bd/netbsd-amd64-9_0
2021-10-30T00:47:26-a6c6f4b-5d6d9f5/netbsd-386-9_0
2021-10-30T00:47:26-a6c6f4b-5d6d9f5/netbsd-amd64-9_0
2021-10-30T00:47:26-a6c6f4b-4a84298/netbsd-386-9_0
2021-10-29T23:15:00-a2be0cd-8dfb447/netbsd-amd64-9_0
2021-10-29T22:29:31-c96bc14-c812b97/netbsd-386-9_0
2021-10-29T22:29:31-b76863e-c812b97/netbsd-amd64-9_0
2021-10-29T22:29:31-39ade5b-c812b97/netbsd-386-9_0
2021-10-29T22:27:54-71e6ab8/netbsd-amd64-9_0
2021-10-29T22:27:26-3571ab5/netbsd-386-9_0
2021-10-29T21:19:39-a2be0cd-d19c5bd/netbsd-amd64-9_0
2021-10-29T21:19:39-a2be0cd-994049a/netbsd-amd64-9_0
2021-10-29T19:42:51-540bb53-4a84298/netbsd-amd64-9_0
2021-10-29T19:42:43-4ff95d6-d19c5bd/netbsd-386-9_0
2021-10-29T19:42:43-089bfa5-d19c5bd/netbsd-amd64-9_0
2021-10-29T19:30:03-4592933/netbsd-386-9_0
2021-10-29T18:35:01-8e112a7/netbsd-amd64-9_0
2021-10-29T18:23:03-413672f/netbsd-386-9_0
2021-10-29T17:44:15-d8fc7f7/netbsd-386-9_0
2021-10-29T17:44:15-d8fc7f7/netbsd-amd64-9_0
2021-10-29T04:21:07-d6a9af8-f288526/netbsd-386-9_0
2021-10-29T04:21:07-d6a9af8-f288526/netbsd-amd64-9_0
2021-10-29T02:16:47-089bfa5-2c7cdec/netbsd-386-9_0
2021-10-29T02:12:05-d6a9af8-d3d8852/netbsd-amd64-9_0
2021-10-29T02:12:05-036812b-d3d8852/netbsd-386-9_0
2021-10-28T22:38:00-af05d8b/netbsd-amd64-9_0
2021-10-28T21:17:17-089bfa5-f6f024f/netbsd-386-9_0
2021-10-28T20:43:39-b954024/netbsd-386-9_0
2021-10-28T20:43:39-6944b10-b954024/netbsd-amd64-9_0
2021-10-28T20:27:44-5e21728/netbsd-386-9_0
2021-10-28T19:04:41-4e1c44d-18b9702/netbsd-386-9_0
2021-10-28T18:48:25-e7eb6f6-9004433/netbsd-386-9_0
2021-10-28T18:17:57-f229e70/netbsd-386-9_0
2021-10-28T18:01:38-a07c284-9004433/netbsd-amd64-9_0
2021-10-28T18:01:38-39ade5b-9004433/netbsd-386-9_0
2021-10-28T18:01:34-03971e3-18b9702/netbsd-386-9_0
2021-10-28T16:54:58-c8ad2e1-6bd0e7f/netbsd-amd64-9_0
2021-10-28T15:35:25-39ade5b-278b9b3/netbsd-amd64-9_0
2021-10-28T15:31:34-8de2a7f-8c9c148/netbsd-amd64-9_0
2021-10-28T15:31:34-1f47c86-8c9c148/netbsd-386-9_0
2021-10-28T15:31:34-089bfa5-8c9c148/netbsd-amd64-9_0
2021-10-28T15:08:31-26ed8fd-5c98bcb/netbsd-amd64-9_0
2021-10-28T03:44:12-103d89b-2ff1074/netbsd-amd64-9_0
2021-10-28T03:35:34-b8f928b/netbsd-amd64-9_0
2021-10-28T02:35:22-39ade5b-056dfe6/netbsd-amd64-9_0
2021-10-28T01:15:26-b2fe2eb/netbsd-386-9_0
2021-10-28T01:15:26-103d89b-b2fe2eb/netbsd-386-9_0
2021-10-27T21:37:54-7b0b504-749f6e9/netbsd-amd64-9_0
2021-10-27T21:34:46-de1abf7/netbsd-386-9_0
2021-10-27T21:34:46-de1abf7/netbsd-amd64-9_0
2021-10-27T20:29:07-51be206/netbsd-386-9_0
2021-10-27T20:25:06-bbc0595/netbsd-amd64-9_0
2021-10-27T20:24:34-56dcf97/netbsd-386-9_0
2021-10-27T20:03:17-7b0b504-68bd512/netbsd-386-9_0
2021-10-27T19:34:49-a66eb64-ffd2284/netbsd-386-9_0
2021-10-27T19:33:43-7b0b504-11b64b4/netbsd-amd64-9_0
2021-10-27T19:33:07-1b02aa2-514ebae/netbsd-386-9_0
2021-10-27T18:08:49-08f2e89-bb49eb3/netbsd-amd64-9_0
2021-10-27T17:22:43-714a425-11b64b4/netbsd-386-9_0
2021-10-27T16:59:43-c0ac39c/netbsd-amd64-9_0
2021-10-27T16:39:27-94870a3-4f73fd0/netbsd-386-9_0
2021-10-27T16:39:27-39ade5b-4f73fd0/netbsd-386-9_0
2021-10-27T13:12:49-e1e2965-cfb5321/netbsd-amd64-9_0
2021-10-27T13:12:49-d418f37-cfb5321/netbsd-386-9_0
2021-10-27T08:50:27-39ade5b-bdefb77/netbsd-386-9_0
2021-10-27T06:23:35-5786a54/netbsd-386-9_0
2021-10-27T05:33:58-ca5f65d/netbsd-386-9_0
2021-10-26T22:24:36-591e12a-80be4a4/netbsd-386-9_0
2021-10-26T22:05:53-80be4a4/netbsd-amd64-9_0
2021-10-26T22:05:53-036812b-80be4a4/netbsd-amd64-9_0
2021-10-26T22:00:36-26dbf47-86f6bf1/netbsd-amd64-9_0
2021-10-26T21:32:57-86f6bf1/netbsd-amd64-9_0
2021-10-26T21:17:38-03971e3-091db63/netbsd-386-9_0
2021-10-26T20:41:32-03fcf44-3a0cd11/netbsd-amd64-9_0
2021-10-26T18:40:06-9626607-11b64b4/netbsd-386-9_0
2021-10-26T18:40:06-9626607-11b64b4/netbsd-amd64-9_0
2021-10-26T18:33:39-244f92e-11b64b4/netbsd-amd64-9_0
2021-10-26T17:10:45-2c98350-76cef81/netbsd-386-9_0
2021-10-26T15:46:18-c4ead46-1b2362b/netbsd-386-9_0
2021-10-26T15:20:53-903c757-1b2362b/netbsd-amd64-9_0
2021-10-26T15:20:53-39ade5b-1b2362b/netbsd-386-9_0
2021-10-26T15:20:53-036812b-1b2362b/netbsd-amd64-9_0
2021-10-26T14:24:17-39ade5b-283d8a3/netbsd-386-9_0
2021-10-26T14:05:47-39ade5b-a2b8c18/netbsd-386-9_0
2021-10-26T11:58:05-1e2820a/netbsd-386-9_0
2021-10-26T11:58:05-1e2820a/netbsd-amd64-9_0
2021-10-26T02:02:46-903c757-11b64b4/netbsd-amd64-9_0
2021-10-26T01:22:47-adfb85b/netbsd-386-9_0
2021-10-26T01:22:47-adfb85b/netbsd-amd64-9_0
2021-10-26T00:38:52-22652b4-0ec1b62/netbsd-386-9_0
2021-10-26T00:30:24-22652b4-b36b001/netbsd-amd64-9_0
2021-10-25T22:17:34-fd2f4b5/netbsd-386-9_0
2021-10-25T20:40:31-5a40697-2ac3bdf/netbsd-amd64-9_0
2021-10-19T18:56:08-07e5527/netbsd-386-9_0
2021-10-19T08:44:22-98f6e03-1e49210/netbsd-386-9_0
2021-10-19T07:45:46-ee92daa/netbsd-amd64-9_0
2021-10-19T07:45:46-98f6e03-ee92daa/netbsd-386-9_0
2021-10-19T07:45:46-98f6e03-ee92daa/netbsd-amd64-9_0
2021-10-18T21:57:36-98f6e03-3befaf0/netbsd-amd64-9_0
2021-10-18T21:52:05-98f6e03-425db64/netbsd-386-9_0
2021-10-18T21:52:05-425db64/netbsd-amd64-9_0
2021-10-18T20:39:22-267abbe/netbsd-amd64-9_0
2021-10-18T19:46:27-a07c284-394a1ad/netbsd-amd64-9_0
2021-10-18T19:46:27-394a1ad/netbsd-386-9_0
2021-10-14T07:18:59-1349c6e/netbsd-amd64-9_0
2021-10-12T21:14:34-03971e3-c580180/netbsd-amd64-9_0
2021-10-11T15:48:29-3b03d30-2478b6f/netbsd-amd64-9_0
2021-10-07T19:49:45-39ade5b-c580180/netbsd-386-9_0
2021-08-16T20:21:13-8d20661/netbsd-386-9_0
2021-06-21T16:58:36-16e82be/netbsd-amd64-9_0
2021-05-11T14:51:35-2db0265-04cd717/netbsd-386-9_0
2021-04-20T16:44:18-109d758/netbsd-386-9_0
2021-04-09T12:56:04-519f223/netbsd-amd64-9_0
2021-04-01T00:51:26-ec721d9/netbsd-386-9_0
2021-03-10T09:01:05-cf59850/netbsd-amd64-9_0
2021-02-19T15:31:06-fa18f22/netbsd-amd64-9_0
2021-02-19T15:30:59-01eb70e/netbsd-amd64-9_0
2021-02-19T15:30:52-49add6a/netbsd-amd64-9_0
2021-02-19T14:46:21-fce2a94/netbsd-amd64-9_0
2021-02-19T14:46:21-3c2d99c-fce2a94/netbsd-amd64-9_0
2021-02-19T08:48:55-7764ee5/netbsd-amd64-9_0
2021-02-19T01:22:14-87f425d/netbsd-amd64-9_0
2021-02-19T01:22:14-5f4716e-87f425d/netbsd-amd64-9_0
2021-02-19T01:19:01-4da0188/netbsd-amd64-9_0
2021-02-19T00:41:47-5f4716e-5f2e24e/netbsd-amd64-9_0
2021-02-19T00:41:47-5f2e24e/netbsd-amd64-9_0
2021-02-19T00:41:41-ee7038f/netbsd-amd64-9_0
2021-02-19T00:41:36-40765ff/netbsd-amd64-9_0
2021-02-19T00:41:28-b445d6e/netbsd-amd64-9_0
2021-02-19T00:41:17-b110a43/netbsd-amd64-9_0
2021-02-19T00:41:09-474d5f4/netbsd-amd64-9_0
2021-02-19T00:41:03-c7c6c11/netbsd-amd64-9_0
2021-02-19T00:40:56-3527caa/netbsd-amd64-9_0
2021-02-19T00:40:42-f6c4b4b/netbsd-amd64-9_0
2021-02-19T00:40:30-ac024a0/netbsd-amd64-9_0
2021-02-19T00:40:22-a3b97e7/netbsd-amd64-9_0
2021-02-19T00:40:16-985d087/netbsd-amd64-9_0
2021-02-19T00:40:00-0ca0551/netbsd-amd64-9_0
2021-02-19T00:05:03-0c63312/netbsd-amd64-9_0
2021-02-19T00:04:56-bb6efb9/netbsd-amd64-9_0
2021-02-19T00:04:49-a1222b7/netbsd-amd64-9_0
2021-02-19T00:04:42-1c659f2/netbsd-amd64-9_0
2021-02-19T00:04:30-b6379f1/netbsd-amd64-9_0
2021-02-19T00:04:22-09e059a/netbsd-amd64-9_0
2021-02-19T00:04:14-b19e7b5/netbsd-amd64-9_0
2021-02-19T00:04:07-5421c37/netbsd-amd64-9_0
2021-02-19T00:03:59-91cc484/netbsd-amd64-9_0
2021-02-19T00:03:52-38672d3/netbsd-amd64-9_0
2021-02-19T00:03:44-a1e9148/netbsd-amd64-9_0
2021-02-19T00:03:36-75e273f/netbsd-amd64-9_0
2021-02-19T00:03:30-76ab626/netbsd-amd64-9_0
2021-02-19T00:03:22-ece954d/netbsd-amd64-9_0
2021-02-19T00:03:14-a54f7fc/netbsd-amd64-9_0
2021-02-19T00:02:58-5ecd9e3/netbsd-amd64-9_0
2021-02-19T00:02:49-54da3ab/netbsd-amd64-9_0
2021-02-19T00:02:40-fbe74db/netbsd-amd64-9_0

The freeIndex is not valid failures have followed a similar trajectory to the memory-corruption failures in general, but have a much crisper starting point: 2021-10-18.

greplogs --dashboard -md -l -e 'freeIndex is not valid' --since=2021-01-01

2021-11-08T20:32:21-b766c28-830b393/netbsd-386-9_0
2021-11-05T23:32:57-b8b8e7f-09e8de7/netbsd-386-9_0
2021-11-05T22:26:07-4ab7496-d3a80c7/netbsd-386-9_0
2021-11-05T22:23:52-35a5881/netbsd-386-9_0
2021-11-05T19:48:29-c353f1b/netbsd-amd64-9_0
2021-11-05T18:20:07-a66bbe2-53bab19/netbsd-386-9_0
2021-11-05T17:08:36-1c6f3cc-3796df1/netbsd-386-9_0
2021-11-05T16:54:01-a66bbe2-3796df1/netbsd-386-9_0
2021-11-04T21:42:40-37ea4aa-23991f5/netbsd-386-9_0
2021-11-04T21:42:40-37ea4aa-156abe5/netbsd-386-9_0
2021-11-04T18:22:03-84e69e7-b2149ac/netbsd-386-9_0
2021-11-04T15:43:59-84e69e7-6ba68a0/netbsd-386-9_0
2021-11-04T14:24:17-d7068e7-f58c78a/netbsd-386-9_0
2021-11-04T14:11:01-84e69e7-a3f7be9/netbsd-386-9_0
2021-11-04T00:36:13-84e69e7-1292e21/netbsd-386-9_0
2021-11-04T00:19:43-9cf6711/netbsd-amd64-9_0
2021-11-03T00:07:03-d6f7203/netbsd-386-9_0
2021-11-02T23:05:17-714668c-eb63cbd/netbsd-amd64-9_0
2021-11-02T20:59:34-6561d8c-79024cf/netbsd-386-9_0
2021-11-02T19:37:42-bb4add0-1011e26/netbsd-386-9_0
2021-11-02T19:37:42-1011e26/openbsd-386-68
2021-11-02T18:24:18-bb4add0-f7a95d2/netbsd-386-9_0
2021-11-02T15:54:27-058ed05-f9cb33c/netbsd-386-9_0
2021-11-01T21:27:26-4ff95d6-631b567/netbsd-386-9_0
2021-11-01T15:20:08-b7d65fb-82f902a/netbsd-amd64-9_0
2021-11-01T13:54:03-42daa65-4a84298/netbsd-386-9_0
2021-11-01T02:47:30-fde4cc2/netbsd-amd64-9_0
2021-10-30T18:30:34-351c04c-6113dac/netbsd-386-9_0
2021-10-30T00:47:26-a6c6f4b-5d6d9f5/netbsd-amd64-9_0
2021-10-30T00:47:26-a6c6f4b-4a84298/netbsd-386-9_0
2021-10-29T19:30:03-4592933/netbsd-386-9_0
2021-10-29T17:44:15-d8fc7f7/netbsd-386-9_0
2021-10-29T04:21:07-d6a9af8-f288526/netbsd-386-9_0
2021-10-29T04:21:07-d6a9af8-f288526/netbsd-amd64-9_0
2021-10-29T02:12:05-d6a9af8-d3d8852/netbsd-amd64-9_0
2021-10-29T02:12:05-036812b-d3d8852/netbsd-386-9_0
2021-10-28T22:38:00-af05d8b/netbsd-amd64-9_0
2021-10-28T19:04:41-4e1c44d-18b9702/netbsd-386-9_0
2021-10-28T18:17:57-f229e70/netbsd-386-9_0
2021-10-28T18:01:34-03971e3-18b9702/netbsd-386-9_0
2021-10-28T01:15:26-103d89b-b2fe2eb/netbsd-386-9_0
2021-10-27T20:03:17-7b0b504-68bd512/netbsd-386-9_0
2021-10-27T16:39:27-94870a3-4f73fd0/netbsd-386-9_0
2021-10-27T13:12:49-d418f37-cfb5321/netbsd-386-9_0
2021-10-27T06:23:35-5786a54/netbsd-386-9_0
2021-10-27T05:33:58-ca5f65d/netbsd-386-9_0
2021-10-26T22:24:36-591e12a-80be4a4/netbsd-386-9_0
2021-10-26T22:05:53-80be4a4/netbsd-amd64-9_0
2021-10-26T18:40:06-9626607-11b64b4/netbsd-386-9_0
2021-10-26T15:46:18-c4ead46-1b2362b/netbsd-386-9_0
2021-10-19T07:45:46-98f6e03-ee92daa/netbsd-386-9_0
2021-10-18T21:52:05-98f6e03-425db64/netbsd-386-9_0
2021-10-18T21:52:05-425db64/netbsd-amd64-9_0
2021-04-01T15:50:43-45ca9ef/linux-arm-scaleway
2021-04-01T01:26:29-5f646f0/linux-arm-scaleway
2021-04-01T00:51:26-ec721d9/linux-arm-scaleway
2021-04-01T00:51:24-1f29e69/linux-arm-scaleway
2021-04-01T00:51:23-3304b22/linux-arm-scaleway
2021-03-31T20:21:57-5d6581d/linux-arm-scaleway

Loading

@mknyszek
Copy link
Contributor

@mknyszek mknyszek commented Nov 16, 2021

Got a core dump... for a go1.4 failure. I feel like I'm cursed.

Building Go cmd/dist using /tmp/workdir/go1.4. (devel +0b3b511 Tue Dec 5 18:44:31 2017 +0000 netbsd/386)
Building Go toolchain1 using /tmp/workdir/go1.4.
fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x1f8 pc=0x80553c7]

runtime stack:
runtime.throw(0x8428440, 0x2a)
	/home/bradfitz/go-netbsd-386-bootstrap/src/runtime/panic.go:616 +0x6b fp=0x1893bec8 sp=0x1893bebc pc=0x806e24b
runtime.sigpanic()
	/home/bradfitz/go-netbsd-386-bootstrap/src/runtime/signal_unix.go:366 +0x230 fp=0x1893beec sp=0x1893bec8 pc=0x8081480
runtime.unlock(0x86290b0)
	/home/bradfitz/go-netbsd-386-bootstrap/src/runtime/lock_sema.go:107 +0x97 fp=0x1893bf08 sp=0x1893beec pc=0x80553c7
runtime.goschedImpl(0x18960fc0)
	/home/bradfitz/go-netbsd-386-bootstrap/src/runtime/proc.go:2606 +0xcb fp=0x1893bf18 sp=0x1893bf08 pc=0x8074c7b
runtime.gopreempt_m(0x18960fc0)
	/home/bradfitz/go-netbsd-386-bootstrap/src/runtime/proc.go:2636 +0x26 fp=0x1893bf20 sp=0x1893bf18 pc=0x8074d36
runtime.newstack()
	/home/bradfitz/go-netbsd-386-bootstrap/src/runtime/stack.go:1039 +0x213 fp=0x1893bfd8 sp=0x1893bf20 pc=0x8084393
runtime.morestack()
	/home/bradfitz/go-netbsd-386-bootstrap/src/runtime/asm_386.s:536 +0x70 fp=0x1893bfdc sp=0x1893bfd8 pc=0x8092b50

Loading

@mknyszek
Copy link
Contributor

@mknyszek mknyszek commented Nov 16, 2021

Got 2 crashes in one go: a freeIndex crash (!) and a scanstack - bad status crash. I also got a whole bunch of cores.

Loading

@mknyszek
Copy link
Contributor

@mknyszek mknyszek commented Nov 16, 2021

OK so I can't seem to find the core for the scanstack issue, but the freeIndex issue is kind of interesting. First let me say that I'm pretty confident nothing is off here -- all the pointers line up with what's in the stack trace, nothing's inconsistent so far.

So, the check that's failing has code that looks like

 871     s = c.alloc[spc]
 872     shouldhelpgc = false
 873     freeIndex := s.nextFreeIndex()
 874     if freeIndex == s.nelems {
 875         // The span is full.
 876         if uintptr(s.allocCount) != s.nelems {
 877             println("runtime: s.allocCount=", s.allocCount, "s.nelems=", s.nelems)
 878             throw("s.allocCount != s.nelems && freeIndex == s.nelems")
 879         }
 880         c.refill(spc)
 881         shouldhelpgc = true
 882         s = c.alloc[spc]                                                                     
 883
 884         freeIndex = s.nextFreeIndex()
 885     }
 886
 887     if freeIndex >= s.nelems {
 888         throw("freeIndex is not valid")
 889     }

I accessed the relevant mspan in the core dump by: finding the crashing goroutine by goid, looking up its p, accessing that p's mcache, and then picking out the span class from the backtrace (note that this is reliable because that value never changes for the duration of nextFree, also it lines up with the size passed to mallocgc: span class 0x49 (size class 36) for size 0x600). Note also that s.nextFreeIndex always writes to mspan.freeindex before returning.

When I looked at s.nelems and s.freeindex, I found the values 5 and 0 respectively. That should not be causing the check to fire.

If all this makes sense, then I think that means what we have here is some kind of stack corruption. Specifically, the value of freeIndex on the stack is getting corrupted. Unfortunately I can't figure out how to coax gdb into letting me inspect an off-thread stack and get into that frame to print locals. That would confirm it for sure.

Loading

@bcmills
Copy link
Member Author

@bcmills bcmills commented Nov 17, 2021

Loading

@bcmills bcmills changed the title runtime,cmd/compile: frequent memory corruption on NetBSD since 2021-10-11 runtime,cmd/compile: frequent memory corruption on NetBSD and OpenBSD since 2021-10-11 Nov 17, 2021
@gopherbot
Copy link

@gopherbot gopherbot commented Nov 19, 2021

Change https://golang.org/cl/365776 mentions this issue: dashboard: add n1 hosts for OpenBSD and NetBSD

Loading

gopherbot pushed a commit to golang/build that referenced this issue Nov 19, 2021
We recently migrated in CL 354757 from n1 to e2 hosts around the same
time issues started appearing with these builders. n1 hosts are notably
different in that they are older generation CPUs with hyperthreading
disabled.

For golang/go#49209

Change-Id: I8b6822fbd4910602f61d504b34580282c97c0ca7
Reviewed-on: https://go-review.googlesource.com/c/build/+/365776
Trust: Alexander Rakoczy <alex@golang.org>
Run-TryBot: Alexander Rakoczy <alex@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
@gopherbot
Copy link

@gopherbot gopherbot commented Nov 29, 2021

Change https://golang.org/cl/367534 mentions this issue: dashboard: add n2d hosts for OpenBSD and NetBSD

Loading

gopherbot pushed a commit to golang/build that referenced this issue Nov 29, 2021
We recently migrated in CL 354757 from n1 to e2 hosts around the same
time issues started appearing with these builders. n1 hosts are notably
different in that they are older generation CPUs with hyperthreading
disabled.

e2 hosts may sometimes be AMD CPUs. n2d hosts are always AMD,
so this helps bisect the problem. n2 hosts are always Intel.

For golang/go#49209

Change-Id: I60334b44e192b0b95293891b380f2a89227b9235
Reviewed-on: https://go-review.googlesource.com/c/build/+/367534
Trust: Alexander Rakoczy <alex@golang.org>
Run-TryBot: Alexander Rakoczy <alex@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
@prattmic
Copy link
Member

@prattmic prattmic commented Nov 29, 2021

We have narrowed this down a bit to being specifically related to AMD CPUs. The E2 instances we switched to are a mix of Intel or AMD machines. https://golang.org/cl/367534 added explicit Intel-only (-n2) and AMD-only (-n2d) builders and we found:

  • Near 100% failure rate for AMD openbsd-386, netbsd-386, netbsd-amd64
  • Near 0% failure rate for AMD openbsd-amd64 (but the failures are memory corruption)
  • 0% failure rate for Intel openbsd-386, openbsd-amd64, netbsd-386, netbsd-amd64

Thus far we've only been able to test on GCE instances, but would love to know if these crashes reproduce on OpenBSD/NetBSD on bare-metal AMD machines.

cc @4a6f656c @bsiegert @tklauser or anyone else that may have an OpenBSD or NetBSD AMD machine, just running GOARCH=386 ./all.bash (perhaps a couple of times) should be sufficient to reproduce some kind of memory corruption crash.

Loading

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
7 participants