Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: OpenBSD wirep related failures #49259

Closed
mknyszek opened this issue Nov 1, 2021 · 14 comments
Closed

runtime: OpenBSD wirep related failures #49259

mknyszek opened this issue Nov 1, 2021 · 14 comments
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. okay-after-beta1 Used by release team to mark a release-blocker issue as okay to resolve either before or after beta1 OS-OpenBSD release-blocker
Milestone

Comments

@mknyszek
Copy link
Contributor

mknyszek commented Nov 1, 2021

On trybots and in CI I've noticed a recent uptick in failures on OpenBSD, looking like

2021-11-01T02:47:30-fde4cc2/openbsd-amd64-68
2021-10-29T22:29:31-c812b97/openbsd-386-68
2021-10-28T03:44:12-2ff1074/openbsd-amd64-68

At first I thought this was one of my CLs, but the oldest one seems to indicate it's something else.

There are two other older ones that may be related:

2021-10-26T05:05:24-23fdd7f/openbsd-386-68
2021-10-26T01:22:47-adfb85b/openbsd-386-68

@mknyszek mknyszek added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Nov 1, 2021
@mknyszek mknyszek added this to the Go1.18 milestone Nov 1, 2021
@mknyszek
Copy link
Contributor Author

mknyszek commented Nov 1, 2021

@bcmills
Copy link
Member

bcmills commented Nov 2, 2021

The wirep failure mode may be new, but the OpenBSD runtime crashes are very much not. I suspect that this is a new manifestation of some existing source of memory corruption.

greplogs --dashboard -md -l -e '(?ms)\Aopenbsd-386.*fatal error:' --since=2021-06-01

2021-11-01T13:08:16-732db40/openbsd-386-68
2021-10-29T22:29:31-c812b97/openbsd-386-68
2021-10-26T05:05:24-23fdd7f/openbsd-386-68
2021-10-26T01:22:47-adfb85b/openbsd-386-68
2021-10-21T17:03:18-4100dac-6ec9a1d/openbsd-386-64
2021-10-21T17:03:18-4100dac-2ac3bdf/openbsd-386-64
2021-10-21T15:45:05-9b675d0-6ec9a1d/openbsd-386-64
2021-10-21T13:05:07-9b675d0-8b9c0d1/openbsd-386-64
2021-10-21T02:35:47-9b675d0-fca46d0/openbsd-386-64
2021-10-20T20:58:15-9b675d0-73652af/openbsd-386-64
2021-10-20T20:41:51-9b675d0-2ac3bdf/openbsd-386-64
2021-10-20T18:48:51-cbf1d01-2ac3bdf/openbsd-386-64
2021-10-20T07:21:22-4ea6123-d9421ce/openbsd-386-64
2021-10-20T05:28:45-4ea6123-b091189/openbsd-386-64
2021-10-20T00:05:22-4ea6123-4cc6a91/openbsd-386-64
2021-10-19T23:27:41-4ea6123-c580180/openbsd-386-64
2021-10-19T23:13:35-81f084e-2be5b84/openbsd-386-64
2021-10-19T23:13:35-81f084e-2ac3bdf/openbsd-386-64
2021-10-19T22:56:52-3a9f390-c580180/openbsd-386-64
2021-10-19T22:56:52-3a9f390-2ac3bdf/openbsd-386-64
2021-10-19T21:17:55-9f721e8-c580180/openbsd-386-64
2021-10-19T21:17:55-9f721e8-2ac3bdf/openbsd-386-64
2021-10-19T21:05:11-1feb683-1b24c9e/openbsd-386-64
2021-10-19T20:35:15-1feb683-c580180/openbsd-386-64
2021-10-19T20:35:15-1feb683-07e5527/openbsd-386-64
2021-10-19T18:22:59-98f6e03-6c3cd5d/openbsd-386-64
2021-10-19T17:12:53-98f6e03-404f84d/openbsd-386-64
2021-10-19T17:12:53-404f84d/openbsd-386-68
2021-10-19T17:01:56-98f6e03-99fad12/openbsd-386-64
2021-10-19T16:19:27-98f6e03-f92a358/openbsd-386-64
2021-10-19T15:48:56-98f6e03-067d796/openbsd-386-64
2021-10-19T15:03:39-98f6e03-6294207/openbsd-386-64
2021-10-19T09:49:38-98f6e03-b0351bf/openbsd-386-64
2021-10-19T08:44:22-98f6e03-1e49210/openbsd-386-64
2021-10-18T22:56:07-98f6e03-eba0e86/openbsd-386-64
2021-10-18T21:21:54-98f6e03-6c0daa7/openbsd-386-64
2021-10-18T20:39:22-98f6e03-267abbe/openbsd-386-64
2021-10-18T19:46:27-98f6e03-394a1ad/openbsd-386-64
2021-10-17T04:27:13-98f6e03-cf51fb5/openbsd-386-64
2021-10-16T16:27:40-98f6e03-640a49b/openbsd-386-64
2021-10-15T19:54:44-98f6e03-22951fb/openbsd-386-64
2021-10-15T19:03:33-98f6e03-cfe6763/openbsd-386-64
2021-10-15T16:00:01-98f6e03-fad4a16/openbsd-386-64
2021-10-15T05:20:52-288d0b3-81484cf/openbsd-386-64
2021-10-15T03:49:17-288d0b3-8dab959/openbsd-386-64
2021-10-15T02:45:22-288d0b3-3da0ff8/openbsd-386-64
2021-10-15T02:02:50-ce04ca3-3da0ff8/openbsd-386-64
2021-10-15T01:35:56-ce04ca3-0c45ed0/openbsd-386-64
2021-10-15T00:16:20-ce04ca3-9e8ed86/openbsd-386-64
2021-10-14T22:21:32-3a269dc-c580180/openbsd-386-64
2021-10-14T22:21:32-3a269dc-2ac3bdf/openbsd-386-64
2021-10-14T17:38:39-e69ba9d-011fd00/openbsd-386-64
2021-10-14T15:17:03-e69ba9d-fbdf830/openbsd-386-64
2021-10-14T14:04:14-e69ba9d-2ac3bdf/openbsd-386-64
2021-10-14T14:04:14-e69ba9d-24e798e/openbsd-386-64
2021-10-14T13:09:28-88e80df-24e798e/openbsd-386-64
2021-10-14T12:38:31-88e80df-c580180/openbsd-386-64
2021-10-14T12:38:31-88e80df-2ac3bdf/openbsd-386-64
2021-10-14T07:19:29-acac939-2feb2cc/openbsd-386-64
2021-10-14T04:18:44-acac939-9e4dc6f/openbsd-386-64
2021-10-14T04:18:44-09e6c7a-9e4dc6f/openbsd-386-64
2021-10-14T01:51:22-acac939-276fb27/openbsd-386-64
2021-10-14T00:44:41-acac939-2ac3bdf/openbsd-386-64
2021-10-13T22:21:08-ac2ed98-b90d258/openbsd-386-64
2021-10-13T22:21:08-ac2ed98-2ac3bdf/openbsd-386-64
2021-10-13T20:31:25-378b9e1-b90d258/openbsd-386-64
2021-10-13T17:16:12-378b9e1-4a3daee/openbsd-386-64
2021-10-13T16:41:20-378b9e1-aded167/openbsd-386-64
2021-10-13T15:56:29-378b9e1-5326094/openbsd-386-64
2021-10-13T15:12:46-378b9e1-f687831/openbsd-386-64
2021-10-13T04:56:02-378b9e1-e8f99da/openbsd-386-64
2021-10-13T04:56:02-378b9e1-c580180/openbsd-386-64
2021-10-13T04:56:02-378b9e1-2ac3bdf/openbsd-386-64
2021-10-13T03:12:54-24389d4-c580180/openbsd-386-64
2021-10-13T00:11:47-074820e-4fb2e1c/openbsd-386-64
2021-10-12T22:26:55-074820e-78d01be/openbsd-386-64
2021-10-12T22:12:04-94178a2-732f6fa/openbsd-386-64
2021-10-12T22:12:04-94178a2-2ac3bdf/openbsd-386-64
2021-10-12T21:30:45-0b930fb-3283d1a/openbsd-386-64
2021-10-12T20:20:41-0b930fb-d032b2b/openbsd-386-64
2021-10-12T20:09:58-0b930fb-4186db6/openbsd-386-64
2021-10-12T19:23:00-0b930fb-2ac3bdf/openbsd-386-64
2021-10-12T19:23:00-0b930fb-26b6833/openbsd-386-64
2021-10-12T18:52:28-c83725b-26b6833/openbsd-386-64
2021-10-12T18:00:16-c83725b-6e0adde/openbsd-386-64
2021-10-12T06:55:50-c83725b-d887d3b/openbsd-386-64
2021-10-11T22:57:32-c83725b-d90f0b9/openbsd-386-64
2021-10-11T22:34:49-18fa840-d90f0b9/openbsd-386-64
2021-10-11T20:46:14-18fa840-7023535/openbsd-386-64
2021-10-11T16:48:22-18fa840-aa1c6f5/openbsd-386-64
2021-10-11T16:05:38-18fa840-bd41f2d/openbsd-386-64
2021-10-11T15:23:58-18fa840-2ac3bdf/openbsd-386-64
2021-10-10T13:37:52-1af23bd-b99abf3/openbsd-386-64
2021-10-09T01:04:29-ee04797-e1c294a/openbsd-386-64
2021-10-08T22:21:41-ee04797-2ac3bdf/openbsd-386-64
2021-10-08T21:38:44-c5188f2-74abcab/openbsd-386-64
2021-10-08T19:41:16-c5188f2-7fcf9a1/openbsd-386-64
2021-10-08T18:07:25-c5188f2-d480b5c/openbsd-386-64
2021-10-08T17:58:41-c5188f2-16a3cef/openbsd-386-64
2021-10-08T17:32:31-c5188f2-d7ba1d2/openbsd-386-64
2021-10-08T16:26:20-c5188f2-99c1b24/openbsd-386-64
2021-10-08T15:27:20-c5188f2-b3a4b2d/openbsd-386-64
2021-10-08T14:08:12-c5188f2-5b9206f/openbsd-386-64
2021-10-08T02:09:58-c5188f2-e74db46/openbsd-386-64
2021-10-08T01:18:01-c5188f2-8c4ea31/openbsd-386-64
2021-10-08T00:24:52-c5188f2-78d749f/openbsd-386-64
2021-10-08T00:18:29-c5188f2-7cef831/openbsd-386-64
2021-10-07T20:02:29-d477ef3-0f52292/openbsd-386-64
2021-10-07T19:49:45-d477ef3-c580180/openbsd-386-64
2021-10-07T18:58:33-d477ef3-7286502/openbsd-386-64
2021-10-07T18:45:53-d477ef3-77f2750/openbsd-386-64
2021-10-07T16:38:24-d477ef3-be571a3/openbsd-386-64
2021-10-07T14:54:47-d477ef3-4925e07/openbsd-386-64
2021-10-07T14:37:45-d477ef3-ebeab63/openbsd-386-64
2021-10-07T14:01:52-d477ef3-ecb2f23/openbsd-386-64
2021-10-07T02:30:47-d477ef3-39bbf08/openbsd-386-64
2021-10-07T00:39:17-d477ef3-812a33d/openbsd-386-64
2021-10-06T23:26:57-d477ef3-c8dd89e/openbsd-386-64
2021-10-06T23:01:01-d477ef3-4ffa2f1/openbsd-386-64
2021-10-06T22:42:28-d477ef3-4002616/openbsd-386-64
2021-10-06T22:28:59-d477ef3-b18ba59/openbsd-386-64
2021-10-06T21:06:48-d477ef3-197b328/openbsd-386-64
2021-10-06T20:30:12-d477ef3-4a37a1d/openbsd-386-64
2021-10-06T20:21:49-d477ef3-f580b75/openbsd-386-64
2021-10-06T19:59:27-d477ef3-17c513e/openbsd-386-64
2021-10-06T19:08:15-d477ef3-6487b15/openbsd-386-64
2021-10-06T17:38:01-d477ef3-415f0a3/openbsd-386-64
2021-10-05T22:42:11-2dc2755-96fface/openbsd-386-64
2021-10-05T20:53:02-2dc2755-695a59b/openbsd-386-64
2021-10-05T20:35:54-2dc2755-990c9c6/openbsd-386-64
2021-10-05T20:15:01-2dc2755-0b4d499/openbsd-386-64
2021-10-05T17:10:45-2dc2755-5140ad1/openbsd-386-64
2021-10-05T16:44:45-2dc2755-060cd73/openbsd-386-64
2021-10-05T15:58:05-81efdbc-f1f626d/openbsd-386-64
2021-10-05T15:47:57-81efdbc-017ffcd/openbsd-386-64
2021-10-05T07:32:41-81efdbc-d55009c/openbsd-386-64
2021-10-05T03:23:35-81efdbc-123393a/openbsd-386-64
2021-10-05T03:23:35-81efdbc-04242ac/openbsd-386-64
2021-10-05T00:29:40-36e7bf9-81b7ec1/openbsd-386-64
2021-10-04T23:29:20-36e7bf9-e5f6d8d/openbsd-386-64
2021-10-04T22:59:44-36e7bf9-78c2529/openbsd-386-64
2021-10-04T22:28:37-36e7bf9-752cc07/openbsd-386-64
2021-10-04T22:08:53-db89b5a-7ee4c16/openbsd-386-64
2021-10-04T22:00:53-db89b5a-d9952ff/openbsd-386-64
2021-10-04T21:10:16-db89b5a-f0db7ea/openbsd-386-64
2021-10-04T20:57:50-db89b5a-e8a85e9/openbsd-386-64
2021-10-04T19:52:26-db89b5a-579ff8b/openbsd-386-64
2021-10-04T19:52:26-db89b5a-04242ac/openbsd-386-64
2021-10-04T18:58:42-ccaa907-7d822af/openbsd-386-64
2021-10-04T18:15:09-ccaa907-9f8d558/openbsd-386-64
2021-10-04T17:17:11-ccaa907-cc5e3de/openbsd-386-64
2021-10-04T15:50:43-ccaa907-9432320/openbsd-386-64
2021-10-02T17:29:46-0ebff1a-3bbc823/openbsd-386-64
2021-10-02T16:05:55-0ebff1a-a7fe161/openbsd-386-64
2021-10-02T13:15:42-0ebff1a-7bf7bbc/openbsd-386-64
2021-10-02T00:31:26-0ebff1a-64da5e0/openbsd-386-64
2021-10-02T00:18:01-0ebff1a-cf241a3/openbsd-386-64
2021-10-01T23:51:16-0ebff1a-c129af9/openbsd-386-64
2021-10-01T21:26:43-0ebff1a-7d57324/openbsd-386-64
2021-10-01T21:26:43-0ebff1a-04242ac/openbsd-386-64
2021-10-01T20:36:29-942994f-0d65c27/openbsd-386-64
2021-10-01T20:36:29-942994f-04242ac/openbsd-386-64
2021-10-01T18:18:46-58ad7eb-0d65c27/openbsd-386-64
2021-10-01T16:45:09-58ad7eb-f19b2d5/openbsd-386-64
2021-10-01T16:45:09-58ad7eb-7d57324/openbsd-386-64
2021-10-01T16:45:09-58ad7eb-04242ac/openbsd-386-64
2021-10-01T16:29:05-3883e4a-f19b2d5/openbsd-386-64
2021-10-01T16:29:05-3883e4a-7d57324/openbsd-386-64
2021-10-01T16:29:05-3883e4a-04242ac/openbsd-386-64
2021-10-01T15:56:03-6f5fd9b-8d494b0/openbsd-386-64
2021-10-01T04:28:36-e89823e-6a79f35/openbsd-386-64
2021-09-30T21:33:51-1c35f2a-6a79f35/openbsd-386-64
2021-09-30T20:30:12-1c35f2a-c035d82/openbsd-386-64
2021-09-30T18:10:18-1c35f2a-205640e/openbsd-386-64
2021-09-30T16:43:32-1c35f2a-9112d29/openbsd-386-64
2021-09-30T16:14:49-1c35f2a-b043d23/openbsd-386-64
2021-09-30T15:20:44-1c35f2a-7162c4c/openbsd-386-64
2021-09-29T22:14:44-1c35f2a-2d6d558/openbsd-386-64
2021-09-29T20:45:11-1c35f2a-40fa8c2/openbsd-386-64
2021-09-29T20:06:10-1c35f2a-5930cff/openbsd-386-64
2021-09-29T19:22:48-1c35f2a-82ac9ab/openbsd-386-64
2021-09-29T19:22:48-1c35f2a-04242ac/openbsd-386-64
2021-09-29T19:20:39-7d467dc-82ac9ab/openbsd-386-64
2021-09-29T19:20:39-7d467dc-7d57324/openbsd-386-64
2021-09-29T17:50:31-d9648c9-a9d5ea6/openbsd-386-64
2021-09-29T17:29:23-d9648c9-c9af2bd/openbsd-386-64
2021-09-29T17:29:23-d9648c9-04242ac/openbsd-386-64
2021-09-29T17:15:06-d42c92b-c9af2bd/openbsd-386-64
2021-09-29T16:42:30-d42c92b-7d57324/openbsd-386-64
2021-09-29T16:42:30-d42c92b-04242ac/openbsd-386-64
2021-09-29T15:31:12-5dbd5e4-aeb4fba/openbsd-386-64
2021-09-29T15:31:12-5dbd5e4-04242ac/openbsd-386-64
2021-09-28T23:59:17-df9a164-435718e/openbsd-386-64
2021-09-28T23:30:19-df9a164-b8a6017/openbsd-386-64
2021-09-28T22:34:40-0df0ca0-b8a6017/openbsd-386-64
2021-09-28T22:17:57-7898fe6-b8a6017/openbsd-386-64
2021-09-28T22:17:57-7898fe6-7d57324/openbsd-386-64
2021-09-28T21:42:57-da36f79-b8a6017/openbsd-386-64
2021-09-28T20:53:23-64bbad1-7d57324/openbsd-386-64
2021-09-28T20:53:23-64bbad1-04242ac/openbsd-386-64
2021-09-28T20:40:48-b182fde-daec057/openbsd-386-64
2021-09-28T20:30:45-b182fde-04242ac/openbsd-386-64
2021-09-28T20:12:24-b182fde-e7a2573/openbsd-386-64
2021-09-28T20:12:24-b182fde-c5c1d06/openbsd-386-64
2021-09-28T17:50:53-ba6b94c-8cf0a08/openbsd-386-64
2021-09-28T00:35:10-ba6b94c-850a4ff/openbsd-386-64
2021-09-27T22:22:35-ba6b94c-cd4d592/openbsd-386-64
2021-09-27T21:53:31-ba6b94c-7887313/openbsd-386-64
2021-09-27T21:31:14-ba6b94c-04f7521/openbsd-386-64
2021-09-27T20:42:34-ba6b94c-301f6c8/openbsd-386-64
2021-09-24T21:18:44-ba6b94c-7d57324/openbsd-386-64
2021-09-24T19:46:09-939195f-cfd0868/openbsd-386-64
2021-09-24T19:46:09-939195f-c5c1d06/openbsd-386-64
2021-09-24T19:46:09-939195f-7d57324/openbsd-386-64
2021-09-24T18:21:14-6d1e33f-f6b5ffb/openbsd-386-64
2021-09-24T18:11:24-6d1e33f-812c99f/openbsd-386-64
2021-09-24T14:52:47-6d1e33f-217507e/openbsd-386-64
2021-09-24T04:59:50-6d1e33f-242d02d/openbsd-386-64
2021-09-24T04:32:28-6d1e33f-9ce601d/openbsd-386-64
2021-09-24T04:20:30-6d1e33f-aa878ee/openbsd-386-64
2021-09-24T01:25:24-6d1e33f-483533d/openbsd-386-64
2021-09-24T01:18:01-6d1e33f-0f1159b/openbsd-386-64
2021-09-23T21:59:30-6d1e33f-4e308d7/openbsd-386-64
2021-09-23T20:15:19-6d1e33f-93f964b/openbsd-386-64
2021-09-23T17:47:39-6d1e33f-2fc7697/openbsd-386-64
2021-09-23T17:04:30-6d1e33f-13f3c57/openbsd-386-64
2021-09-23T16:54:46-6d1e33f-5961134/openbsd-386-64
2021-09-22T22:01:15-c8db761-abc4f09/openbsd-386-64
2021-09-22T22:01:15-c8db761-1537f14/openbsd-386-64
2021-09-22T21:47:18-efaec4e-1537f14/openbsd-386-64
2021-09-22T20:53:48-efaec4e-41bb744/openbsd-386-64
2021-09-22T19:37:42-efaec4e-5b75a9b/openbsd-386-64
2021-09-22T18:44:36-efaec4e-7a03ca6/openbsd-386-64
2021-09-22T18:32:21-efaec4e-ccfc41e/openbsd-386-64
2021-09-22T18:10:24-efaec4e-635e493/openbsd-386-64
2021-09-22T15:50:34-efaec4e-abc4f09/openbsd-386-64
2021-09-22T15:47:37-ecbcf3a-552410f/openbsd-386-64
2021-09-22T15:00:53-b98090b-91c2318/openbsd-386-64
2021-09-22T14:28:57-b98090b-dc5b1d3/openbsd-386-64
2021-09-22T13:55:47-b98090b-09d3df0/openbsd-386-64
2021-09-22T03:45:00-b98090b-051df0d/openbsd-386-64
2021-09-22T03:36:42-b98090b-085c609/openbsd-386-64
2021-09-22T00:15:27-b98090b-04572fa/openbsd-386-64
2021-09-21T20:35:14-b98090b-abc4f09/openbsd-386-64
2021-09-21T20:13:48-0d12d39-2f70ce3/openbsd-386-64
2021-09-21T17:25:35-fe076c8-fa5c504/openbsd-386-64
2021-09-21T16:34:59-fe076c8-13aa0d8/openbsd-386-64
2021-09-21T15:38:11-fe076c8-9ab6af9/openbsd-386-64
2021-09-21T14:02:52-fe076c8-7d67f8d/openbsd-386-64
2021-09-21T13:18:09-fe076c8-39e08c6/openbsd-386-64
2021-09-21T03:32:29-fe076c8-c7543e5/openbsd-386-64
2021-09-21T03:32:29-fe076c8-abc4f09/openbsd-386-64
2021-09-21T03:32:29-fe076c8-552410f/openbsd-386-64
2021-09-21T02:24:06-91c880c-986f8ea/openbsd-386-64
2021-09-20T19:55:18-d2417d5-af72ddf/openbsd-386-64
2021-09-20T18:41:57-464a887-3c764ba/openbsd-386-64
2021-09-20T18:36:32-464a887-6268468/openbsd-386-64
2021-09-20T18:06:25-a0016a2-988f18d/openbsd-386-64
2021-09-20T15:19:44-a0016a2-1192135/openbsd-386-64
2021-09-20T15:14:47-d2417d5-6e81f78/openbsd-386-64
2021-09-20T15:08:51-a0016a2-abc4f09/openbsd-386-64
2021-09-19T23:50:45-7559231-315dbd1/openbsd-386-64
2021-09-19T16:34:52-7559231-abc4f09/openbsd-386-64
2021-09-19T16:34:52-7559231-771b8ea/openbsd-386-64
2021-09-17T18:33:15-2758b04-70493b3/openbsd-386-64
2021-09-17T16:17:54-aba0c5f-abc4f09/openbsd-386-64
2021-09-16T23:57:40-1a7ca93-8d2a9c3/openbsd-386-64
2021-09-16T23:13:58-1a7ca93-e09dcc2/openbsd-386-64
2021-09-16T19:38:19-1a7ca93-bcdc61d/openbsd-386-64
2021-09-16T17:44:44-1a7ca93-b1bedc0/openbsd-386-64
2021-09-15T23:28:10-1a7ca93-abc4f09/openbsd-386-64
2021-09-15T22:53:42-1a7ca93-cfa233d/openbsd-386-64
2021-09-15T18:56:22-1a7ca93-170a72e/openbsd-386-64
2021-09-15T17:32:52-76d4494-6196979/openbsd-386-64
2021-09-15T17:19:52-76d4494-21a4e67/openbsd-386-64
2021-09-15T09:43:47-02e5238-21a4e67/openbsd-386-64
2021-09-15T09:43:47-02e5238-170a72e/openbsd-386-64
2021-09-15T00:13:36-0cffec9-21a4e67/openbsd-386-64
2021-09-14T23:55:15-5492d01-bf26e43/openbsd-386-64
2021-09-14T23:07:15-9207707-137543b/openbsd-386-64
2021-09-14T22:36:57-9207707-170a72e/openbsd-386-64
2021-09-14T21:57:30-258ee27-b2c04f0/openbsd-386-64
2021-09-14T19:06:02-258ee27-21a4e67/openbsd-386-64
2021-09-14T19:06:02-258ee27-181e8cd/openbsd-386-64
2021-09-14T19:06:02-258ee27-170a72e/openbsd-386-64
2021-09-14T14:09:54-384e5da-8699425/openbsd-386-64
2021-09-14T13:53:08-384e5da-b3c6de9/openbsd-386-64
2021-09-14T02:53:17-384e5da-ee91bb8/openbsd-386-64
2021-09-14T00:49:39-384e5da-4a4221e/openbsd-386-64
2021-09-14T00:29:16-384e5da-71adc65/openbsd-386-64
2021-09-14T00:16:36-384e5da-146e8d4/openbsd-386-64
2021-09-14T00:08:19-384e5da-21a4e67/openbsd-386-64
2021-09-13T21:48:41-e7de652-42057e9/openbsd-386-64
2021-09-13T18:05:59-915f620-c8a58f2/openbsd-386-64
2021-09-12T01:06:53-915f620-0d8a4bf/openbsd-386-64
2021-09-11T05:05:56-915f620-23832ba/openbsd-386-64
2021-09-11T04:35:30-915f620-cf2fe5d/openbsd-386-64
2021-09-10T10:45:59-295a39b-c69f5c0/openbsd-386-64
2021-09-10T02:44:36-295a39b-2091bd3/openbsd-386-64
2021-09-09T23:18:18-295a39b-b32209d/openbsd-386-64
2021-09-09T22:20:16-295a39b-1a708bc/openbsd-386-64
2021-09-09T21:49:46-295a39b-c981874/openbsd-386-64
2021-09-09T21:49:46-295a39b-21a4e67/openbsd-386-64
2021-09-09T21:49:46-295a39b-170a72e/openbsd-386-64
2021-09-09T21:28:56-c163c31-c981874/openbsd-386-64
2021-09-09T19:43:39-c163c31-2c4f389/openbsd-386-64
2021-09-09T17:02:45-c163c31-f9271e4/openbsd-386-64
2021-09-09T16:48:19-c163c31-ea43445/openbsd-386-64
2021-09-09T15:34:08-c163c31-19457a5/openbsd-386-64
2021-09-09T15:27:16-c163c31-170a72e/openbsd-386-64
2021-09-09T15:07:50-c163c31-a295b3c/openbsd-386-64
2021-09-09T13:25:25-c163c31-66f0d35/openbsd-386-64
2021-09-09T13:25:25-c163c31-328cf2e/openbsd-386-64
2021-09-09T12:57:26-076821b-66f0d35/openbsd-386-64
2021-09-09T09:06:46-076821b-b86e8dd/openbsd-386-64
2021-09-09T04:47:41-076821b-c84f3a4/openbsd-386-64
2021-09-09T04:37:47-076821b-376a079/openbsd-386-64
2021-09-09T04:23:56-076821b-6edc579/openbsd-386-64
2021-09-09T01:25:40-076821b-d62866e/openbsd-386-64
2021-09-09T01:07:21-076821b-42563f8/openbsd-386-64
2021-09-09T00:29:35-076821b-4c52eac/openbsd-386-64
2021-09-09T00:10:46-076821b-e30a090/openbsd-386-64
2021-09-08T23:54:26-076821b-f5f8a91/openbsd-386-64
2021-09-08T23:54:26-076821b-328cf2e/openbsd-386-64
2021-09-08T23:35:19-2cae65c-f5f8a91/openbsd-386-64
2021-09-08T22:47:57-2cae65c-bff39cf/openbsd-386-64
2021-09-08T20:41:51-2cae65c-328cf2e/openbsd-386-64
2021-09-08T19:08:28-2cae65c-fcce86c/openbsd-386-64
2021-09-08T19:08:28-2cae65c-8226cb0/openbsd-386-64
2021-09-08T19:08:28-2cae65c-65f0d24/openbsd-386-64
2021-09-08T18:48:02-0a6f080-65f0d24/openbsd-386-64
2021-09-08T17:59:29-0a6f080-c8d4fe2/openbsd-386-64
2021-09-08T14:51:59-e5f719f-8226cb0/openbsd-386-64
2021-09-08T14:51:59-e5f719f-8214257/openbsd-386-64
2021-09-08T11:57:03-3604566-9295723/openbsd-386-64
2021-09-08T06:59:06-3604566-50c69cc/openbsd-386-64
2021-09-07T22:23:36-3604566-a9a01a3/openbsd-386-64
2021-09-07T20:37:05-3604566-8078355/openbsd-386-64
2021-09-07T19:14:23-3604566-903958d/openbsd-386-64
2021-09-07T17:50:42-3604566-8118866/openbsd-386-64
2021-09-07T17:38:14-3604566-2a48452/openbsd-386-64
2021-09-04T01:07:09-3604566-8226cb0/openbsd-386-64
2021-09-03T22:56:58-d39bbca-4957976/openbsd-386-64
2021-09-03T21:31:30-d39bbca-d9244f8/openbsd-386-64
2021-09-03T19:56:09-d39bbca-04d8d24/openbsd-386-64
2021-09-03T18:04:29-d39bbca-8226cb0/openbsd-386-64
2021-09-03T18:01:36-4e109c5-fcce86c/openbsd-386-64
2021-09-03T17:12:04-7391fc1-fcce86c/openbsd-386-64
2021-09-03T05:27:58-3b801c8-ab7c904/openbsd-386-64
2021-09-03T01:38:54-3b801c8-9f69a44/openbsd-386-64
2021-09-02T19:36:22-3b801c8-a8aa6cf/openbsd-386-64
2021-09-02T19:16:19-3b801c8-9633195/openbsd-386-64
2021-09-02T18:21:15-3b801c8-fcce86c/openbsd-386-64
2021-09-02T18:21:15-3b801c8-8226cb0/openbsd-386-64
2021-09-02T17:30:32-8373dc3-7609b50/openbsd-386-64
2021-09-02T17:00:30-8373dc3-acc2957/openbsd-386-64
2021-09-02T16:37:53-8373dc3-ead3fe0/openbsd-386-64
2021-09-02T16:15:00-8373dc3-d01388b/openbsd-386-64
2021-09-02T15:14:50-8373dc3-782aa42/openbsd-386-64
2021-09-02T15:05:39-8373dc3-d8b1567/openbsd-386-64
2021-09-02T14:44:50-8373dc3-014a972/openbsd-386-64
2021-09-02T14:13:09-8373dc3-8226cb0/openbsd-386-64
2021-09-02T07:31:53-8373dc3-840b429/openbsd-386-64
2021-09-02T06:44:04-8373dc3-1ae7ddc/openbsd-386-64
2021-09-01T22:54:04-8373dc3-fcce86c/openbsd-386-64
2021-09-01T16:41:15-8373dc3-1dd24ca/openbsd-386-64
2021-08-31T21:49:48-70fb962-1f83a8c/openbsd-386-64
2021-08-31T18:41:52-70fb962-4612130/openbsd-386-64
2021-08-31T17:42:36-70fb962-144e0b1/openbsd-386-64
2021-08-31T17:21:38-3e0d083-f27d6a2/openbsd-386-64
2021-08-31T16:43:46-3e0d083-6815235/openbsd-386-64
2021-08-31T16:20:54-3e0d083-605d1aa/openbsd-386-64
2021-08-31T16:14:01-3e0d083-ded10d7/openbsd-386-64
2021-08-31T08:41:16-3e0d083-f118d14/openbsd-386-64
2021-08-30T17:13:00-3e0d083-b602dae/openbsd-386-64
2021-08-30T16:00:10-3e0d083-7b38dd8/openbsd-386-64
2021-08-30T02:40:46-3e0d083-56c3856/openbsd-386-64
2021-08-25T15:30:19-c32dd46-099b819/openbsd-386-64
2021-08-25T03:13:06-c32dd46-e1fcf88/openbsd-386-64
2021-08-24T22:23:12-c32dd46-54cdef1/openbsd-386-64
2021-08-24T18:31:34-c32dd46-c2f96e6/openbsd-386-64
2021-08-24T15:55:38-c32dd46-d70c69d/openbsd-386-64
2021-08-24T14:51:57-c32dd46-ec51703/openbsd-386-64
2021-08-24T00:01:29-d4cc65f-8eeb1bf/openbsd-386-64
2021-08-23T22:55:34-d4cc65f-be1a693/openbsd-386-64
2021-08-23T20:32:04-d4cc65f-22540ab/openbsd-386-64
2021-08-23T20:04:34-d4cc65f-fa34678/openbsd-386-64
2021-08-23T19:49:23-d4cc65f-0a7f00a/openbsd-386-64
2021-08-23T17:29:25-d4cc65f-aeec6db/openbsd-386-64
2021-08-23T16:57:27-d4cc65f-f457ecc/openbsd-386-64
2021-08-23T15:27:42-d4cc65f-f1d8ea1/openbsd-386-64
2021-08-23T13:09:18-d4cc65f-4fbb5c8/openbsd-386-64
2021-08-22T23:49:55-d4cc65f-86ee892/openbsd-386-64
2021-08-20T21:27:50-d4cc65f-ec51703/openbsd-386-64
2021-08-20T21:00:08-bf6c7f2-97d17dc/openbsd-386-64
2021-08-20T19:58:13-bf6c7f2-f67e31d/openbsd-386-64
2021-08-20T19:44:02-bf6c7f2-ab9aaf4/openbsd-386-64
2021-08-20T18:55:57-bf6c7f2-5045477/openbsd-386-64
2021-08-20T18:45:25-bf6c7f2-0f25251/openbsd-386-64
2021-08-20T16:23:47-bf6c7f2-3034463/openbsd-386-64
2021-08-20T03:25:36-bf6c7f2-e9e0d1e/openbsd-386-64
2021-08-19T15:09:47-bf6c7f2-9871726/openbsd-386-64
2021-08-19T09:11:02-bf6c7f2-3bdc179/openbsd-386-64
2021-08-19T00:26:22-bf6c7f2-c85695a/openbsd-386-64
2021-08-18T17:22:40-bf6c7f2-ec51703/openbsd-386-64
2021-08-18T15:11:52-e32504c-0c83e01/openbsd-386-64
2021-08-17T23:59:53-e32504c-ddfcc02/openbsd-386-64
2021-08-17T16:22:15-96f147c-cf12b0d/openbsd-386-64
2021-08-17T15:00:04-96f147c-3001b0a/openbsd-386-64
2021-08-17T14:56:21-96f147c-0d53084/openbsd-386-64
2021-08-17T03:47:47-6932d22-d3deb2c/openbsd-386-64
2021-08-17T02:54:26-6932d22-ec51703/openbsd-386-64
2021-08-17T02:54:26-6932d22-29ec74f/openbsd-386-64
2021-08-17T01:49:12-10bcabd-a8d39f1/openbsd-386-64
2021-08-16T23:57:10-10bcabd-2a19333/openbsd-386-64
2021-08-16T23:03:03-10bcabd-df9c5d8/openbsd-386-64
2021-08-16T21:38:38-10bcabd-c04a32e/openbsd-386-64
2021-08-16T20:51:50-10bcabd-ec51703/openbsd-386-64
2021-08-16T20:23:26-758a1a1-a05a7d4/openbsd-386-64
2021-08-16T20:02:13-758a1a1-ec27168/openbsd-386-64
2021-08-16T18:52:20-758a1a1-d35035f/openbsd-386-64
2021-08-16T17:24:37-758a1a1-850768b/openbsd-386-64
2021-08-16T16:58:24-758a1a1-a0adf91/openbsd-386-64
2021-08-16T16:11:44-758a1a1-5da2010/openbsd-386-64
2021-08-16T15:27:42-a55d515-5da2010/openbsd-386-64
2021-08-16T13:39:19-a55d515-7b7d7d7/openbsd-386-64
2021-08-16T12:54:44-a55d515-c88e3ff/openbsd-386-64
2021-08-16T08:37:46-a55d515-fcdc3c0/openbsd-386-64
2021-08-16T07:44:47-a55d515-160d797/openbsd-386-64
2021-08-15T19:08:56-a55d515-6ed9463/openbsd-386-64
2021-08-15T15:51:49-a55d515-48dfddb/openbsd-386-64
2021-08-15T02:18:46-45389f5-48dfddb/openbsd-386-64
2021-08-14T22:44:35-45389f5-ff3469b/openbsd-386-64
2021-08-13T16:57:31-45389f5-acbe242/openbsd-386-64
2021-08-13T03:36:45-8fae06a-d66b211/openbsd-386-64
2021-08-13T03:36:45-8fae06a-4c8ffb3/openbsd-386-64
2021-08-13T03:35:06-3fce476-4c8ffb3/openbsd-386-64
2021-08-13T02:31:39-03a91dd-d66b211/openbsd-386-64
2021-08-12T21:33:09-d52cb71-0d01934/openbsd-386-64
2021-08-12T21:18:49-d52cb71-acbe242/openbsd-386-64
2021-08-12T21:18:49-d52cb71-46fd547/openbsd-386-64
2021-08-12T18:45:58-9f49561-5805efc/openbsd-386-64
2021-08-12T17:14:32-15a2ab8-095bb79/openbsd-386-64
2021-08-12T16:12:36-749a5cd-acbe242/openbsd-386-64
2021-08-12T15:58:41-f91c4a5-095bb79/openbsd-386-64
2021-08-11T22:07:50-f91c4a5-dea23e9/openbsd-386-64
2021-08-11T16:14:09-5f5a173-d66b211/openbsd-386-64
2021-08-11T15:16:17-5f5a173-acbe242/openbsd-386-64
2021-08-11T15:16:17-5f5a173-1f9c9d8/openbsd-386-64
2021-08-11T04:51:32-92b2fbe-1f9c9d8/openbsd-386-64
2021-08-09T22:50:32-337cebd-acbe242/openbsd-386-64
2021-08-09T20:21:32-fcc905b-fa6aa87/openbsd-386-64
2021-08-09T20:21:32-fcc905b-f1dce31/openbsd-386-64
2021-08-09T20:21:06-bfe69c3-acbe242/openbsd-386-64
2021-08-09T20:20:24-0d28b7d-fa6aa87/openbsd-386-64
2021-08-09T20:20:24-0d28b7d-acbe242/openbsd-386-64
2021-08-09T20:06:35-4869184-f1dce31/openbsd-386-64
2021-08-09T17:59:45-4869184-fa6aa87/openbsd-386-64
2021-08-09T17:59:45-4869184-7aeaad5/openbsd-386-64
2021-08-07T01:29:22-d529aec-891547e/openbsd-386-64
2021-08-06T20:44:03-d529aec-63b968f/openbsd-386-64
2021-08-06T16:51:12-f367f01-70546f6/openbsd-386-64
2021-08-05T20:22:31-f367f01-fd45e26/openbsd-386-64
2021-08-04T20:44:12-f68a40b-fa6aa87/openbsd-386-64
2021-08-04T20:44:05-f68a40b-acbe242/openbsd-386-64
2021-08-04T17:02:12-f68a40b-e90f402/openbsd-386-64
2021-08-04T17:02:12-f68a40b-6e73886/openbsd-386-64
2021-08-04T15:26:45-309db04-6e73886/openbsd-386-64
2021-08-04T13:50:16-309db04-8a7ee4c/openbsd-386-64
2021-08-03T20:45:05-2f64839-8a7ee4c/openbsd-386-64
2021-08-03T19:59:22-32c652e-7921829/openbsd-386-64
2021-08-03T19:59:22-32c652e-16ab7e4/openbsd-386-64
2021-08-03T17:18:10-7a2ec09-8a7ee4c/openbsd-386-64
2021-08-03T16:06:16-594b3a2-8a7ee4c/openbsd-386-64
2021-08-03T15:39:08-3228160-8a7ee4c/openbsd-386-64
2021-08-03T15:00:55-3395cb0-1b68e07/openbsd-386-64
2021-08-02T22:26:52-3395cb0-3d5afa9/openbsd-386-64
2021-08-02T22:23:30-3395cb0-0fb1e14/openbsd-386-64
2021-08-02T22:22:46-3395cb0-ba93baa/openbsd-386-64
2021-08-02T21:22:11-3395cb0-8a7ee4c/openbsd-386-64
2021-08-02T20:18:28-bb69444-accf363/openbsd-386-64
2021-08-02T20:18:07-f8cfada-c6d89db/openbsd-386-64
2021-08-02T20:18:07-f8cfada-accf363/openbsd-386-64
2021-08-02T20:18:07-f8cfada-8a7ee4c/openbsd-386-64
2021-08-02T20:16:24-45eff0f-accf363/openbsd-386-64
2021-08-02T20:16:24-45eff0f-8a7ee4c/openbsd-386-64
2021-08-02T19:06:04-a668498-c6d89db/openbsd-386-64
2021-08-02T17:18:57-3810fa8-8a7ee4c/openbsd-386-64
2021-07-31T00:26:46-3810fa8-c6d89db/openbsd-386-64
2021-07-29T15:30:38-ab1fe72-70fd4e4/openbsd-386-64
2021-07-27T22:01:54-07bc1bf-7cd10c1/openbsd-386-64
2021-07-27T00:15:55-07bc1bf-33ff155/openbsd-386-64
2021-07-26T20:36:31-07bc1bf-ecaa681/openbsd-386-64
2021-07-26T20:19:17-4ad98e9-ed8cbbc/openbsd-386-64
2021-07-26T17:47:47-ebce39e-ecaa681/openbsd-386-64
2021-07-26T16:44:13-4fe0d6c-1868f82/openbsd-386-64
2021-07-26T16:25:21-0c506a2-1868f82/openbsd-386-64
2021-07-23T18:08:39-46d1522-052da57/openbsd-386-64
2021-07-22T21:53:16-c740bfd-ed8cbbc/openbsd-386-64
2021-07-22T17:13:07-251092d-fdb45ac/openbsd-386-64
2021-07-21T20:52:30-7f68387-c6d89db/openbsd-386-64
2021-07-21T20:52:30-7f68387-3e48c03/openbsd-386-64
2021-07-21T20:29:18-7aa8294-3e48c03/openbsd-386-64
2021-07-21T16:31:48-7aa8294-c6d89db/openbsd-386-64
2021-07-21T16:31:48-7aa8294-bc51e93/openbsd-386-64
2021-07-19T18:00:46-6e9046b-404127c/openbsd-386-64
2021-07-19T17:27:58-6e9046b-6298cfe/openbsd-386-64
2021-07-18T21:03:43-6e9046b-a66190e/openbsd-386-64
2021-07-16T02:43:48-6e9046b-aa4e0f5/openbsd-386-64
2021-07-15T20:16:41-6e9046b-69728ea/openbsd-386-64
2021-07-15T17:35:20-6e9046b-c1cc9f9/openbsd-386-64
2021-07-15T13:35:41-6e9046b-c6d89db/openbsd-386-64
2021-07-15T00:09:22-0cf4e27-21a04e3/openbsd-386-64
2021-07-14T22:52:15-0cf4e27-2b00a54/openbsd-386-64
2021-07-14T22:42:09-0cf4e27-60ddf42/openbsd-386-64
2021-07-14T17:25:06-5061c41-60ddf42/openbsd-386-64
2021-07-13T20:15:39-ef97713-a985897/openbsd-386-64
2021-07-13T16:40:23-8e85a28-a985897/openbsd-386-64
2021-07-13T13:36:40-d36a54b-bc51e93/openbsd-386-64
2021-07-13T13:36:40-d36a54b-a985897/openbsd-386-64
2021-07-13T12:52:46-3844600-c6d89db/openbsd-386-64
2021-07-12T22:00:43-de44776-c6d89db/openbsd-386-64
2021-07-12T22:00:43-de44776-bc51e93/openbsd-386-64
2021-07-12T20:58:00-a7dfe3d-a985897/openbsd-386-64
2021-07-12T17:19:11-a7dfe3d-c77980b/openbsd-386-64
2021-07-12T16:18:15-a7dfe3d-fb052db/openbsd-386-64
2021-07-10T00:41:40-cb1acef-fb052db/openbsd-386-64
2021-07-10T00:41:40-cb1acef-3d1d066/openbsd-386-64
2021-07-09T18:00:16-8e32e9f-ab4085c/openbsd-386-64
2021-07-09T17:21:04-8e32e9f-296ddf2/openbsd-386-64
2021-07-08T23:16:08-6994825-fb052db/openbsd-386-64
2021-07-08T21:10:20-e688b94-fb052db/openbsd-386-64
2021-07-08T19:58:58-2f04284-296ddf2/openbsd-386-64
2021-07-08T19:58:23-41a65bd-296ddf2/openbsd-386-64
2021-07-08T19:56:33-febfa9d-3d1d066/openbsd-386-64
2021-07-08T19:56:33-febfa9d-296ddf2/openbsd-386-64
2021-07-08T19:56:07-71eae3a-296ddf2/openbsd-386-64
2021-07-08T19:20:05-55cd480-3d1d066/openbsd-386-64
2021-07-08T17:53:43-55cd480-296ddf2/openbsd-386-64
2021-07-08T16:22:39-55cd480-2ca44fe/openbsd-386-64
2021-07-08T16:04:41-77c1b7a-b51bf4f/openbsd-386-64
2021-07-08T16:04:41-77c1b7a-2ca44fe/openbsd-386-64
2021-07-08T02:47:23-640c1de-5c59e11/openbsd-386-64
2021-07-07T20:06:16-fd00574-c96833e/openbsd-386-64
2021-07-02T20:11:05-7edcfe5-287c5e8/openbsd-386-64
2021-07-01T08:39:23-20dafe5-cb4cd9e/openbsd-386-64
2021-07-01T08:39:23-20dafe5-b51bf4f/openbsd-386-64
2021-07-01T08:39:23-20dafe5-9d65578/openbsd-386-64
2021-06-30T22:02:09-f0847e0-4711bf3/openbsd-386-64
2021-06-30T18:03:06-f0847e0-c080d03/openbsd-386-64
2021-06-30T16:45:04-f0847e0-b51bf4f/openbsd-386-64
2021-06-30T02:00:49-72e4d1b-d19a533/openbsd-386-64
2021-06-30T01:29:49-72e4d1b-c45e800/openbsd-386-64
2021-06-29T19:12:30-72e4d1b-b51bf4f/openbsd-386-64
2021-06-29T17:38:19-100b229-b51bf4f/openbsd-386-64
2021-06-29T14:11:33-100b229-fd4b587/openbsd-386-64
2021-06-29T02:34:20-12f8456-e2e05af/openbsd-386-64
2021-06-27T05:06:30-12f8456-c95464f/openbsd-386-64
2021-06-25T15:37:24-636017e-fd129a6/openbsd-386-64
2021-06-25T15:05:05-48cad5e-37f9a8f/openbsd-386-64
2021-06-24T03:45:33-bfc1674-a9bb382/openbsd-386-64
2021-06-24T03:06:00-bfc1674-44a12e5/openbsd-386-64
2021-06-22T19:12:15-6d3e439-fd129a6/openbsd-386-64
2021-06-22T16:59:10-4c651fc-0ebd5a8/openbsd-386-64
2021-06-22T16:07:17-4c651fc-5bd09e5/openbsd-386-64
2021-06-22T11:31:57-d25f906-63daa77/openbsd-386-64
2021-06-21T22:46:57-d25f906-fd129a6/openbsd-386-64
2021-06-21T21:21:09-d25f906-1bd5a20/openbsd-386-64
2021-06-21T20:53:11-d25f906-761edf7/openbsd-386-64
2021-06-21T16:58:36-d25f906-16e82be/openbsd-386-64
2021-06-18T23:57:09-d25f906-86743e7/openbsd-386-64
2021-06-18T18:34:00-d25f906-57aaa19/openbsd-386-64
2021-06-15T18:42:11-4b484fb-219fe9d/openbsd-386-64
2021-06-15T17:05:31-4b484fb-033d885/openbsd-386-64
2021-06-15T10:13:08-4b484fb-abc56fd/openbsd-386-64
2021-06-14T22:42:48-4b484fb-7841cb1/openbsd-386-64
2021-06-14T19:26:24-4b484fb-d4f34f8/openbsd-386-64
2021-06-11T16:06:07-9a55cb1-2f11284/openbsd-386-64
2021-06-10T18:32:07-9aa007e-dc00dc6/openbsd-386-64
2021-06-10T17:33:43-f6327c5-dc00dc6/openbsd-386-64
2021-06-10T17:33:43-f6327c5-ab7f829/openbsd-386-64
2021-06-09T21:40:20-13cf844-7677616/openbsd-386-64
2021-06-09T15:58:30-e0b9cf7-e6dda19/openbsd-386-64
2021-06-09T15:58:30-e0b9cf7-ab7f829/openbsd-386-64
2021-06-09T15:58:24-16e5f55-e6dda19/openbsd-386-64
2021-06-09T14:15:55-890984b-ab7f829/openbsd-386-64
2021-06-08T16:06:52-4e58f8f-2169deb/openbsd-386-64
2021-06-07T17:02:02-234f954-8212707/openbsd-386-64
2021-06-07T15:22:48-126df1d-ab7f829/openbsd-386-64
2021-06-07T15:22:48-126df1d-7677616/openbsd-386-64
2021-06-05T19:52:26-1225b6f-e1fa260/openbsd-386-64
2021-06-05T09:28:43-1225b6f-e3cb381/openbsd-386-64
2021-06-04T20:51:27-1225b6f-95939e8/openbsd-386-64
2021-06-04T20:34:05-1225b6f-ab7f829/openbsd-386-64
2021-06-04T17:45:44-7295a4e-ab7f829/openbsd-386-64
2021-06-04T17:45:44-7295a4e-831f937/openbsd-386-64
2021-06-04T09:39:32-bf13205-0214440/openbsd-386-64
2021-06-04T01:02:31-bf13205-7677616/openbsd-386-64
2021-06-03T19:38:56-8f2cf6c-6d98301/openbsd-386-64
2021-06-03T17:14:37-4abb1e2-ab7f829/openbsd-386-64
2021-06-03T16:41:39-4abb1e2-e0d029f/openbsd-386-64
2021-06-03T16:41:39-4abb1e2-0d6115c/openbsd-386-64
2021-06-03T03:03:28-1c2154a-6b411e9/openbsd-386-64
2021-06-02T21:39:28-384c392-dd7ba3b/openbsd-386-64
2021-06-02T19:45:53-384c392-4f572d7/openbsd-386-64
2021-06-02T19:26:46-29f5b8f-f2222d8/openbsd-386-64
2021-06-02T19:23:16-7922491-fbf844b/openbsd-386-64
2021-06-02T18:55:23-726034e-dbf69b7/openbsd-386-64
2021-06-02T18:55:23-726034e-448be06/openbsd-386-64
2021-06-02T16:25:20-7271753-6e189af/openbsd-386-64
2021-06-02T01:35:15-377464f-e3c9537/openbsd-386-64
2021-06-01T22:58:04-377464f-0410005/openbsd-386-64
2021-06-01T17:08:12-df07577-2725522/openbsd-386-64

@bcmills
Copy link
Member

bcmills commented Nov 2, 2021

The openbsd-amd64 involvement may be new, though:

greplogs --dashboard -md -l -e '(?ms)\Aopenbsd-amd64.*fatal error:' --since=2021-06-01

2021-11-02T00:12:17-4ff95d6-81fea0b/openbsd-amd64-68
2021-11-01T02:47:30-fde4cc2/openbsd-amd64-68
2021-10-28T03:44:12-2ff1074/openbsd-amd64-68
2021-10-27T18:08:49-08f2e89-bb49eb3/openbsd-amd64-68

@mknyszek
Copy link
Contributor Author

mknyszek commented Nov 2, 2021

Yeah, I think the amd64 one is the new thing. It's also higher priority since that's a trybot and it's causing trybot flakes.

@bcmills
Copy link
Member

bcmills commented Nov 3, 2021

@bcmills
Copy link
Member

bcmills commented Nov 3, 2021

This is a release blocker for 1.18 in that we need to figure out what to do about it.

OpenBSD is not a first-class port, so this does not necessarily block the 1.18 release — but if we are intentionally releasing 1.18 with unstable OpenBSD support its status should be documented in the release notes.

@ALTree
Copy link
Member

ALTree commented Nov 6, 2021

trybot error (not sure if related): https://storage.googleapis.com/go-build-log/25f64550/openbsd-amd64-68_09e5b260.log

fatal error: bad g->status in ready

@mknyszek
Copy link
Contributor Author

mknyszek commented Nov 9, 2021

Spinning up gomotes for this too. Going to attempt at getting a core dump.

@mknyszek
Copy link
Contributor Author

mknyszek commented Nov 10, 2021

Took about 4 hours and 15 minutes of 10 gomotes spinning, but I reproduced it. No core file. :( I'd poke around the gomote but unfortunately I got to it too late. I'll try again tomorrow.

@mknyszek
Copy link
Contributor Author

mknyszek commented Nov 11, 2021

Wow. I've been spinning in 10 gomotes for 22h without a crash.

@bcmills
Copy link
Member

bcmills commented Nov 11, 2021

Neato! Most (but not all) of the openbsd-386 crashes for the past day are in clock_gettime_trampoline. I see one unlock2, and the only amd64 failure is a new-looking netpoll failure mode.

greplogs --dashboard -md -l -e '(?ms)\Aopenbsd-.*fatal error:' --since=2021-11-10

2021-11-11T16:17:21-666fc17/openbsd-386-68
2021-11-11T04:02:33-3949faf/openbsd-386-68
2021-11-10T21:59:24-23f653d/openbsd-amd64-68
2021-11-10T21:53:03-229b909/openbsd-386-68
2021-11-10T18:06:32-097aaa9/openbsd-386-68
2021-11-10T17:15:54-8a3be15/openbsd-386-68
2021-11-10T05:08:25-17980df/openbsd-386-68
2021-11-10T02:26:41-02d7eab/openbsd-386-68
2021-11-10T00:45:37-ec86bb5/openbsd-386-68
2021-11-10T00:44:59-f5f9434/openbsd-386-68

@bcmills
Copy link
Member

bcmills commented Nov 11, 2021

(Filed clock_gettime_trampoline separately as #49532, since it looks like it may be an older bug.)

@toothrot toothrot added the okay-after-beta1 Used by release team to mark a release-blocker issue as okay to resolve either before or after beta1 label Nov 19, 2021
@aclements
Copy link
Member

aclements commented Dec 1, 2021

I think we should focus this issue on the wirep failures, and use #49209 to track the other OpenBSD/NetBSD failures that are more obviously memory corruption. The wirep failures might also be that problem, but I'm not sure.

I think the wirep failures have two modes:

  1. The runtime prints "wirep: invalid p state" and crashes.
  2. The runtime prints "runtime: newstack at runtime.printlock" with a traceback from exitsyscallfast -> wirep. In this case, wirep is attempting to print "wirep: invalid p state" at a time when it can't call splittable functions like print.

Widening the net to both failures modes, we get

greplogs -dashboard -e "(?ms)^fatal error: wirep: invalid p state|^runtime: newstack.*^runtime.wirep" -l -md -since=2021-05-01

2021-11-30T03:11:18-f463b20/openbsd-386-70-n2d
2021-11-30T01:25:55-f90a42b/openbsd-386-70-n2d
2021-11-29T22:57:42-3ca57c7/openbsd-386-70-n2d
2021-11-29T22:25:04-186f375/openbsd-386-70-n2d
2021-11-29T22:02:51-ebd0b77/openbsd-386-70
2021-11-29T22:02:50-bc32dd1/openbsd-386-70-n2d
2021-11-29T22:02:48-2f6d382/openbsd-386-70
2021-11-29T22:02:45-1970e3e/openbsd-386-70-n2d
2021-11-29T19:24:34-68da368/openbsd-386-70
2021-11-29T19:24:34-68da368/openbsd-386-70-n2d
2021-11-29T16:08:23-a59ab29/openbsd-386-70
2021-11-27T19:49:32-a142d65/netbsd-386-9_0-n2d
2021-11-27T19:49:32-a142d65/openbsd-386-70-n2d
2021-11-26T20:49:40-03fcf44-bf88ada/netbsd-amd64-9_0
2021-11-26T13:18:11-7703804/openbsd-386-70
2021-11-26T13:18:11-7703804/openbsd-386-70-n2d
2021-11-25T04:02:39-45bae64/openbsd-386-70-n2d
2021-11-25T00:07:28-cb80a01-f7e34e7/openbsd-386-70
2021-11-24T21:11:34-8cdfe40/openbsd-386-70-n2d
2021-11-24T21:09:36-7e5331a/openbsd-386-70-n2d
2021-11-24T21:08:29-5527d7f/openbsd-386-70-n2d
2021-11-24T20:53:54-a3b8f62/openbsd-386-70-n2d
2021-11-24T19:11:35-4da06e7/openbsd-amd64-68
2021-11-23T17:30:32-12d9285-00045b7/openbsd-386-70
2021-11-22T23:51:43-83bfed9/openbsd-386-68
2021-11-22T12:30:26-b2aa138/openbsd-386-70
2021-11-20T08:47:36-d0c7211-91abe4b/netbsd-amd64-9_0
2021-11-20T01:00:16-be18cd5/openbsd-386-70
2021-11-19T17:42:26-16e8c4c-a944096/netbsd-386-9_0
2021-11-19T00:05:59-e8cda0a/openbsd-386-70
2021-11-18T22:40:42-2375b6e/openbsd-386-70
2021-11-18T16:18:50-feb330d/openbsd-amd64-68
2021-11-17T19:50:58-88474d4/openbsd-386-70
2021-11-17T17:16:18-ab75484/openbsd-386-70
2021-11-17T04:32:05-3c00a28/openbsd-amd64-68
2021-11-17T04:31:50-9c60a09/openbsd-amd64-68
2021-11-15T21:22:15-fda9261/openbsd-386-68
2021-11-13T01:37:51-2fd720b/openbsd-386-68
2021-11-12T23:35:31-b69b2f6/openbsd-386-68
2021-11-12T18:10:48-0c6a6cd/openbsd-386-68
2021-11-12T00:16:15-7bed3c7/openbsd-386-68
2021-11-09T19:01:20-526b2ef/plan9-arm
2021-11-09T19:01:06-81f37a7/freebsd-amd64-race
2021-11-05T22:28:32-feb024f/openbsd-386-68
2021-11-05T21:28:07-ad6ce55/openbsd-386-68
2021-11-03T19:32:33-74f99d0/netbsd-amd64-9_0
2021-11-03T00:07:03-d6f7203/netbsd-amd64-9_0
2021-11-02T23:05:17-eb63cbd/netbsd-386-9_0
2021-11-02T22:20:40-2157498/netbsd-386-9_0
2021-11-01T02:47:30-fde4cc2/openbsd-amd64-68
2021-10-29T22:29:31-c812b97/openbsd-386-68
2021-10-28T03:44:12-2ff1074/openbsd-amd64-68
2021-10-26T01:18:46-49fda9e/freebsd-amd64-12_2
2021-10-25T20:08:25-252324e/freebsd-386-12_2
2021-10-08T19:30:58-f430bda/plan9-arm
2021-10-04T15:50:43-9432320/freebsd-386-12_2
2021-09-23T20:15:19-29bf010-93f964b/freebsd-amd64-12_2
2021-09-13T18:02:42-e74e363/freebsd-386-12_2
2021-09-03T19:56:09-04d8d24/plan9-arm
2021-09-01T21:17:04-ea51e22/plan9-arm
2021-08-31T08:41:16-f118d14/plan9-arm

@aclements
Copy link
Member

aclements commented Dec 1, 2021

Actually, now I'm pretty confident this has the same root cause as #49209. Since 11-24, we've had several failures on the n2d (AMD) builders and none on the n2 (Intel) builders. And if we look at the two crashes that were able to print the invalid p state, in both cases p.m is 0:

greplogs -dashboard -e "(?ms)^fatal error: wirep: invalid p state" -l -md -since=2021-05-01
2021-11-18T16:18:50-feb330d/openbsd-amd64-68
2021-11-17T04:31:50-9c60a09/openbsd-amd64-68

Closing in favor of #49209.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. okay-after-beta1 Used by release team to mark a release-blocker issue as okay to resolve either before or after beta1 OS-OpenBSD release-blocker
Projects
None yet
Development

No branches or pull requests

5 participants