Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: frequent crashes with "exit status 16962" on netbsd-arm64-bsiegert builder #42422

Open
4a6f656c opened this issue Nov 6, 2020 · 38 comments
Labels
arch-arm64 compiler/runtime Issues related to the Go compiler and/or runtime. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-NetBSD
Milestone

Comments

@4a6f656c
Copy link
Contributor

4a6f656c commented Nov 6, 2020

#!watchflakes
post <- goos == "netbsd" && goarch == "arm64" && `exit status 16962`

The netbsd/arm64 builder regularly fails with a process (usually compile) exiting with exit status 16962 - some examples:

https://build.golang.org/log/21955ac5c870203c1e6ff0c9c4f3437c10778dbf
https://build.golang.org/log/f58d31621ecf6eb5317790d7683479bb44b9561d
https://build.golang.org/log/96764e29f501a683cccf7006af49b865d0e2f133

16962 is 0x4242 hexadecimal, which seems to suggest that sigreturn_tramp is failing to call setcontext:

https://github.com/golang/go/blob/master/src/runtime/sys_netbsd_arm64.s#L275

@4a6f656c
Copy link
Contributor Author

4a6f656c commented Nov 6, 2020

/cc @bsiegert

@bcmills bcmills added arch-arm64 OS-NetBSD NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Nov 6, 2020
@bcmills bcmills added this to the Backlog milestone Nov 6, 2020
@bcmills
Copy link
Member

bcmills commented Nov 6, 2020

2020-11-05T22:14:40-8e5778e/netbsd-arm64-bsiegert
2020-11-05T14:54:35-74ec40f/netbsd-arm64-bsiegert
2020-11-05T10:46:08-3510a1e/netbsd-arm64-bsiegert
2020-11-04T21:02:29-e4b4e4b/netbsd-arm64-bsiegert
2020-11-04T16:54:48-594b4a3/netbsd-arm64-bsiegert
2020-11-04T15:53:19-5f0fca1/netbsd-arm64-bsiegert
2020-11-04T06:42:33-633f9e2/netbsd-arm64-bsiegert
2020-11-04T06:14:02-5ca43ac/netbsd-arm64-bsiegert
2020-11-03T15:13:25-ecd7b7e/netbsd-arm64-bsiegert
2020-11-03T12:59:51-393f2bb/netbsd-arm64-bsiegert
2020-11-03T01:27:45-cc0930c/netbsd-arm64-bsiegert
2020-11-03T00:50:57-ebc1b8e/netbsd-arm64-bsiegert
2020-11-02T23:58:08-3a76627/netbsd-arm64-bsiegert
2020-11-02T21:08:14-4fcb506/netbsd-arm64-bsiegert
2020-11-01T01:06:56-063a91c/netbsd-arm64-bsiegert
2020-10-31T17:29:05-9933f66/netbsd-arm64-bsiegert
2020-10-31T08:41:25-f14119b/netbsd-arm64-bsiegert
2020-10-31T00:17:28-07e4f0f/netbsd-arm64-bsiegert
2020-10-30T16:29:11-1af388f/netbsd-arm64-bsiegert
2020-10-30T00:23:50-faa4426/netbsd-arm64-bsiegert
2020-10-29T21:46:54-fe70a3a/netbsd-arm64-bsiegert
2020-10-29T15:27:22-68e30af/netbsd-arm64-bsiegert
2020-10-29T03:06:12-7fe2a84/netbsd-arm64-bsiegert
2020-10-29T01:50:09-308ec22/netbsd-arm64-bsiegert
2020-10-28T16:17:54-421d4e7/netbsd-arm64-bsiegert
2020-10-28T05:02:44-150d244/netbsd-arm64-bsiegert
2020-10-27T21:29:13-009d714/netbsd-arm64-bsiegert
2020-10-27T15:15:13-d73d5d9/netbsd-arm64-bsiegert
2020-10-26T20:12:53-c305e49/netbsd-arm64-bsiegert
2020-10-26T17:27:14-dc02578/netbsd-arm64-bsiegert
2020-10-26T14:50:38-532c199/netbsd-arm64-bsiegert
2020-10-24T22:08:35-d03437a/netbsd-arm64-bsiegert
2020-10-24T03:15:23-bc0b198/netbsd-arm64-bsiegert
2020-10-24T02:36:08-d05c035/netbsd-arm64-bsiegert
2020-10-23T22:33:17-e01a1c0/netbsd-arm64-bsiegert
2020-10-23T21:57:39-a972e8c/netbsd-arm64-bsiegert
2020-10-23T20:54:41-f24ff38/netbsd-arm64-bsiegert
2020-10-23T19:57:55-f8376a5/netbsd-arm64-bsiegert
2020-10-23T19:36:38-3931cc1/netbsd-arm64-bsiegert
2020-10-22T23:12:55-08b3837/netbsd-arm64-bsiegert
2020-10-22T22:40:17-6f7b553/netbsd-arm64-bsiegert
2020-10-22T15:10:01-de74ea5/netbsd-arm64-bsiegert
2020-10-21T23:51:30-cb768c8/netbsd-arm64-bsiegert
2020-10-21T19:08:53-b2a8317/netbsd-arm64-bsiegert
2020-10-21T16:40:47-6f45b39/netbsd-arm64-bsiegert
2020-10-20T23:37:53-f121e0e/netbsd-arm64-bsiegert
2020-10-20T17:53:14-b64202b/netbsd-arm64-bsiegert
2020-10-20T14:04:18-0709e58/netbsd-arm64-bsiegert
2020-10-20T11:09:52-de932da/netbsd-arm64-bsiegert
2020-10-20T08:08:15-55b2d47/netbsd-arm64-bsiegert
2020-10-20T02:32:46-fcb9d6b/netbsd-arm64-bsiegert
2020-10-19T19:51:19-5647d01/netbsd-arm64-bsiegert
2020-10-19T18:48:17-ab541a0/netbsd-arm64-bsiegert
2020-10-19T17:23:00-178b0f5/netbsd-arm64-bsiegert
2020-10-17T22:58:33-fc98165/netbsd-arm64-bsiegert
2020-10-15T16:42:52-623319a/netbsd-arm64-bsiegert
2020-10-14T22:35:32-8cd75f3/netbsd-arm64-bsiegert
2020-10-14T21:32:26-e4ec309/netbsd-arm64-bsiegert
2020-10-14T19:35:59-fbf62be/netbsd-arm64-bsiegert
2020-10-14T08:00:39-e293161/netbsd-arm64-bsiegert
2020-10-13T22:50:26-e43ef8d/netbsd-arm64-bsiegert
2020-10-13T16:27:46-58e51b1/netbsd-arm64-bsiegert
2020-10-13T14:50:14-575ea5a/netbsd-arm64-bsiegert
2020-10-13T13:26:46-2da7054/netbsd-arm64-bsiegert
2020-10-13T03:12:23-c8fdfa7/netbsd-arm64-bsiegert
2020-10-12T20:51:51-3a34395/netbsd-arm64-bsiegert
2020-10-12T18:31:36-8994607/netbsd-arm64-bsiegert
2020-10-12T18:31:29-112c4d5/netbsd-arm64-bsiegert
2020-10-12T18:01:22-e0cb80e/netbsd-arm64-bsiegert
2020-10-10T16:06:07-3036b76/netbsd-arm64-bsiegert
2020-10-10T00:55:54-d317ba5/netbsd-arm64-bsiegert
2020-10-09T21:47:07-712cba3/netbsd-arm64-bsiegert
2020-10-09T15:13:57-eb67eab/netbsd-arm64-bsiegert
2020-10-09T15:11:54-68bc950/netbsd-arm64-bsiegert
2020-10-09T09:55:58-33511fb/netbsd-arm64-bsiegert
2020-10-08T20:35:54-a4b95cd/netbsd-arm64-bsiegert
2020-10-08T18:55:53-f3b58ed/netbsd-arm64-bsiegert
2020-10-08T08:53:13-542693e/netbsd-arm64-bsiegert
2020-10-08T00:28:09-5d13781/netbsd-arm64-bsiegert
2020-10-07T21:18:29-5012e80/netbsd-arm64-bsiegert
2020-10-07T20:07:07-83dfc0d/netbsd-arm64-bsiegert
2020-10-07T19:50:44-470829d/netbsd-arm64-bsiegert
2020-10-06T22:02:30-930fa89/netbsd-arm64-bsiegert
2020-10-06T07:37:55-8e20388/netbsd-arm64-bsiegert
2020-10-06T01:14:39-694025e/netbsd-arm64-bsiegert
2020-10-06T00:34:55-5d12434/netbsd-arm64-bsiegert
2020-10-05T18:52:43-a9c75ec/netbsd-arm64-bsiegert
2020-10-03T17:02:58-869c02c/netbsd-arm64-bsiegert
2020-10-03T14:20:51-bb48f99/netbsd-arm64-bsiegert
2020-10-02T20:42:56-f89d05e/netbsd-arm64-bsiegert
2020-10-01T20:52:34-f4cbf34/netbsd-arm64-bsiegert
2020-10-01T20:36:41-c78c79f/netbsd-arm64-bsiegert
2020-10-01T18:51:18-cc2a5cf/netbsd-arm64-bsiegert
2020-10-01T17:49:01-4ad5dd6/netbsd-arm64-bsiegert
2020-10-01T17:48:42-8c84dcf/netbsd-arm64-bsiegert
2020-10-01T17:47:21-adef4de/netbsd-arm64-bsiegert
2020-10-01T17:46:30-75ea995/netbsd-arm64-bsiegert
2020-10-01T16:43:20-ad8447b/netbsd-arm64-bsiegert
2020-10-01T15:04:29-2ca2e94/netbsd-arm64-bsiegert
2020-09-30T19:59:48-846dce9/netbsd-arm64-bsiegert
2020-09-29T19:02:47-af9c5e5/netbsd-arm64-bsiegert
2020-09-29T19:01:28-567ef8b/netbsd-arm64-bsiegert
2020-09-26T16:04:46-f5c3eda/netbsd-arm64-bsiegert

@bcmills
Copy link
Member

bcmills commented Nov 6, 2020

CC @coypoop (via #29398 / CL 155739)

@coypoop
Copy link
Contributor

coypoop commented Nov 6, 2020

@bsiegert: which version of netbsd/arm64 is this running?
(NetBSD-9 is the first release of NetBSD with arm64 support, so there were so many changes and fixes that it was hard to backport them)

@bsiegert
Copy link
Contributor

bsiegert commented Nov 6, 2020

It's a -current from May:

NetBSD 9.99.63 (GENERIC64) #0: Thu May 21 09:36:24 UTC 2020  mkrepro@mkrepro.NetBSD.org:/usr/src/sys/arch/evbarm/compile/GENERIC64 evbarm

I can try a newer kernel if you would like.

@bsiegert
Copy link
Contributor

bsiegert commented Nov 11, 2020

/cc @tklauser

As a result of an off-list discussion with @coypoop, I upgraded the kernel last weekend to NetBSD 9.99.75. This did not solve the issue. For now, we do not have usable coredumps, which suggests that some sort of memory corruption may be involved.

@4a6f656c
Copy link
Contributor Author

4a6f656c commented Dec 7, 2020

/cc @tklauser

As a result of an off-list discussion with @coypoop, I upgraded the kernel last weekend to NetBSD 9.99.75. This did not solve the issue. For now, we do not have usable coredumps, which suggests that some sort of memory corruption may be involved.

The code in question specifically calls SYS_exit when SYS_setcontext fails - as such, I do not see how you would get a coredump from this. It may be preferable to crash in this case (e.g. dereference an invalid pointer) so that it will generate a coredump, or potentially dump out the ucontext for analysis.

@bcmills
Copy link
Member

bcmills commented Feb 22, 2021

2021-02-21T02:25:41-0f66fb7/netbsd-arm64-bsiegert
2021-02-20T03:54:48-0625460/netbsd-arm64-bsiegert
2021-02-19T23:22:33-26713b5/netbsd-arm64-bsiegert
2021-02-19T16:09:17-02e5a8f/netbsd-arm64-bsiegert
2021-02-19T15:31:06-fa18f22/netbsd-arm64-bsiegert
2021-02-19T15:30:52-49add6a/netbsd-arm64-bsiegert
2021-02-19T08:48:55-7764ee5/netbsd-arm64-bsiegert
2021-02-19T00:41:28-b445d6e/netbsd-arm64-bsiegert
2021-02-19T00:41:03-c7c6c11/netbsd-arm64-bsiegert
2021-02-19T00:40:56-3527caa/netbsd-arm64-bsiegert
2021-02-19T00:40:22-a3b97e7/netbsd-arm64-bsiegert
2021-02-19T00:40:16-985d087/netbsd-arm64-bsiegert
2021-02-19T00:04:49-a1222b7/netbsd-arm64-bsiegert
2021-02-18T18:12:04-f3c2208/netbsd-arm64-bsiegert
2021-02-18T04:30:23-f0be3cc/netbsd-arm64-bsiegert
2021-02-17T00:04:02-70c37ee/netbsd-arm64-bsiegert
2021-02-16T11:01:41-f0d23c9/netbsd-arm64-bsiegert
2021-02-16T02:06:56-626ef08/netbsd-arm64-bsiegert
2021-02-13T15:15:13-66c2709/netbsd-arm64-bsiegert
2021-02-11T18:02:48-864d4f1/netbsd-arm64-bsiegert
2021-02-11T01:10:35-930c2c9/netbsd-arm64-bsiegert
2021-02-09T05:28:50-e0ac989/netbsd-arm64-bsiegert
2021-02-09T01:22:45-cea4e21/netbsd-arm64-bsiegert
2021-02-08T23:32:14-dc725bf/netbsd-arm64-bsiegert
2021-02-08T16:09:01-1901853/netbsd-arm64-bsiegert
2021-02-08T15:56:59-ed3e4af/netbsd-arm64-bsiegert
2021-02-05T23:13:11-724d072/netbsd-arm64-bsiegert
2021-02-02T16:35:53-fca94ab/netbsd-arm64-bsiegert
2021-01-29T19:11:07-68058ed/netbsd-arm64-bsiegert
2021-01-29T18:11:47-0cef57b/netbsd-arm64-bsiegert
2021-01-27T21:04:09-35334ca/netbsd-arm64-bsiegert
2021-01-27T19:17:38-8cfa019/netbsd-arm64-bsiegert
2021-01-27T12:18:00-cd176b3/netbsd-arm64-bsiegert
2021-01-27T12:01:17-5cdf0da/netbsd-arm64-bsiegert
2021-01-27T08:36:43-210f70e/netbsd-arm64-bsiegert
2021-01-26T07:10:57-8634a23/netbsd-arm64-bsiegert
2021-01-25T23:54:44-bf0f7c9/netbsd-arm64-bsiegert
2021-01-25T23:54:36-deaf29a/netbsd-arm64-bsiegert
2021-01-25T22:54:28-e6b6d10/netbsd-arm64-bsiegert
2021-01-25T16:08:46-54b251f/netbsd-arm64-bsiegert
2021-01-25T15:12:58-ff82cc9/netbsd-arm64-bsiegert
2021-01-23T19:46:15-b634f5d/netbsd-arm64-bsiegert
2021-01-23T19:46:06-9897655/netbsd-arm64-bsiegert
2021-01-23T03:19:29-66ee8b1/netbsd-arm64-bsiegert
2021-01-22T21:24:52-3a778ff/netbsd-arm64-bsiegert
2021-01-22T18:23:05-b268b60/netbsd-arm64-bsiegert
2021-01-21T19:08:04-46e2e2e/netbsd-arm64-bsiegert
2021-01-20T16:51:16-7e51fbd/netbsd-arm64-bsiegert
2021-01-19T20:58:35-824f2d6/netbsd-arm64-bsiegert
2021-01-19T19:43:25-5f4716e/netbsd-arm64-bsiegert
2021-01-19T17:33:33-ccb2e90/netbsd-arm64-bsiegert
2021-01-18T17:21:53-5a8fbb0/netbsd-arm64-bsiegert
2021-01-15T20:36:53-b386c73/netbsd-arm64-bsiegert
2021-01-15T18:42:27-bb5075a/netbsd-arm64-bsiegert
2021-01-14T22:01:23-e125ccd/netbsd-arm64-bsiegert
2021-01-14T21:55:29-eb33002/netbsd-arm64-bsiegert
2021-01-14T20:38:03-0c86b99/netbsd-arm64-bsiegert
2021-01-13T23:37:31-7eb31d9/netbsd-arm64-bsiegert
2021-01-12T18:33:07-1e6ecd4/netbsd-arm64-bsiegert
2021-01-08T22:55:41-59bfc18/netbsd-arm64-bsiegert
2021-01-08T18:15:53-6250833/netbsd-arm64-bsiegert
2021-01-08T18:15:43-8f6a9ac/netbsd-arm64-bsiegert
2021-01-08T17:14:20-6598c65/netbsd-arm64-bsiegert
2021-01-08T17:08:44-e65c543/netbsd-arm64-bsiegert
2021-01-08T02:03:24-d92f8ad/netbsd-arm64-bsiegert
2021-01-08T01:48:01-ee4d322/netbsd-arm64-bsiegert
2021-01-08T01:33:44-54bd1cc/netbsd-arm64-bsiegert
2021-01-07T19:31:03-fa90aac/netbsd-arm64-bsiegert
2021-01-07T18:26:51-c4d5d8f/netbsd-arm64-bsiegert
2021-01-06T18:54:25-c9658be/netbsd-arm64-bsiegert
2021-01-05T20:00:43-6b37b15/netbsd-arm64-bsiegert
2021-01-04T12:06:08-b01fb2a/netbsd-arm64-bsiegert
2020-12-30T22:51:30-20d0991/netbsd-arm64-bsiegert
2020-12-30T22:39:59-ed30173/netbsd-arm64-bsiegert
2020-12-30T15:44:40-ef3185b/netbsd-arm64-bsiegert
2020-12-29T21:49:26-780b4de/netbsd-arm64-bsiegert
2020-12-29T20:34:36-f83e0f6/netbsd-arm64-bsiegert
2020-12-28T18:51:49-0661ca7/netbsd-arm64-bsiegert
2020-12-28T17:52:27-13ff221/netbsd-arm64-bsiegert
2020-12-28T17:44:24-0f6027f/netbsd-arm64-bsiegert
2020-12-25T23:54:24-2018b68/netbsd-arm64-bsiegert
2020-12-24T01:40:10-6772e93/netbsd-arm64-bsiegert
2020-12-23T17:31:18-b116404/netbsd-arm64-bsiegert
2020-12-23T17:30:35-fb96f07/netbsd-arm64-bsiegert
2020-12-23T17:29:16-d1502b3/netbsd-arm64-bsiegert
2020-12-23T17:26:56-30c99cb/netbsd-arm64-bsiegert
2020-12-23T17:01:35-49d0b23/netbsd-arm64-bsiegert
2020-12-21T20:10:19-1965356/netbsd-arm64-bsiegert
2020-12-18T23:47:11-626cc7c/netbsd-arm64-bsiegert
2020-12-18T04:42:39-ae652a4/netbsd-arm64-bsiegert
2020-12-18T02:47:24-ae774e9/netbsd-arm64-bsiegert
2020-12-18T00:10:44-f1778c2/netbsd-arm64-bsiegert
2020-12-17T20:04:25-520f3b7/netbsd-arm64-bsiegert
2020-12-16T09:01:04-75e16f5/netbsd-arm64-bsiegert
2020-12-15T16:31:07-a508840/netbsd-arm64-bsiegert
2020-12-15T13:59:00-3298300/netbsd-arm64-bsiegert
2020-12-14T15:03:28-451b6b3/netbsd-arm64-bsiegert
2020-12-11T22:02:02-1430552/netbsd-arm64-bsiegert
2020-12-11T17:26:14-1341a3d/netbsd-arm64-bsiegert
2020-12-11T06:46:11-58e381b/netbsd-arm64-bsiegert
2020-12-10T22:14:50-6a64f6d/netbsd-arm64-bsiegert
2020-12-10T21:08:17-e0d20e5/netbsd-arm64-bsiegert
2020-12-10T21:08:10-d0f40d2/netbsd-arm64-bsiegert
2020-12-10T18:27:38-e5522c8/netbsd-arm64-bsiegert
2020-12-10T16:38:15-56b783a/netbsd-arm64-bsiegert
2020-12-10T16:13:38-b71f123/netbsd-arm64-bsiegert
2020-12-10T15:37:19-0aba8f2/netbsd-arm64-bsiegert
2020-12-09T19:12:27-f1980ef/netbsd-arm64-bsiegert
2020-12-09T16:38:26-db6032d/netbsd-arm64-bsiegert
2020-12-09T12:38:23-ac852fb/netbsd-arm64-bsiegert
2020-12-09T03:01:58-6fa06d9/netbsd-arm64-bsiegert
2020-12-08T22:58:14-31496cf/netbsd-arm64-bsiegert
2020-12-08T19:40:06-5737bec/netbsd-arm64-bsiegert
2020-12-07T21:01:46-7ad6596/netbsd-arm64-bsiegert
2020-12-07T18:02:09-50cdb2d/netbsd-arm64-bsiegert
2020-12-07T10:56:21-ac0ba67/netbsd-arm64-bsiegert
2020-12-07T07:59:54-c155931/netbsd-arm64-bsiegert
2020-12-05T16:20:01-3b2a578/netbsd-arm64-bsiegert
2020-12-05T12:36:42-be9379f/netbsd-arm64-bsiegert
2020-12-04T21:49:30-0b99ea3/netbsd-arm64-bsiegert
2020-12-04T21:35:57-ab68579/netbsd-arm64-bsiegert
2020-12-04T21:17:34-9a0e0bb/netbsd-arm64-bsiegert
2020-12-04T15:43:07-7358064/netbsd-arm64-bsiegert
2020-12-03T16:31:53-dda2991/netbsd-arm64-bsiegert
2020-12-02T20:03:35-bef1c47/netbsd-arm64-bsiegert
2020-12-02T16:19:06-c7110b5/netbsd-arm64-bsiegert
2020-12-01T23:39:15-cf7aa58/netbsd-arm64-bsiegert
2020-12-01T18:02:36-182cd11/netbsd-arm64-bsiegert
2020-12-01T06:27:05-f0400ba/netbsd-arm64-bsiegert
2020-12-01T06:06:10-02d27ec/netbsd-arm64-bsiegert
2020-12-01T05:11:44-f5978a0/netbsd-arm64-bsiegert
2020-11-30T21:08:57-4f42a9b/netbsd-arm64-bsiegert
2020-11-30T17:21:22-294c214/netbsd-arm64-bsiegert
2020-11-30T03:18:36-e5da18d/netbsd-arm64-bsiegert
2020-11-28T03:50:50-358d354/netbsd-arm64-bsiegert
2020-11-25T19:46:00-4481ad6/netbsd-arm64-bsiegert
2020-11-25T16:36:14-9dc2350/netbsd-arm64-bsiegert
2020-11-25T02:18:15-f6dcc97/netbsd-arm64-bsiegert
2020-11-24T16:59:54-ed677e9/netbsd-arm64-bsiegert
2020-11-24T12:36:21-a966329/netbsd-arm64-bsiegert
2020-11-24T03:06:15-7dc5d90/netbsd-arm64-bsiegert
2020-11-23T21:59:33-b5535b1/netbsd-arm64-bsiegert
2020-11-23T05:57:35-d902791/netbsd-arm64-bsiegert
2020-11-21T07:37:02-f734259/netbsd-arm64-bsiegert
2020-11-20T17:35:55-a19c925/netbsd-arm64-bsiegert
2020-11-20T16:23:19-012efc6/netbsd-arm64-bsiegert
2020-11-20T15:53:55-20be4ac/netbsd-arm64-bsiegert
2020-11-20T15:37:42-9264067/netbsd-arm64-bsiegert
2020-11-20T04:21:00-66c0264/netbsd-arm64-bsiegert
2020-11-20T02:27:53-0dcc7d6/netbsd-arm64-bsiegert
2020-11-20T00:54:43-c72a448/netbsd-arm64-bsiegert
2020-11-19T18:23:23-cb674b5/netbsd-arm64-bsiegert
2020-11-19T04:02:59-5ba1c3f/netbsd-arm64-bsiegert
2020-11-19T01:27:31-96b943a/netbsd-arm64-bsiegert
2020-11-18T17:45:08-6ed8ff9/netbsd-arm64-bsiegert
2020-11-18T14:45:14-b194b51/netbsd-arm64-bsiegert
2020-11-18T04:43:50-bcfaeca/netbsd-arm64-bsiegert
2020-11-16T18:39:54-38367d0/netbsd-arm64-bsiegert
2020-11-16T08:22:10-f2eea4c/netbsd-arm64-bsiegert
2020-11-12T21:21:29-f016172/netbsd-arm64-bsiegert
2020-11-12T20:58:06-062e0e5/netbsd-arm64-bsiegert
2020-11-12T20:42:40-1e1fa59/netbsd-arm64-bsiegert
2020-11-12T14:38:38-c167635/netbsd-arm64-bsiegert
2020-11-12T14:17:24-e75aef8/netbsd-arm64-bsiegert
2020-11-12T13:40:55-4bc5f6f/netbsd-arm64-bsiegert
2020-11-11T20:48:31-4c174a7/netbsd-arm64-bsiegert
2020-11-11T13:33:15-69daaf9/netbsd-arm64-bsiegert
2020-11-10T18:33:37-b2ef159/netbsd-arm64-bsiegert
2020-11-10T16:09:36-e3de852/netbsd-arm64-bsiegert
2020-11-10T15:48:04-1899312/netbsd-arm64-bsiegert
2020-11-10T12:42:07-079ba7b/netbsd-arm64-bsiegert
2020-11-10T09:01:41-1c7650a/netbsd-arm64-bsiegert
2020-11-10T03:05:25-169ad6d/netbsd-arm64-bsiegert
2020-11-09T19:00:00-01cdd36/netbsd-arm64-bsiegert
2020-11-09T17:38:50-a6462a6/netbsd-arm64-bsiegert
2020-11-09T14:41:38-feccfb8/netbsd-arm64-bsiegert
2020-11-09T14:17:30-f858c22/netbsd-arm64-bsiegert
2020-11-07T16:59:55-5e371e0/netbsd-arm64-bsiegert
2020-11-07T16:59:40-e8379ab/netbsd-arm64-bsiegert
2020-11-07T14:23:20-bb9a96d/netbsd-arm64-bsiegert
2020-11-07T07:33:23-a6755fc/netbsd-arm64-bsiegert
2020-11-07T03:52:47-d51ae66/netbsd-arm64-bsiegert
2020-11-07T03:19:27-33bc8ce/netbsd-arm64-bsiegert
2020-11-07T02:12:20-c9b9cd7/netbsd-arm64-bsiegert
2020-11-06T23:59:04-f7ef5ca/netbsd-arm64-bsiegert

@bcmills
Copy link
Member

bcmills commented Oct 5, 2021

This continues to occurr frequently on the netbsd-arm64-bsiegert builder. I don't see it at all on the other NetBSD builders.

greplogs --dashboard -md -l -e 'exit status 16962' --since=2021-02-23

2021-10-05T03:23:35-81efdbc-04242ac/netbsd-arm64-bsiegert
2021-10-04T03:19:02-ccaa907-7d57324/netbsd-arm64-bsiegert
2021-10-04T03:19:02-ccaa907-04242ac/netbsd-arm64-bsiegert
2021-10-01T17:40:49-243d65c/netbsd-arm64-bsiegert
2021-10-01T16:45:09-58ad7eb-04242ac/netbsd-arm64-bsiegert
2021-10-01T16:29:05-3883e4a-04242ac/netbsd-arm64-bsiegert
2021-10-01T14:30:33-8d494b0/netbsd-arm64-bsiegert
2021-09-29T22:12:05-88ea8a5/netbsd-arm64-bsiegert
2021-09-29T22:11:50-ed57d7b/netbsd-arm64-bsiegert
2021-09-29T19:22:48-1c35f2a-7d57324/netbsd-arm64-bsiegert
2021-09-29T17:29:23-d9648c9-04242ac/netbsd-arm64-bsiegert
2021-09-28T20:54:26-b8a6017/netbsd-arm64-bsiegert
2021-09-28T03:21:58-534dfb2/netbsd-arm64-bsiegert
2021-09-27T22:57:08-12e8ffc/netbsd-arm64-bsiegert
2021-09-27T22:19:43-45134ac/netbsd-arm64-bsiegert
2021-09-24T18:21:14-f6b5ffb/netbsd-arm64-bsiegert
2021-09-24T04:59:50-242d02d/netbsd-arm64-bsiegert
2021-09-23T17:47:39-2fc7697/netbsd-arm64-bsiegert
2021-09-23T17:04:30-13f3c57/netbsd-arm64-bsiegert
2021-09-22T22:01:15-c8db761-552410f/netbsd-arm64-bsiegert
2021-09-22T19:37:42-5b75a9b/netbsd-arm64-bsiegert
2021-09-22T16:24:17-74ba70b/netbsd-arm64-bsiegert
2021-09-22T03:45:00-051df0d/netbsd-arm64-bsiegert
2021-09-21T18:37:00-0d12d39-552410f/netbsd-arm64-bsiegert
2021-09-21T14:02:52-7d67f8d/netbsd-arm64-bsiegert
2021-09-21T03:32:29-fe076c8-abc4f09/netbsd-arm64-bsiegert
2021-09-21T02:24:06-91c880c-abc4f09/netbsd-arm64-bsiegert
2021-09-21T02:24:06-91c880c-552410f/netbsd-arm64-bsiegert
2021-09-21T00:59:42-986f8ea/netbsd-arm64-bsiegert
2021-09-20T15:19:44-1192135/netbsd-arm64-bsiegert
2021-09-19T16:34:52-7559231-552410f/netbsd-arm64-bsiegert
2021-09-17T23:18:47-ac7c347/netbsd-arm64-bsiegert
2021-09-17T18:33:15-70493b3/netbsd-arm64-bsiegert
2021-09-17T16:17:54-aba0c5f-abc4f09/netbsd-arm64-bsiegert
2021-09-16T17:44:44-b1bedc0/netbsd-arm64-bsiegert
2021-09-16T14:25:50-04f5116/netbsd-arm64-bsiegert
2021-09-15T23:27:54-1a7ca93-552410f/netbsd-arm64-bsiegert
2021-09-14T21:57:30-b2c04f0/netbsd-arm64-bsiegert
2021-09-14T19:06:02-258ee27-170a72e/netbsd-arm64-bsiegert
2021-09-13T18:05:59-c8a58f2/netbsd-arm64-bsiegert
2021-09-13T17:26:12-c3b217a/netbsd-arm64-bsiegert
2021-09-10T17:11:27-a568412-170a72e/netbsd-arm64-bsiegert
2021-09-10T17:09:57-025308f/netbsd-arm64-bsiegert
2021-09-09T22:18:05-426ff37/netbsd-arm64-bsiegert
2021-09-09T19:37:25-fb84e99/netbsd-arm64-bsiegert
2021-09-09T15:34:08-19457a5/netbsd-arm64-bsiegert
2021-09-09T15:27:21-4be982b-21a4e67/netbsd-arm64-bsiegert
2021-09-04T10:58:11-5ec298d/netbsd-arm64-bsiegert
2021-09-04T10:42:46-8a01010/netbsd-arm64-bsiegert
2021-09-03T21:31:30-d9244f8/netbsd-arm64-bsiegert
2021-09-03T18:01:36-4e109c5-fcce86c/netbsd-arm64-bsiegert
2021-09-02T19:56:24-065f380/netbsd-arm64-bsiegert
2021-09-02T16:26:26-08588e6/netbsd-arm64-bsiegert
2021-09-02T13:52:06-a35c5c9/netbsd-arm64-bsiegert
2021-08-31T13:54:19-3920d6f/netbsd-arm64-bsiegert
2021-08-30T22:07:49-b06cfe9/netbsd-arm64-bsiegert
2021-08-30T20:08:45-61120c6/netbsd-arm64-bsiegert
2021-08-30T02:40:46-56c3856/netbsd-arm64-bsiegert
2021-08-28T18:48:37-f4cd001/netbsd-arm64-bsiegert
2021-08-28T16:00:51-d7a43e8/netbsd-arm64-bsiegert
2021-08-27T20:43:38-68ecdc2/netbsd-arm64-bsiegert
2021-08-27T03:24:35-d350a66/netbsd-arm64-bsiegert
2021-08-27T00:36:19-f153b67/netbsd-arm64-bsiegert
2021-08-26T12:19:23-770df2e/netbsd-arm64-bsiegert
2021-08-26T02:06:43-a6ff433/netbsd-arm64-bsiegert
2021-08-25T19:29:15-5baf60d/netbsd-arm64-bsiegert
2021-08-25T18:31:07-4f2ebfe/netbsd-arm64-bsiegert
2021-08-25T03:07:54-41b99da/netbsd-arm64-bsiegert
2021-08-24T22:10:11-5d863f8/netbsd-arm64-bsiegert
2021-08-24T16:55:17-daa55b2/netbsd-arm64-bsiegert
2021-08-24T00:01:29-8eeb1bf/netbsd-arm64-bsiegert
2021-08-23T17:29:25-aeec6db/netbsd-arm64-bsiegert
2021-08-23T15:27:42-f1d8ea1/netbsd-arm64-bsiegert
2021-08-23T12:26:30-c7e354d/netbsd-arm64-bsiegert
2021-08-23T11:23:58-c1a1478/netbsd-arm64-bsiegert
2021-08-22T22:30:37-8fcc614/netbsd-arm64-bsiegert
2021-08-20T18:45:25-0f25251/netbsd-arm64-bsiegert
2021-08-20T03:25:36-e9e0d1e/netbsd-arm64-bsiegert
2021-08-19T09:09:02-740f7d7/netbsd-arm64-bsiegert
2021-08-18T23:19:19-322879d/netbsd-arm64-bsiegert
2021-08-17T02:54:26-6932d22-0d53084/netbsd-arm64-bsiegert
2021-08-14T15:14:35-fc27eb5/netbsd-arm64-bsiegert
2021-08-13T20:51:21-89a4f99/netbsd-arm64-bsiegert
2021-08-13T20:45:17-98f3d7f/netbsd-arm64-bsiegert
2021-08-13T16:34:57-8fae06a-0d53084/netbsd-arm64-bsiegert
2021-08-13T02:31:39-03a91dd-d66b211/netbsd-arm64-bsiegert
2021-08-12T21:29:44-044ec4f/netbsd-arm64-bsiegert
2021-08-12T17:54:49-5805efc/netbsd-arm64-bsiegert
2021-08-12T17:43:16-39634e7/netbsd-arm64-bsiegert
2021-08-12T16:12:36-749a5cd-d66b211/netbsd-arm64-bsiegert
2021-08-12T15:58:41-095bb79/netbsd-arm64-bsiegert
2021-08-11T22:07:50-dea23e9/netbsd-arm64-bsiegert
2021-08-11T16:51:24-d4c0ed2/netbsd-arm64-bsiegert
2021-08-10T01:13:06-aaa1db6-1f9c9d8/netbsd-arm64-bsiegert
2021-08-10T01:13:06-1f9c9d8/netbsd-arm64-bsiegert
2021-08-09T22:50:32-337cebd-f1dce31/netbsd-arm64-bsiegert
2021-08-09T20:06:35-f1dce31/netbsd-arm64-bsiegert
2021-08-08T18:37:08-d529aec-507cc34/netbsd-arm64-bsiegert
2021-08-06T20:44:03-63b968f/netbsd-arm64-bsiegert
2021-08-06T19:38:52-d529aec-fa6aa87/netbsd-arm64-bsiegert
2021-08-04T16:46:57-3115f89-a6ca6d9/netbsd-arm64-bsiegert
2021-08-04T16:46:57-0f9fa26-a6ca6d9/netbsd-arm64-bsiegert
2021-08-03T19:59:22-32c652e-8a7ee4c/netbsd-arm64-bsiegert
2021-08-03T17:18:10-7a2ec09-8a7ee4c/netbsd-arm64-bsiegert
2021-08-03T15:01:00-c6fcb2d-16ab7e4/netbsd-arm64-bsiegert
2021-08-03T15:01:00-3115f89-16ab7e4/netbsd-arm64-bsiegert
2021-08-02T22:28:53-3395cb0-8b6ae9b/netbsd-arm64-bsiegert
2021-08-02T19:06:04-a668498-accf363/netbsd-arm64-bsiegert
2021-07-27T22:01:54-7cd10c1/netbsd-arm64-bsiegert
2021-07-27T22:01:54-07bc1bf-7cd10c1/netbsd-arm64-bsiegert
2021-07-27T05:07:46-7ba8e79/netbsd-arm64-bsiegert
2021-07-27T00:15:55-33ff155/netbsd-arm64-bsiegert
2021-07-26T23:46:13-840e583/netbsd-arm64-bsiegert
2021-07-26T17:47:47-ecaa681/netbsd-arm64-bsiegert
2021-07-26T14:24:27-1868f82/netbsd-arm64-bsiegert
2021-07-25T17:16:20-849b791/netbsd-arm64-bsiegert
2021-07-22T21:53:16-c740bfd-ed8cbbc/netbsd-arm64-bsiegert
2021-07-22T21:53:16-0f08993-ed8cbbc/netbsd-arm64-bsiegert
2021-07-22T19:26:40-798ec73/netbsd-arm64-bsiegert
2021-07-22T17:23:26-c740bfd-bc51e93/netbsd-arm64-bsiegert
2021-07-22T17:13:07-251092d-bc51e93/netbsd-arm64-bsiegert
2021-07-21T20:52:30-7f68387-bc51e93/netbsd-arm64-bsiegert
2021-07-21T20:29:18-3e48c03/netbsd-arm64-bsiegert
2021-07-21T19:25:48-48c88f1/netbsd-arm64-bsiegert
2021-07-19T18:00:46-404127c/netbsd-arm64-bsiegert
2021-07-19T17:27:58-6298cfe/netbsd-arm64-bsiegert
2021-07-15T20:39:22-0941dbc/netbsd-arm64-bsiegert
2021-07-15T13:35:41-6e9046b-bc51e93/netbsd-arm64-bsiegert
2021-07-15T00:09:22-21a04e3/netbsd-arm64-bsiegert
2021-07-14T22:52:15-2b00a54/netbsd-arm64-bsiegert
2021-07-14T22:42:09-0cf4e27-bc51e93/netbsd-arm64-bsiegert
2021-07-13T21:57:39-d8f348a/netbsd-arm64-bsiegert
2021-07-12T22:00:43-de44776-bc51e93/netbsd-arm64-bsiegert
2021-07-12T21:21:15-ccff732-a985897/netbsd-arm64-bsiegert
2021-07-12T20:58:00-a7dfe3d-a985897/netbsd-arm64-bsiegert
2021-07-12T19:45:05-a7dfe3d-bc51e93/netbsd-arm64-bsiegert
2021-07-12T16:23:25-cfbd73b/netbsd-arm64-bsiegert
2021-07-12T16:16:36-980829d-ab4085c/netbsd-arm64-bsiegert
2021-07-10T00:41:40-cb1acef-fb052db/netbsd-arm64-bsiegert
2021-07-08T19:58:58-2f04284-296ddf2/netbsd-arm64-bsiegert
2021-07-08T19:53:41-c979f92-fb052db/netbsd-arm64-bsiegert
2021-07-08T19:17:45-55cd480-fb052db/netbsd-arm64-bsiegert
2021-07-08T16:59:21-ce76298/netbsd-arm64-bsiegert
2021-07-08T16:27:19-55cd480-a91ed83/netbsd-arm64-bsiegert
2021-07-08T16:22:39-55cd480-2ca44fe/netbsd-arm64-bsiegert
2021-07-07T22:29:01-5c59e11/netbsd-arm64-bsiegert
2021-07-07T22:28:19-b003a8b/netbsd-arm64-bsiegert
2021-07-07T22:27:48-11f5df2/netbsd-arm64-bsiegert
2021-07-07T20:26:23-00c0055/netbsd-arm64-bsiegert
2021-07-07T20:25:29-f264879/netbsd-arm64-bsiegert
2021-07-07T20:06:16-fd00574-cb4cd9e/netbsd-arm64-bsiegert
2021-07-07T16:41:12-7edcfe5-c96833e/netbsd-arm64-bsiegert
2021-07-02T21:06:08-7edcfe5-912f075/netbsd-arm64-bsiegert
2021-06-30T22:02:09-f0847e0-4711bf3/netbsd-arm64-bsiegert
2021-06-30T22:02:09-4711bf3/netbsd-arm64-bsiegert
2021-06-30T20:03:34-ed56ea7/netbsd-arm64-bsiegert
2021-06-30T17:58:50-7d0e9e6/netbsd-arm64-bsiegert
2021-06-30T02:00:49-72e4d1b-d19a533/netbsd-arm64-bsiegert
2021-06-29T21:54:24-f9d5095/netbsd-arm64-bsiegert
2021-06-29T21:54:24-72e4d1b-f9d5095/netbsd-arm64-bsiegert
2021-06-29T17:42:45-7dc0b73-cb4cd9e/netbsd-arm64-bsiegert
2021-06-29T17:42:45-04defd4-cb4cd9e/netbsd-arm64-bsiegert
2021-06-28T23:31:13-4bb0847/netbsd-arm64-bsiegert
2021-06-28T23:31:05-1519271/netbsd-arm64-bsiegert
2021-06-28T21:17:28-5385e23/netbsd-arm64-bsiegert
2021-06-28T20:51:30-956c81b/netbsd-arm64-bsiegert
2021-06-28T16:41:43-361159c/netbsd-arm64-bsiegert
2021-06-27T05:06:30-12f8456-c95464f/netbsd-arm64-bsiegert
2021-06-25T21:07:44-ed01cea/netbsd-arm64-bsiegert
2021-06-25T21:04:07-5160896/netbsd-arm64-bsiegert
2021-06-25T19:21:47-00129ff-fd129a6/netbsd-arm64-bsiegert
2021-06-25T19:21:17-fe2294a-d01bc57/netbsd-arm64-bsiegert
2021-06-25T01:08:06-da404b5-37f9a8f/netbsd-arm64-bsiegert
2021-06-24T18:12:53-600a2a4/netbsd-arm64-bsiegert
2021-06-24T04:20:08-4833ac5-fd129a6/netbsd-arm64-bsiegert
2021-06-24T03:45:33-a9bb382/netbsd-arm64-bsiegert
2021-06-24T03:20:33-86d72fa/netbsd-arm64-bsiegert
2021-06-24T00:39:41-d824a74-73496e0/netbsd-arm64-bsiegert
2021-06-24T00:39:41-73496e0/netbsd-arm64-bsiegert
2021-06-23T15:16:55-d824a74-fd129a6/netbsd-arm64-bsiegert
2021-06-22T16:01:58-5bd09e5/netbsd-arm64-bsiegert
2021-06-21T22:46:57-d25f906-fd129a6/netbsd-arm64-bsiegert
2021-06-21T18:59:43-3f9ec83/netbsd-arm64-bsiegert
2021-06-21T16:58:36-16e82be/netbsd-arm64-bsiegert
2021-06-21T14:10:22-117ebe0/netbsd-arm64-bsiegert
2021-06-19T00:46:21-b73cc4b/netbsd-arm64-bsiegert
2021-06-18T18:34:00-d25f906-7677616/netbsd-arm64-bsiegert
2021-06-18T16:56:48-57aaa19/netbsd-arm64-bsiegert
2021-06-17T21:58:54-45f251a/netbsd-arm64-bsiegert
2021-06-17T19:52:18-ed83485/netbsd-arm64-bsiegert
2021-06-17T04:32:50-97cee43/netbsd-arm64-bsiegert
2021-06-16T20:38:07-0e67ce3/netbsd-arm64-bsiegert
2021-06-16T16:38:43-a294e4e/netbsd-arm64-bsiegert
2021-06-16T14:23:50-a6a853f/netbsd-arm64-bsiegert
2021-06-16T06:05:23-785a8f6/netbsd-arm64-bsiegert
2021-06-16T00:40:27-3f7c326-7677616/netbsd-arm64-bsiegert
2021-06-15T20:59:42-d77f4c0/netbsd-arm64-bsiegert
2021-06-15T18:42:11-219fe9d/netbsd-arm64-bsiegert
2021-06-15T18:09:44-723f199/netbsd-arm64-bsiegert
2021-06-15T17:05:31-033d885/netbsd-arm64-bsiegert
2021-06-15T00:14:03-cf4e3e3/netbsd-arm64-bsiegert
2021-06-15T00:14:03-4b484fb-cf4e3e3/netbsd-arm64-bsiegert
2021-06-14T22:42:48-7841cb1/netbsd-arm64-bsiegert
2021-06-14T22:13:47-8a5a6f4/netbsd-arm64-bsiegert
2021-06-14T15:30:14-fdab5be/netbsd-arm64-bsiegert
2021-06-14T07:12:21-3249b64/netbsd-arm64-bsiegert
2021-06-13T23:53:43-4b484fb-14305bf/netbsd-arm64-bsiegert
2021-06-13T23:53:43-14305bf/netbsd-arm64-bsiegert
2021-06-13T08:17:17-4b484fb-24cff0f/netbsd-arm64-bsiegert
2021-06-13T08:17:17-24cff0f/netbsd-arm64-bsiegert
2021-06-12T16:07:12-1ed0d12/netbsd-arm64-bsiegert
2021-06-11T21:43:54-e552a6d/netbsd-arm64-bsiegert
2021-06-11T19:00:10-77aa209/netbsd-arm64-bsiegert
2021-06-11T16:09:15-e2dc6dd/netbsd-arm64-bsiegert
2021-06-11T16:09:15-9a55cb1-e2dc6dd/netbsd-arm64-bsiegert
2021-06-11T16:06:07-9a55cb1-7677616/netbsd-arm64-bsiegert
2021-06-11T14:48:06-2721da2/netbsd-arm64-bsiegert
2021-06-10T20:20:58-770f1de/netbsd-arm64-bsiegert
2021-06-10T19:59:02-8d11b1d/netbsd-arm64-bsiegert
2021-06-10T12:41:37-dc00dc6/netbsd-arm64-bsiegert
2021-06-09T18:34:25-182157c/netbsd-arm64-bsiegert
2021-06-09T18:16:27-1402b27/netbsd-arm64-bsiegert
2021-06-09T17:11:44-df35ade/netbsd-arm64-bsiegert
2021-06-09T16:59:02-e4e7807/netbsd-arm64-bsiegert
2021-06-09T16:53:08-937957b-7677616/netbsd-arm64-bsiegert
2021-06-09T14:15:55-890984b-aa5540c/netbsd-arm64-bsiegert
2021-06-08T22:32:06-63dcab2/netbsd-arm64-bsiegert
2021-06-08T20:39:45-cb80937/netbsd-arm64-bsiegert
2021-06-08T20:23:22-d3e3d03/netbsd-arm64-bsiegert
2021-06-08T20:19:17-da4a640/netbsd-arm64-bsiegert
2021-06-08T20:19:02-689f4c7/netbsd-arm64-bsiegert
2021-06-08T19:49:05-9afe071/netbsd-arm64-bsiegert
2021-06-08T19:41:01-f753d72/netbsd-arm64-bsiegert
2021-06-08T16:06:52-4e58f8f-2169deb/netbsd-arm64-bsiegert
2021-06-08T12:17:50-2169deb/netbsd-arm64-bsiegert
2021-06-08T05:02:19-c20bcb6/netbsd-arm64-bsiegert
2021-06-07T21:12:46-909dd5e/netbsd-arm64-bsiegert
2021-06-05T19:52:26-e1fa260/netbsd-arm64-bsiegert
2021-06-05T04:40:03-9d669ed/netbsd-arm64-bsiegert
2021-06-04T20:51:27-1225b6f-95939e8/netbsd-arm64-bsiegert
2021-06-04T17:21:40-105c5b5/netbsd-arm64-bsiegert
2021-06-04T15:53:06-79cd407/netbsd-arm64-bsiegert
2021-06-04T14:31:24-c6b6211/netbsd-arm64-bsiegert
2021-06-04T09:39:32-0214440/netbsd-arm64-bsiegert
2021-06-03T18:03:08-8f2cf6c-7677616/netbsd-arm64-bsiegert
2021-06-03T17:14:44-4abb1e2-7677616/netbsd-arm64-bsiegert
2021-06-03T08:11:09-ebe580a-0d6115c/netbsd-arm64-bsiegert
2021-06-03T03:03:28-1c2154a-0d6115c/netbsd-arm64-bsiegert
2021-06-02T23:09:01-384c392-0d6115c/netbsd-arm64-bsiegert
2021-06-02T21:39:28-dd7ba3b/netbsd-arm64-bsiegert
2021-06-02T17:34:43-e11d142/netbsd-arm64-bsiegert
2021-06-02T08:17:59-dc8f87b/netbsd-arm64-bsiegert
2021-06-02T08:17:59-377464f-dc8f87b/netbsd-arm64-bsiegert
2021-06-01T16:26:24-2bec019/netbsd-arm64-bsiegert
2021-06-01T12:20:06-0b80cf1/netbsd-arm64-bsiegert
2021-05-30T17:47:50-3b770f2/netbsd-arm64-bsiegert
2021-05-29T01:59:17-79bda65/netbsd-arm64-bsiegert
2021-05-28T18:41:07-5ab822f-1419ca7/netbsd-arm64-bsiegert
2021-05-28T18:41:07-1419ca7/netbsd-arm64-bsiegert
2021-05-28T18:34:18-6f58088/netbsd-arm64-bsiegert
2021-05-28T13:53:35-5ab822f-0410005/netbsd-arm64-bsiegert
2021-05-28T04:09:07-ccd9784/netbsd-arm64-bsiegert
2021-05-27T21:41:58-5ab822f-193d514/netbsd-arm64-bsiegert
2021-05-27T20:59:35-ab2ef4a/netbsd-arm64-bsiegert
2021-05-27T20:29:21-5ab822f-df6a737/netbsd-arm64-bsiegert
2021-05-27T19:55:46-cdcd028/netbsd-arm64-bsiegert
2021-05-27T19:50:37-a3eb095-9210eaf/netbsd-arm64-bsiegert
2021-05-27T18:49:23-8bf5bf5/netbsd-arm64-bsiegert
2021-05-27T15:33:46-fca7b8f/netbsd-arm64-bsiegert
2021-05-27T15:00:58-a3eb095-950fa11/netbsd-arm64-bsiegert
2021-05-27T14:03:15-9bc5268/netbsd-arm64-bsiegert
2021-05-26T23:33:38-a3eb095-3075ffc/netbsd-arm64-bsiegert
2021-05-26T23:33:38-3075ffc/netbsd-arm64-bsiegert
2021-05-26T22:54:09-a62c087/netbsd-arm64-bsiegert
2021-05-26T22:43:54-1d5298d/netbsd-arm64-bsiegert
2021-05-26T18:32:27-55aefbb/netbsd-arm64-bsiegert
2021-05-26T18:24:48-39da9ae/netbsd-arm64-bsiegert
2021-05-26T05:45:56-d050238/netbsd-arm64-bsiegert
2021-05-25T19:05:14-c3e30ff-8b462d7/netbsd-arm64-bsiegert
2021-05-25T19:03:32-f22ec51/netbsd-arm64-bsiegert
2021-05-24T22:27:31-c89f122/netbsd-arm64-bsiegert
2021-05-24T22:12:19-08a8fa9/netbsd-arm64-bsiegert
2021-05-24T20:38:17-d5449c0-305fa95/netbsd-arm64-bsiegert
2021-05-24T19:06:05-873401d/netbsd-arm64-bsiegert
2021-05-24T17:43:50-a22e317/netbsd-arm64-bsiegert
2021-05-24T15:03:18-52d7033/netbsd-arm64-bsiegert
2021-05-22T00:51:17-1e0c960-cca23a7/netbsd-arm64-bsiegert
2021-05-21T22:39:16-217f5dd/netbsd-arm64-bsiegert
2021-05-21T21:07:03-3c65644/netbsd-arm64-bsiegert
2021-05-21T17:49:01-4fb10b2/netbsd-arm64-bsiegert
2021-05-21T16:51:34-3148694/netbsd-arm64-bsiegert
2021-05-21T13:21:00-7e63c8b/netbsd-arm64-bsiegert
2021-05-20T17:40:58-3063790-7c692cc/netbsd-arm64-bsiegert
2021-05-20T16:41:09-f8be906/netbsd-arm64-bsiegert
2021-05-20T16:08:35-bb7495a/netbsd-arm64-bsiegert
2021-05-20T15:46:45-3063790-c7248a0/netbsd-arm64-bsiegert
2021-05-20T14:59:01-0886cdd-c7248a0/netbsd-arm64-bsiegert
2021-05-20T13:19:43-f07e4da/netbsd-arm64-bsiegert
2021-05-20T01:22:28-a8d8591/netbsd-arm64-bsiegert
2021-05-19T15:20:08-6c1c055/netbsd-arm64-bsiegert
2021-05-19T14:12:35-a0f4b7b-04cd717/netbsd-arm64-bsiegert
2021-05-19T01:09:20-17b3466-15a374d/netbsd-arm64-bsiegert
2021-05-19T01:09:20-15a374d/netbsd-arm64-bsiegert
2021-05-18T18:21:53-17b3466-690a8c3/netbsd-arm64-bsiegert
2021-05-18T18:21:53-17b3466-04cd717/netbsd-arm64-bsiegert
2021-05-18T15:00:55-6da3d7a-690a8c3/netbsd-arm64-bsiegert
2021-05-18T02:39:04-5e191f8/netbsd-arm64-bsiegert
2021-05-18T02:12:20-6da3d7a-bfe3573/netbsd-arm64-bsiegert
2021-05-17T22:19:03-bfe3573/netbsd-arm64-bsiegert
2021-05-17T18:03:56-a2c07a9/netbsd-arm64-bsiegert
2021-05-17T17:18:20-8f301ca-b1aff42/netbsd-arm64-bsiegert
2021-05-15T02:39:08-ce92a20/netbsd-arm64-bsiegert
2021-05-14T19:27:25-02699f8/netbsd-arm64-bsiegert
2021-05-14T16:42:01-09ab05b-3d324f1/netbsd-arm64-bsiegert
2021-05-14T16:15:28-0eb38f2/netbsd-arm64-bsiegert
2021-05-14T15:35:28-a938e52/netbsd-arm64-bsiegert
2021-05-14T00:56:52-09ab05b-12d383c/netbsd-arm64-bsiegert
2021-05-13T17:35:42-09ab05b-cde2d85/netbsd-arm64-bsiegert
2021-05-13T12:32:21-a63cded/netbsd-arm64-bsiegert
2021-05-13T09:12:17-2c76a6f/netbsd-arm64-bsiegert
2021-05-12T22:01:29-57c3a74-04cd717/netbsd-arm64-bsiegert
2021-05-12T20:59:48-8287d5d-6db7480/netbsd-arm64-bsiegert
2021-05-12T20:20:11-9dfac01-6db7480/netbsd-arm64-bsiegert
2021-05-12T16:18:01-07ff596/netbsd-arm64-bsiegert
2021-05-12T14:43:27-485474d/netbsd-arm64-bsiegert
2021-05-12T02:04:57-1a0ea1a/netbsd-arm64-bsiegert
2021-05-11T17:48:41-cd1d088-04cd717/netbsd-arm64-bsiegert
2021-05-11T14:51:35-2db0265-2520e72/netbsd-arm64-bsiegert
2021-05-11T14:29:16-9cddb0e-2520e72/netbsd-arm64-bsiegert
2021-05-11T03:28:22-18795da-04cd717/netbsd-arm64-bsiegert
2021-05-11T02:46:21-326a792/netbsd-arm64-bsiegert
2021-05-10T23:21:19-fa05545-04cd717/netbsd-arm64-bsiegert
2021-05-10T21:57:30-5a66778-dc50683/netbsd-arm64-bsiegert
2021-05-10T21:57:30-5a66778-04cd717/netbsd-arm64-bsiegert
2021-05-10T18:10:43-73d5aef/netbsd-arm64-bsiegert
2021-05-10T17:21:32-ecb7392-d9e068d/netbsd-arm64-bsiegert
2021-05-10T17:21:32-d9e068d/netbsd-arm64-bsiegert
2021-05-10T15:53:43-ecb7392-deb3403/netbsd-arm64-bsiegert
2021-05-10T15:53:43-deb3403/netbsd-arm64-bsiegert
2021-05-10T15:48:57-0318541/netbsd-arm64-bsiegert
2021-05-10T13:16:56-2870259/netbsd-arm64-bsiegert
2021-05-09T17:07:22-f05e912-c14ecac/netbsd-arm64-bsiegert
2021-05-09T17:07:01-bedf2c4/netbsd-arm64-bsiegert
2021-05-09T17:06:45-603f43c/netbsd-arm64-bsiegert
2021-05-09T17:05:59-83df4a5/netbsd-arm64-bsiegert
2021-05-09T17:05:25-8ec8f6a/netbsd-arm64-bsiegert
2021-05-09T02:47:29-f05e912-5203357/netbsd-arm64-bsiegert
2021-05-09T00:08:44-f05e912-ea93e68/netbsd-arm64-bsiegert
2021-05-09T00:08:05-14c3d2a/netbsd-arm64-bsiegert
2021-05-09T00:07:34-ec4efa4/netbsd-arm64-bsiegert
2021-05-08T14:59:49-68327e1/netbsd-arm64-bsiegert
2021-05-08T05:57:31-4c8f48e/netbsd-arm64-bsiegert
2021-05-08T05:12:07-02ce411/netbsd-arm64-bsiegert
2021-05-08T03:27:59-b211fe0/netbsd-arm64-bsiegert
2021-05-07T21:35:41-f24eac4/netbsd-arm64-bsiegert
2021-05-07T21:01:09-3980c4d/netbsd-arm64-bsiegert
2021-05-07T18:14:25-af6123a/netbsd-arm64-bsiegert
2021-05-07T16:08:24-9bfa6f7/netbsd-arm64-bsiegert
2021-05-07T14:16:52-a0eb662/netbsd-arm64-bsiegert
2021-05-07T03:55:29-c0140e8-f5423ea/netbsd-arm64-bsiegert
2021-05-07T03:55:29-0714010-f5423ea/netbsd-arm64-bsiegert
2021-05-07T02:45:26-832c70e/netbsd-arm64-bsiegert
2021-05-07T02:05:28-9524e93/netbsd-arm64-bsiegert
2021-05-07T01:42:23-a11a171/netbsd-arm64-bsiegert
2021-05-06T19:28:34-c0140e8-90d6bbb/netbsd-arm64-bsiegert
2021-05-06T19:28:34-c0140e8-04cd717/netbsd-arm64-bsiegert
2021-05-06T18:57:43-d1ea2c7-90d6bbb/netbsd-arm64-bsiegert
2021-05-06T18:08:01-402f177/netbsd-arm64-bsiegert
2021-05-06T17:09:23-51ff3a6/netbsd-arm64-bsiegert
2021-05-06T13:39:39-d2fd503/netbsd-arm64-bsiegert
2021-05-06T11:58:39-b4ca1ce/netbsd-arm64-bsiegert
2021-05-06T03:15:21-d1ea2c7-87ffba3/netbsd-arm64-bsiegert
2021-05-06T03:14:34-08a4f34-87ffba3/netbsd-arm64-bsiegert
2021-05-06T02:20:28-43c390a/netbsd-arm64-bsiegert
2021-05-05T21:37:16-1949673-87ffba3/netbsd-arm64-bsiegert
2021-05-05T09:00:06-4c97912/netbsd-arm64-bsiegert
2021-05-05T04:48:30-9e0facd/netbsd-arm64-bsiegert
2021-05-05T02:47:14-0287a6f-ce04f86/netbsd-arm64-bsiegert
2021-05-05T01:48:39-caf4c94/netbsd-arm64-bsiegert
2021-05-05T01:11:17-bb5e452/netbsd-arm64-bsiegert
2021-05-04T22:12:31-6a6aa32/netbsd-arm64-bsiegert
2021-05-04T18:47:19-8c3d217/netbsd-arm64-bsiegert
2021-05-04T14:38:36-5e4f9b0/netbsd-arm64-bsiegert
2021-05-04T13:37:39-0921211/netbsd-arm64-bsiegert
2021-05-04T03:22:47-138d2c9/netbsd-arm64-bsiegert
2021-05-04T00:56:00-42984c4-e376929/netbsd-arm64-bsiegert
2021-04-30T14:46:28-291330a-d4adea2/netbsd-arm64-bsiegert
2021-04-30T14:46:28-291330a-06ac303/netbsd-arm64-bsiegert
2021-04-30T12:03:55-06ac303/netbsd-arm64-bsiegert
2021-04-30T05:18:37-aec1372-9c72078/netbsd-arm64-bsiegert
2021-04-30T00:13:38-3498027/netbsd-arm64-bsiegert
2021-04-30T00:08:50-02ab8d1/netbsd-arm64-bsiegert
2021-04-29T23:41:22-a9705e1/netbsd-arm64-bsiegert
2021-04-29T23:31:23-1df309e/netbsd-arm64-bsiegert
2021-04-29T20:23:35-7b32830/netbsd-arm64-bsiegert
2021-04-29T04:59:52-6afa0ae/netbsd-arm64-bsiegert
2021-04-29T01:15:34-800adbe-756fd56/netbsd-arm64-bsiegert
2021-04-29T01:15:34-756fd56/netbsd-arm64-bsiegert
2021-04-28T21:52:02-800adbe-d4adea2/netbsd-arm64-bsiegert
2021-04-28T20:36:50-fa6ed6e/netbsd-arm64-bsiegert
2021-04-28T20:16:36-b36596b/netbsd-arm64-bsiegert
2021-04-28T16:06:21-cbb3f09/netbsd-arm64-bsiegert
2021-04-28T01:46:37-16b25d2-f12cf76/netbsd-arm64-bsiegert
2021-04-28T01:19:40-9726c78/netbsd-arm64-bsiegert
2021-04-28T00:49:58-c9f4350/netbsd-arm64-bsiegert
2021-04-27T22:46:28-983dea9/netbsd-arm64-bsiegert
2021-04-27T21:55:07-214c8dd/netbsd-arm64-bsiegert
2021-04-27T21:44:16-645cb62/netbsd-arm64-bsiegert
2021-04-27T21:16:55-30002e6/netbsd-arm64-bsiegert
2021-04-27T05:44:56-cb34026/netbsd-arm64-bsiegert
2021-04-27T04:44:55-c249491/netbsd-arm64-bsiegert
2021-04-27T03:24:21-7ef0237/netbsd-arm64-bsiegert
2021-04-27T01:16:39-40254ec/netbsd-arm64-bsiegert
2021-04-26T23:06:45-e3dc99f-903b251/netbsd-arm64-bsiegert
2021-04-26T14:55:26-00d42ff/netbsd-arm64-bsiegert
2021-04-23T22:31:20-e7db792/netbsd-arm64-bsiegert
2021-04-23T21:48:49-b8fed48/netbsd-arm64-bsiegert
2021-04-23T21:48:41-a6d3dc4/netbsd-arm64-bsiegert
2021-04-23T21:48:37-9cc3469/netbsd-arm64-bsiegert
2021-04-23T21:43:10-bf92160/netbsd-arm64-bsiegert
2021-04-23T21:43:06-e5a6c52/netbsd-arm64-bsiegert
2021-04-23T21:42:52-19470dc/netbsd-arm64-bsiegert
2021-04-23T20:57:54-691e1b8/netbsd-arm64-bsiegert
2021-04-23T19:38:27-768a399/netbsd-arm64-bsiegert
2021-04-23T19:08:32-c3e2ed7/netbsd-arm64-bsiegert
2021-04-23T14:06:54-d4bfe00/netbsd-arm64-bsiegert
2021-04-23T05:10:41-d310b2a/netbsd-arm64-bsiegert
2021-04-23T02:29:37-1b0a031/netbsd-arm64-bsiegert
2021-04-23T00:41:01-14056d0/netbsd-arm64-bsiegert
2021-04-22T22:03:10-1da05eb/netbsd-arm64-bsiegert
2021-04-22T01:40:02-5daefc5/netbsd-arm64-bsiegert
2021-04-21T21:25:26-7e97e4e/netbsd-arm64-bsiegert
2021-04-21T20:44:00-122fca4/netbsd-arm64-bsiegert
2021-04-21T20:39:43-1d21011/netbsd-arm64-bsiegert
2021-04-21T20:24:34-2550563/netbsd-arm64-bsiegert
2021-04-21T16:46:52-35806ef/netbsd-arm64-bsiegert
2021-04-21T14:16:01-e5bc4f2/netbsd-arm64-bsiegert
2021-04-21T13:27:17-381252f/netbsd-arm64-bsiegert
2021-04-20T21:59:20-f448cb8/netbsd-arm64-bsiegert
2021-04-20T21:00:46-57b0d83/netbsd-arm64-bsiegert
2021-04-20T21:00:32-3f8e648/netbsd-arm64-bsiegert
2021-04-20T20:58:52-dbade77/netbsd-arm64-bsiegert
2021-04-20T20:14:14-7473a6a/netbsd-arm64-bsiegert
2021-04-20T18:34:56-77860ad/netbsd-arm64-bsiegert
2021-04-20T16:00:27-60ab197/netbsd-arm64-bsiegert
2021-04-20T15:45:02-9dd71ba/netbsd-arm64-bsiegert
2021-04-20T14:30:03-3ff6ff7/netbsd-arm64-bsiegert
2021-04-20T00:14:27-9f87943/netbsd-arm64-bsiegert
2021-04-19T23:08:21-b3a5640/netbsd-arm64-bsiegert
2021-04-19T22:06:05-bc5de81/netbsd-arm64-bsiegert
2021-04-19T18:37:15-f889214/netbsd-arm64-bsiegert
2021-04-19T02:43:30-c914e61/netbsd-arm64-bsiegert
2021-04-17T00:02:01-fd3612e/netbsd-arm64-bsiegert
2021-04-16T21:50:38-9481789/netbsd-arm64-bsiegert
2021-04-16T21:48:46-b05903a/netbsd-arm64-bsiegert
2021-04-16T21:19:23-9e8a312/netbsd-arm64-bsiegert
2021-04-16T21:03:51-800fb11/netbsd-arm64-bsiegert
2021-04-16T20:12:20-b6e1c33/netbsd-arm64-bsiegert
2021-04-16T19:34:33-fff236e/netbsd-arm64-bsiegert
2021-04-16T19:23:29-ef57834/netbsd-arm64-bsiegert
2021-04-16T19:21:48-dba8928/netbsd-arm64-bsiegert
2021-04-16T18:29:56-492eb05/netbsd-arm64-bsiegert
2021-04-16T16:40:34-acb189e/netbsd-arm64-bsiegert
2021-04-16T16:36:57-2f0e5bf/netbsd-arm64-bsiegert
2021-04-16T16:36:45-abbb829/netbsd-arm64-bsiegert
2021-04-16T14:15:49-0613c74/netbsd-arm64-bsiegert
2021-04-16T13:38:02-52df929/netbsd-arm64-bsiegert
2021-04-16T01:54:27-f08c552/netbsd-arm64-bsiegert
2021-04-15T21:57:02-c8fb0ec/netbsd-arm64-bsiegert
2021-04-15T18:21:15-ddd8d7c/netbsd-arm64-bsiegert
2021-04-15T16:09:16-5631c4b/netbsd-arm64-bsiegert
2021-04-15T15:48:19-1d20a36/netbsd-arm64-bsiegert
2021-04-15T04:10:33-8f4c506/netbsd-arm64-bsiegert
2021-04-15T01:54:41-48b7432/netbsd-arm64-bsiegert
2021-04-15T00:29:05-083a26c/netbsd-arm64-bsiegert
2021-04-15T00:14:55-bf634c7/netbsd-arm64-bsiegert
2021-04-14T23:26:26-567a932/netbsd-arm64-bsiegert
2021-04-14T21:50:26-cbf9caa/netbsd-arm64-bsiegert
2021-04-14T19:25:38-492faae/netbsd-arm64-bsiegert
2021-04-14T18:46:08-4df3d0e/netbsd-arm64-bsiegert
2021-04-14T16:59:24-4480c82/netbsd-arm64-bsiegert
2021-04-14T16:18:34-ad44dfb/netbsd-arm64-bsiegert
2021-04-14T05:49:15-58fdac0/netbsd-arm64-bsiegert
2021-04-14T04:02:01-6d8ba77/netbsd-arm64-bsiegert
2021-04-14T01:28:16-eb433ed/netbsd-arm64-bsiegert
2021-04-13T21:18:03-9913f82/netbsd-arm64-bsiegert
2021-04-13T13:51:17-d948b86/netbsd-arm64-bsiegert
2021-04-12T22:19:11-841bc14/netbsd-arm64-bsiegert
2021-04-12T21:59:20-3d5e3a1/netbsd-arm64-bsiegert
2021-04-12T21:29:02-aad13cb/netbsd-arm64-bsiegert
2021-04-12T19:22:52-1b736b3/netbsd-arm64-bsiegert
2021-04-12T02:29:32-7beb988/netbsd-arm64-bsiegert
2021-04-12T00:45:40-424abc8/netbsd-arm64-bsiegert
2021-04-11T15:56:20-189c694/netbsd-arm64-bsiegert
2021-04-10T19:02:05-36c5f90/netbsd-arm64-bsiegert
2021-04-10T19:02:03-4638545/netbsd-arm64-bsiegert
2021-04-10T03:38:08-6382ec1/netbsd-arm64-bsiegert
2021-04-09T22:57:48-281d168/netbsd-arm64-bsiegert
2021-04-09T14:36:20-77b3269/netbsd-arm64-bsiegert
2021-04-09T14:33:44-a690a5d/netbsd-arm64-bsiegert
2021-04-09T13:19:10-dcc801e/netbsd-arm64-bsiegert
2021-04-09T13:10:36-c432917/netbsd-arm64-bsiegert
2021-04-09T12:56:04-519f223/netbsd-arm64-bsiegert
2021-04-09T00:11:25-d25c4fb/netbsd-arm64-bsiegert
2021-04-09T00:09:47-19034fa/netbsd-arm64-bsiegert
2021-04-08T21:58:35-d67e739/netbsd-arm64-bsiegert
2021-04-08T17:18:24-23e1d36/netbsd-arm64-bsiegert
2021-04-08T07:33:58-a7e16ab/netbsd-arm64-bsiegert
2021-04-07T09:59:58-d6aa162/netbsd-arm64-bsiegert
2021-04-07T03:42:11-8462169/netbsd-arm64-bsiegert
2021-04-06T19:18:47-1271e9a/netbsd-arm64-bsiegert
2021-04-06T14:15:29-b345a30/netbsd-arm64-bsiegert
2021-04-06T13:13:28-939b561/netbsd-arm64-bsiegert
2021-04-05T22:38:48-0723f06/netbsd-arm64-bsiegert
2021-04-05T20:39:06-2701515/netbsd-arm64-bsiegert
2021-04-03T18:16:29-6986c02/netbsd-arm64-bsiegert
2021-04-03T01:45:06-a1e4657/netbsd-arm64-bsiegert
2021-04-03T01:45:04-6454b27/netbsd-arm64-bsiegert
2021-04-02T17:58:59-254948a/netbsd-arm64-bsiegert
2021-04-02T16:53:18-28c5fed/netbsd-arm64-bsiegert
2021-04-02T16:31:47-6996bae/netbsd-arm64-bsiegert
2021-04-02T15:46:42-2d88f8f/netbsd-arm64-bsiegert
2021-04-02T13:30:18-b165085/netbsd-arm64-bsiegert
2021-04-02T01:13:58-41e8a9f/netbsd-arm64-bsiegert
2021-04-02T00:45:56-5579ee1/netbsd-arm64-bsiegert
2021-04-01T17:26:20-e328d63-9baddd3/netbsd-arm64-bsiegert
2021-04-01T05:43:24-43e1dd7-e6ac2df/netbsd-arm64-bsiegert
2021-04-01T00:51:26-ec721d9/netbsd-arm64-bsiegert
2021-04-01T00:51:24-1f29e69/netbsd-arm64-bsiegert
2021-04-01T00:51:23-3304b22/netbsd-arm64-bsiegert
2021-03-31T20:09:03-44dd066/netbsd-arm64-bsiegert
2021-03-31T16:39:16-46fa8af/netbsd-arm64-bsiegert
2021-03-31T13:52:45-c847932/netbsd-arm64-bsiegert
2021-03-31T04:13:28-34fb2b2/netbsd-arm64-bsiegert
2021-03-30T19:44:38-1318fb4/netbsd-arm64-bsiegert
2021-03-30T17:41:58-6cadfe2/netbsd-arm64-bsiegert
2021-03-30T08:39:13-a81b5e4/netbsd-arm64-bsiegert
2021-03-30T03:28:19-3394586/netbsd-arm64-bsiegert
2021-03-29T21:50:16-4e1bf8e/netbsd-arm64-bsiegert
2021-03-29T21:50:14-1ef114d/netbsd-arm64-bsiegert
2021-03-29T20:35:19-164a626/netbsd-arm64-bsiegert
2021-03-29T17:46:51-1a7d921/netbsd-arm64-bsiegert
2021-03-29T16:48:08-2abf280/netbsd-arm64-bsiegert
2021-03-29T15:20:11-8f67614/netbsd-arm64-bsiegert
2021-03-29T06:45:05-6f90ee3/netbsd-arm64-bsiegert
2021-03-29T05:50:11-ba6bd96/netbsd-arm64-bsiegert
2021-03-26T19:40:37-359f449/netbsd-arm64-bsiegert
2021-03-26T06:03:20-98a9023/netbsd-arm64-bsiegert
2021-03-26T01:57:35-3a00618/netbsd-arm64-bsiegert
2021-03-25T21:35:05-374b190/netbsd-arm64-bsiegert
2021-03-25T21:30:55-11b4aee/netbsd-arm64-bsiegert
2021-03-25T19:24:25-569c86d/netbsd-arm64-bsiegert
2021-03-25T19:23:18-ada77d2/netbsd-arm64-bsiegert
2021-03-25T19:20:52-ffa9983/netbsd-arm64-bsiegert
2021-03-25T03:19:56-4889afe/netbsd-arm64-bsiegert
2021-03-25T03:17:51-adb037d/netbsd-arm64-bsiegert
2021-03-25T00:06:23-e7e0995/netbsd-arm64-bsiegert
2021-03-24T20:19:44-dade83a/netbsd-arm64-bsiegert
2021-03-24T19:18:50-fef5a15/netbsd-arm64-bsiegert
2021-03-24T16:45:00-fd5e0bd/netbsd-arm64-bsiegert
2021-03-24T15:51:26-0e31de2/netbsd-arm64-bsiegert
2021-03-24T14:52:14-975b097/netbsd-arm64-bsiegert
2021-03-24T03:03:18-14ef2d8/netbsd-arm64-bsiegert
2021-03-23T23:09:33-87a3ac5/netbsd-arm64-bsiegert
2021-03-23T23:08:19-769d4b6/netbsd-arm64-bsiegert
2021-03-23T19:05:35-2887ef4/netbsd-arm64-bsiegert
2021-03-23T18:14:47-c59b17e/netbsd-arm64-bsiegert
2021-03-23T05:11:09-8f19394/netbsd-arm64-bsiegert
2021-03-23T04:23:52-0265b64/netbsd-arm64-bsiegert
2021-03-22T19:09:53-a93849b/netbsd-arm64-bsiegert
2021-03-22T16:13:55-8fd0f83/netbsd-arm64-bsiegert
2021-03-22T03:52:31-d8394bf/netbsd-arm64-bsiegert
2021-03-20T17:08:03-e0fae78/netbsd-arm64-bsiegert
2021-03-19T23:21:21-635bc2c-6ae3b70/netbsd-arm64-bsiegert
2021-03-19T03:05:26-c4fcb01-ed3ae9a/netbsd-arm64-bsiegert
2021-03-18T21:27:24-095ba22/netbsd-arm64-bsiegert
2021-03-18T18:50:58-b95e4b7/netbsd-arm64-bsiegert
2021-03-18T16:51:25-c71acbf/netbsd-arm64-bsiegert
2021-03-18T16:51:20-06ca809/netbsd-arm64-bsiegert
2021-03-18T15:45:32-b7cb92a/netbsd-arm64-bsiegert
2021-03-18T14:20:08-6b6ea32/netbsd-arm64-bsiegert
2021-03-18T08:25:24-732ea4c/netbsd-arm64-bsiegert
2021-03-18T01:46:58-f5e6d3e/netbsd-arm64-bsiegert
2021-03-17T22:14:28-f38b642/netbsd-arm64-bsiegert
2021-03-17T21:48:40-f82ce7f/netbsd-arm64-bsiegert
2021-03-17T21:24:05-7e00049/netbsd-arm64-bsiegert
2021-03-17T07:01:00-72b501c/netbsd-arm64-bsiegert
2021-03-17T00:54:09-a826f7d/netbsd-arm64-bsiegert
2021-03-17T00:17:33-68f8e1a/netbsd-arm64-bsiegert
2021-03-16T13:06:17-120b9eb/netbsd-arm64-bsiegert
2021-03-14T21:50:39-d0d38f0/netbsd-arm64-bsiegert
2021-03-14T21:10:45-3cdd5c3/netbsd-arm64-bsiegert
2021-03-14T19:58:14-a8b59fe/netbsd-arm64-bsiegert
2021-03-14T13:09:57-88b8a16/netbsd-arm64-bsiegert
2021-03-13T19:03:29-7936efe/netbsd-arm64-bsiegert
2021-03-13T18:35:01-1767d2c/netbsd-arm64-bsiegert
2021-03-13T11:26:35-83e79c7/netbsd-arm64-bsiegert
2021-03-13T00:39:23-7588ef0/netbsd-arm64-bsiegert
2021-03-13T00:38:58-16ad1ea/netbsd-arm64-bsiegert
2021-03-13T00:32:00-7b47f9a/netbsd-arm64-bsiegert
2021-03-12T21:22:32-86bbf4b/netbsd-arm64-bsiegert
2021-03-12T21:19:20-78052f4/netbsd-arm64-bsiegert
2021-03-12T16:00:02-9289c12/netbsd-arm64-bsiegert
2021-03-12T02:56:04-e8b8278/netbsd-arm64-bsiegert
2021-03-12T02:30:33-e87c4bb/netbsd-arm64-bsiegert
2021-03-12T01:47:01-a607408/netbsd-arm64-bsiegert
2021-03-12T00:52:00-71a6c13/netbsd-arm64-bsiegert
2021-03-11T20:01:00-4dd9c7c/netbsd-arm64-bsiegert
2021-03-11T13:49:01-b8e9ec8/netbsd-arm64-bsiegert
2021-03-11T08:19:39-9ece63f/netbsd-arm64-bsiegert
2021-03-11T00:31:14-fdded79/netbsd-arm64-bsiegert
2021-03-10T17:10:00-5edab39/netbsd-arm64-bsiegert
2021-03-10T17:07:49-7457462/netbsd-arm64-bsiegert
2021-03-10T14:45:43-30c28bb/netbsd-arm64-bsiegert
2021-03-10T03:52:48-643d240/netbsd-arm64-bsiegert
2021-03-09T19:25:34-e4f3cfa/netbsd-arm64-bsiegert
2021-03-09T10:12:02-762ef81/netbsd-arm64-bsiegert
2021-03-09T07:44:18-437d229/netbsd-arm64-bsiegert
2021-03-08T23:13:52-b6def6a/netbsd-arm64-bsiegert
2021-03-08T19:43:19-fee3cd4/netbsd-arm64-bsiegert
2021-03-05T23:58:34-414fa8c/netbsd-arm64-bsiegert
2021-03-05T23:47:39-009bfea/netbsd-arm64-bsiegert
2021-03-05T22:42:10-a22bd3d/netbsd-arm64-bsiegert
2021-03-05T22:14:48-a829114/netbsd-arm64-bsiegert
2021-03-05T18:47:16-44721f4/netbsd-arm64-bsiegert
2021-03-05T18:46:48-fbee173/netbsd-arm64-bsiegert
2021-03-05T18:46:43-9e6b1fc/netbsd-arm64-bsiegert
2021-03-05T18:45:59-70b277c/netbsd-arm64-bsiegert
2021-03-05T17:46:57-67b9ecb/netbsd-arm64-bsiegert
2021-03-05T15:34:32-a2f7067/netbsd-arm64-bsiegert
2021-03-05T06:18:33-31df4e3/netbsd-arm64-bsiegert
2021-03-05T04:30:32-f0b6d37/netbsd-arm64-bsiegert
2021-03-05T02:28:11-a7526bb/netbsd-arm64-bsiegert
2021-03-04T23:07:50-5c5552c/netbsd-arm64-bsiegert
2021-03-04T20:18:16-c015f76/netbsd-arm64-bsiegert
2021-03-04T17:47:28-cfb609b/netbsd-arm64-bsiegert
2021-03-04T16:19:12-a2d92b5/netbsd-arm64-bsiegert
2021-03-04T06:35:44-b7f4307/netbsd-arm64-bsiegert
2021-03-04T03:30:55-4532467/netbsd-arm64-bsiegert
2021-03-04T02:09:42-04a4dca/netbsd-arm64-bsiegert
2021-03-04T01:30:59-593f5bb/netbsd-arm64-bsiegert
2021-03-04T01:28:21-355c3a0/netbsd-arm64-bsiegert
2021-03-04T01:26:52-726d704/netbsd-arm64-bsiegert
2021-03-03T20:01:55-6db80d7/netbsd-arm64-bsiegert
2021-03-03T06:52:41-85f62b0/netbsd-arm64-bsiegert
2021-03-03T05:35:24-497feff/netbsd-arm64-bsiegert
2021-03-03T02:31:34-06c72f3/netbsd-arm64-bsiegert
2021-03-02T20:29:03-2a2f99e/netbsd-arm64-bsiegert
2021-03-02T06:08:56-ebb92df/netbsd-arm64-bsiegert
2021-03-02T03:46:25-4c1a7ab/netbsd-arm64-bsiegert
2021-03-02T00:51:00-ff5cf4c/netbsd-arm64-bsiegert
2021-03-01T15:24:01-a400eb3/netbsd-arm64-bsiegert
2021-02-27T18:38:31-998fe70/netbsd-arm64-bsiegert
2021-02-27T03:03:29-d9fd38e/netbsd-arm64-bsiegert
2021-02-26T10:18:26-23943a6/netbsd-arm64-bsiegert
2021-02-26T02:52:33-e25040d/netbsd-arm64-bsiegert
2021-02-25T22:38:21-5f15af1/netbsd-arm64-bsiegert
2021-02-25T21:52:49-9a7fe19/netbsd-arm64-bsiegert
2021-02-25T21:21:51-b83d073/netbsd-arm64-bsiegert
2021-02-25T18:49:25-1a3e968/netbsd-arm64-bsiegert
2021-02-25T18:43:17-ee2a45e/netbsd-arm64-bsiegert
2021-02-25T17:50:59-bcac57f/netbsd-arm64-bsiegert
2021-02-25T17:28:44-3137da8/netbsd-arm64-bsiegert
2021-02-25T16:45:36-2c4c189/netbsd-arm64-bsiegert
2021-02-25T15:23:02-ad17b65/netbsd-arm64-bsiegert
2021-02-24T21:34:21-8027343/netbsd-arm64-bsiegert
2021-02-24T19:26:13-80ddc17/netbsd-arm64-bsiegert
2021-02-24T19:26:11-3deb528/netbsd-arm64-bsiegert
2021-02-24T18:03:45-dc4698f/netbsd-arm64-bsiegert
2021-02-24T17:54:29-7a2f327/netbsd-arm64-bsiegert
2021-02-24T17:00:14-adb467f/netbsd-arm64-bsiegert
2021-02-24T15:12:08-07c6583/netbsd-arm64-bsiegert
2021-02-24T14:12:16-04edf41/netbsd-arm64-bsiegert
2021-02-24T14:10:04-e496120/netbsd-arm64-bsiegert
2021-02-24T08:20:45-eb86324/netbsd-arm64-bsiegert
2021-02-23T15:04:12-ab331c0/netbsd-arm64-bsiegert
2021-02-23T09:16:54-08543f0/netbsd-arm64-bsiegert
2021-02-23T06:01:42-2a18e37/netbsd-arm64-bsiegert
2021-02-23T06:01:29-e521498/netbsd-arm64-bsiegert
2021-02-23T05:02:42-86deb45/netbsd-arm64-bsiegert
2021-02-23T01:53:51-5f3dabb/netbsd-arm64-bsiegert
2021-02-23T01:37:31-7af821a/netbsd-arm64-bsiegert
2021-02-23T00:20:38-f113e9a/netbsd-arm64-bsiegert

@bcmills bcmills changed the title runtime: netbsd/arm64 seems to be flakey - exit status 16962 runtime: frequent crashes with "exit status 16962" on netbsd-arm64-bsiegert builder Oct 5, 2021
@bcmills
Copy link
Member

bcmills commented Dec 9, 2021

@bsiegert, is it possible to make progress on this? These crashes occur multiple times per day (so they seem highly reproducible), and since they seem to occur for all Go programs it's not feasible to just bypass them by adding test skips.

The Go porting policy requires each port to have at least one developer maintaining it. If the netbsd/arm64 builder can't be made to run reliably, we will need to consider removing the port.

greplogs --dashboard -md -l -e 'exit status 16962' --since=2021-10-06

2021-12-08T18:30:00-61011de/netbsd-arm64-bsiegert
2021-12-08T18:01:05-d14b36d-61317ef/netbsd-arm64-bsiegert
2021-12-07T21:54:30-cf1ec17/netbsd-arm64-bsiegert
2021-12-05T18:29:25-97ca703-0f2d0d0/netbsd-arm64-bsiegert
2021-12-05T12:50:44-ecf6b52/netbsd-arm64-bsiegert
2021-12-05T04:19:11-012df41-0f2d0d0/netbsd-arm64-bsiegert
2021-12-04T17:16:07-3396878/netbsd-arm64-bsiegert
2021-12-04T12:00:58-94396e4-0f2d0d0/netbsd-arm64-bsiegert
2021-12-04T11:59:35-1876b38/netbsd-arm64-bsiegert
2021-12-04T01:07:28-fa88ba1/netbsd-arm64-bsiegert
2021-12-04T01:07:21-821bf04/netbsd-arm64-bsiegert
2021-12-04T01:07:10-ba83aa7/netbsd-arm64-bsiegert
2021-12-03T22:57:02-9ae0b35/netbsd-arm64-bsiegert
2021-12-03T21:54:56-f553aa4-0f2d0d0/netbsd-arm64-bsiegert
2021-12-03T21:51:55-c146d37-0f2d0d0/netbsd-arm64-bsiegert
2021-12-03T21:50:33-b8ead20-0f2d0d0/netbsd-arm64-bsiegert
2021-12-03T18:55:11-c882a49-0f2d0d0/netbsd-arm64-bsiegert
2021-12-03T14:35:27-b3d8711-0f2d0d0/netbsd-arm64-bsiegert
2021-12-03T14:28:11-a174638/netbsd-arm64-bsiegert
2021-12-03T04:14:03-4305ba7-0f2d0d0/netbsd-arm64-bsiegert
2021-12-03T01:09:21-0985990/netbsd-arm64-bsiegert
2021-12-03T00:46:20-8da66a3/netbsd-arm64-bsiegert
2021-12-02T23:40:00-9a694f2-0f2d0d0/netbsd-arm64-bsiegert
2021-12-02T23:07:11-ab9c578-0f2d0d0/netbsd-arm64-bsiegert
2021-12-02T23:02:33-8835343/netbsd-arm64-bsiegert
2021-12-02T03:14:37-2ac48c6-0f9838e/netbsd-arm64-bsiegert
2021-11-30T22:42:17-1fd30d2-b7651e5/netbsd-arm64-bsiegert
2021-11-30T18:09:02-931d80e/netbsd-arm64-bsiegert
2021-11-30T15:28:46-682435d/netbsd-arm64-bsiegert
2021-11-29T20:12:13-6e52f51-b7651e5/netbsd-arm64-bsiegert
2021-11-29T16:08:23-a59ab29/netbsd-arm64-bsiegert
2021-11-29T02:12:23-6944b10-f6103e9/netbsd-arm64-bsiegert
2021-11-27T23:27:52-9f2a075/netbsd-arm64-bsiegert
2021-11-27T23:25:39-7e1260f/netbsd-arm64-bsiegert
2021-11-27T19:49:32-a142d65/netbsd-arm64-bsiegert
2021-11-26T22:36:21-1d47a11/netbsd-arm64-bsiegert
2021-11-26T20:49:40-bf88ada/netbsd-arm64-bsiegert
2021-11-26T13:06:40-a0506bd/netbsd-arm64-bsiegert
2021-11-25T04:02:39-45bae64/netbsd-arm64-bsiegert
2021-11-25T00:26:26-e0eb9bf-4eaf0b7/netbsd-arm64-bsiegert
2021-11-25T00:07:28-f7e34e7/netbsd-arm64-bsiegert
2021-11-25T00:07:11-c58243a/netbsd-arm64-bsiegert
2021-11-24T23:47:34-4cbd189-4eaf0b7/netbsd-arm64-bsiegert
2021-11-24T23:04:05-cb80a01-4eaf0b7/netbsd-arm64-bsiegert
2021-11-24T21:15:45-fe61309-4eaf0b7/netbsd-arm64-bsiegert
2021-11-24T21:11:45-b77f5f9/netbsd-arm64-bsiegert
2021-11-24T21:11:34-8cdfe40/netbsd-arm64-bsiegert
2021-11-24T21:09:36-7e5331a/netbsd-arm64-bsiegert
2021-11-24T20:54:12-696515e/netbsd-arm64-bsiegert
2021-11-24T20:54:07-6ea17aa/netbsd-arm64-bsiegert
2021-11-24T20:53:54-a3b8f62/netbsd-arm64-bsiegert
2021-11-24T20:53:45-9394800/netbsd-arm64-bsiegert
2021-11-24T20:51:25-5d8c49a/netbsd-arm64-bsiegert
2021-11-24T19:38:31-d3ed0bb-4eaf0b7/netbsd-arm64-bsiegert
2021-11-24T16:01:55-b38ab0a/netbsd-arm64-bsiegert
2021-11-23T16:39:20-1e71a25-364f15f/netbsd-arm64-bsiegert
2021-11-22T23:09:35-9678f79/netbsd-arm64-bsiegert
2021-11-22T19:49:38-189b4a2/netbsd-arm64-bsiegert
2021-11-22T18:19:28-ed8b8d5-b954f58/netbsd-arm64-bsiegert
2021-11-22T12:30:26-b2aa138/netbsd-arm64-bsiegert
2021-11-22T03:24:01-0c3b4a3/netbsd-arm64-bsiegert
2021-11-20T16:08:51-1765241-3729a67/netbsd-arm64-bsiegert
2021-11-20T08:47:36-91abe4b/netbsd-arm64-bsiegert
2021-11-20T01:00:16-be18cd5/netbsd-arm64-bsiegert
2021-11-20T00:37:28-d2f4c93/netbsd-arm64-bsiegert
2021-11-19T21:59:14-5e774b0/netbsd-arm64-bsiegert
2021-11-19T21:41:33-ba9f0f6/netbsd-arm64-bsiegert
2021-11-19T21:36:17-0f75a9e/netbsd-arm64-bsiegert
2021-11-19T18:23:43-80cb59c/netbsd-arm64-bsiegert
2021-11-19T17:42:26-a944096/netbsd-arm64-bsiegert
2021-11-18T14:08:03-9ea939b/netbsd-arm64-bsiegert
2021-11-18T02:18:12-72f0976/netbsd-arm64-bsiegert
2021-11-18T02:17:34-3404ee3/netbsd-arm64-bsiegert
2021-11-18T02:17:23-ce7e501/netbsd-arm64-bsiegert
2021-11-18T02:16:39-353cb71/netbsd-arm64-bsiegert
2021-11-17T18:14:41-0981724/netbsd-arm64-bsiegert
2021-11-16T17:32:23-40effca/netbsd-arm64-bsiegert
2021-11-15T21:22:19-95e85e3/netbsd-arm64-bsiegert
2021-11-15T15:42:24-ce4a275/netbsd-arm64-bsiegert
2021-11-13T01:37:51-2fd720b/netbsd-arm64-bsiegert
2021-11-13T00:50:04-c893a85/netbsd-arm64-bsiegert
2021-11-11T20:25:49-c622d1d/netbsd-arm64-bsiegert
2021-11-11T11:47:33-c49627e/netbsd-arm64-bsiegert
2021-11-10T02:57:43-e984240/netbsd-arm64-bsiegert
2021-11-09T22:14:26-1c86bee/netbsd-arm64-bsiegert
2021-11-09T22:05:25-a65a095/netbsd-arm64-bsiegert
2021-11-09T01:45:54-fafc446-23991f5/netbsd-arm64-bsiegert
2021-11-08T21:52:47-955f9f5/netbsd-arm64-bsiegert
2021-11-07T04:56:11-85493d5/netbsd-arm64-bsiegert
2021-11-06T12:24:47-70e6271-23991f5/netbsd-arm64-bsiegert
2021-11-05T23:32:57-b8b8e7f-23991f5/netbsd-arm64-bsiegert
2021-11-05T22:54:47-ba79c1e/netbsd-arm64-bsiegert
2021-11-05T22:00:16-755ede0/netbsd-arm64-bsiegert
2021-11-05T19:01:13-93bab8a/netbsd-arm64-bsiegert
2021-11-05T05:29:10-3b5add5/netbsd-arm64-bsiegert
2021-11-04T21:53:05-76c48e9/netbsd-arm64-bsiegert
2021-11-04T16:36:19-f934b83/netbsd-arm64-bsiegert
2021-11-04T07:05:31-2622235/netbsd-arm64-bsiegert
2021-11-03T20:13:57-68536fa-b212ba6/netbsd-arm64-bsiegert
2021-11-03T19:38:32-012c320-88407a8/netbsd-arm64-bsiegert
2021-11-03T16:57:50-4a4e1f2-88407a8/netbsd-arm64-bsiegert
2021-11-01T19:56:50-ee08195-4a84298/netbsd-arm64-bsiegert
2021-11-01T18:53:06-b5a5b7b/netbsd-arm64-bsiegert
2021-11-01T15:20:08-b7d65fb-d19c5bd/netbsd-arm64-bsiegert
2021-11-01T14:45:28-82f902a/netbsd-arm64-bsiegert
2021-10-31T18:39:05-89c5270/netbsd-arm64-bsiegert
2021-10-31T02:24:29-0bef30d/netbsd-arm64-bsiegert
2021-10-30T12:32:06-351c04c-d19c5bd/netbsd-arm64-bsiegert
2021-10-30T00:47:26-a6c6f4b-d19c5bd/netbsd-arm64-bsiegert
2021-10-29T22:54:02-593da8d-d19c5bd/netbsd-arm64-bsiegert
2021-10-29T21:19:39-a2be0cd-d19c5bd/netbsd-arm64-bsiegert
2021-10-29T19:42:43-bb1c798-d19c5bd/netbsd-arm64-bsiegert
2021-10-29T16:52:21-6e78728-f6f024f/netbsd-arm64-bsiegert
2021-10-29T00:04:41-d6a9af8-f6f024f/netbsd-arm64-bsiegert
2021-10-28T21:17:17-e7eb6f6-f6f024f/netbsd-arm64-bsiegert
2021-10-28T20:25:45-6944b10-18b9702/netbsd-arm64-bsiegert
2021-10-27T02:41:01-e1e2965-11b64b4/netbsd-arm64-bsiegert
2021-10-26T22:24:36-591e12a-11b64b4/netbsd-arm64-bsiegert
2021-10-26T17:10:45-76cef81/netbsd-arm64-bsiegert
2021-10-26T15:46:18-c4ead46-11b64b4/netbsd-arm64-bsiegert
2021-10-25T14:51:03-18096c5-2ac3bdf/netbsd-arm64-bsiegert
2021-10-25T02:59:35-f686f6a/netbsd-arm64-bsiegert
2021-10-23T16:18:09-c526cf8/netbsd-arm64-bsiegert
2021-10-23T12:53:38-affba50-2ac3bdf/netbsd-arm64-bsiegert
2021-10-22T02:18:41-6f2254e-2ac3bdf/netbsd-arm64-bsiegert
2021-10-22T00:56:18-23e57e5/netbsd-arm64-bsiegert
2021-10-20T20:39:25-e904928/netbsd-arm64-bsiegert
2021-10-20T20:38:47-3ec8d4b/netbsd-arm64-bsiegert
2021-10-20T17:26:58-4e565f7/netbsd-arm64-bsiegert
2021-10-20T07:21:22-d9421ce/netbsd-arm64-bsiegert
2021-10-19T23:27:41-4ea6123-c580180/netbsd-arm64-bsiegert
2021-10-19T23:27:41-4ea6123-2ac3bdf/netbsd-arm64-bsiegert
2021-10-19T23:23:29-c6ed85c-c580180/netbsd-arm64-bsiegert
2021-10-19T21:17:55-9f721e8-c580180/netbsd-arm64-bsiegert
2021-10-19T21:17:55-9f721e8-2ac3bdf/netbsd-arm64-bsiegert
2021-10-19T21:05:11-1b24c9e/netbsd-arm64-bsiegert
2021-10-19T18:56:08-07e5527/netbsd-arm64-bsiegert
2021-10-18T22:55:32-323e009/netbsd-arm64-bsiegert
2021-10-18T21:33:08-33b3260/netbsd-arm64-bsiegert
2021-10-18T15:38:40-74acbaf/netbsd-arm64-bsiegert
2021-10-16T14:50:21-680caf1/netbsd-arm64-bsiegert
2021-10-15T21:46:06-ed1c8db/netbsd-arm64-bsiegert
2021-10-15T21:44:07-4a7975e/netbsd-arm64-bsiegert
2021-10-15T17:05:16-8c99421/netbsd-arm64-bsiegert
2021-10-15T16:00:01-fad4a16/netbsd-arm64-bsiegert
2021-10-15T14:09:01-98f6e03-2ac3bdf/netbsd-arm64-bsiegert
2021-10-15T02:45:22-288d0b3-2ac3bdf/netbsd-arm64-bsiegert
2021-10-15T00:16:20-ce04ca3-2ac3bdf/netbsd-arm64-bsiegert
2021-10-14T22:15:24-b59467e/netbsd-arm64-bsiegert
2021-10-14T19:24:35-4cbc0a5/netbsd-arm64-bsiegert
2021-10-14T18:43:59-765c911/netbsd-arm64-bsiegert
2021-10-14T17:38:39-011fd00/netbsd-arm64-bsiegert
2021-10-14T13:09:28-24e798e/netbsd-arm64-bsiegert
2021-10-13T16:39:23-3986e5c/netbsd-arm64-bsiegert
2021-10-13T15:11:16-0454d73/netbsd-arm64-bsiegert
2021-10-13T04:56:02-378b9e1-2ac3bdf/netbsd-arm64-bsiegert
2021-10-13T03:12:54-24389d4-c580180/netbsd-arm64-bsiegert
2021-10-12T21:30:42-61890fb/netbsd-arm64-bsiegert
2021-10-11T22:32:23-c1b0ae4/netbsd-arm64-bsiegert
2021-10-11T15:43:15-f2141a7/netbsd-arm64-bsiegert
2021-10-11T12:53:11-34f7b1f/netbsd-arm64-bsiegert
2021-10-10T15:11:26-d2d21d9/netbsd-arm64-bsiegert
2021-10-08T19:48:52-3b03d30-c580180/netbsd-arm64-bsiegert
2021-10-08T17:32:25-a7d3a0e/netbsd-arm64-bsiegert
2021-10-08T16:26:20-99c1b24/netbsd-arm64-bsiegert
2021-10-07T18:58:33-7286502/netbsd-arm64-bsiegert
2021-10-07T00:39:17-812a33d/netbsd-arm64-bsiegert
2021-10-06T23:01:01-4ffa2f1/netbsd-arm64-bsiegert
2021-10-06T19:45:35-e38ec96/netbsd-arm64-bsiegert
2021-10-06T19:08:15-6487b15/netbsd-arm64-bsiegert
2021-10-06T19:08:04-0e13da8/netbsd-arm64-bsiegert
2021-10-06T16:46:09-3160571/netbsd-arm64-bsiegert
2021-10-06T15:53:04-4d8db00/netbsd-arm64-bsiegert

@gopherbot
Copy link

gopherbot commented Jan 6, 2022

Change https://golang.org/cl/375834 mentions this issue: runtime: crash on netbsd-arm64 when setcontext fails

gopherbot pushed a commit that referenced this issue Jan 6, 2022
Instead of exiting with status 16962 when we fail to call SYS_setcontext
in sigreturn, reference a null pointer and crash. Hopefully, this will
enable grabbing a core dump to debug.

Updates #42422

Change-Id: If02c14a0a37084351f3f00db3dc9766cb68ae4b8
Reviewed-on: https://go-review.googlesource.com/c/go/+/375834
Reviewed-by: Tobias Klauser <tobias.klauser@gmail.com>
Trust: Tobias Klauser <tobias.klauser@gmail.com>
Trust: Benny Siegert <bsiegert@gmail.com>
@bsiegert
Copy link
Contributor

bsiegert commented Jan 6, 2022

The thing that's surprising to me is: I actually use Go on netbsd-arm64 productively. My main development laptop is a Pinebook Pro. And I have built Go a number of times, but I have not seen the "exit 16962" crash a single time. Maybe it does have something to do with the builder machine.

@zoulasc
Copy link
Contributor

zoulasc commented Jan 6, 2022

Can you turn off ASLR and see if that fixes the problem?

@bsiegert
Copy link
Contributor

bsiegert commented Jan 7, 2022

I turned off ASLR by setting

security.pax.aslr.enabled = 0
security.pax.aslr.global = 0

I'll keep watching the build results.

FWIW, the Pinebook Pro has been continuously building Go HEAD for netbsd-arm for the last 24 hours, without a single crash.

@zoulasc
Copy link
Contributor

zoulasc commented Jan 14, 2022

So has it failed since?

@zoulasc
Copy link
Contributor

zoulasc commented Jan 14, 2022

I'll keep looking :-)

@bsiegert
Copy link
Contributor

bsiegert commented Feb 4, 2022

The failures are typically early in the bootstrap phase, which means they are in binaries built with Go 1.14.2. I did the following two things:

  • the A64 now uses Go 1.18beta2 as its GOROOT_BOOTSTRAP, which includes the latest change on this issue.
  • I added a second machine, a Raspberry Pi 4 with NetBSD-current from January 15. Amusingly, because its system compiler is gcc 10.3, it uncovered issue misc/cgo/test: failure with gcc 10 #50987 (now fixed).

@aclements
Copy link
Member

aclements commented Feb 15, 2022

I think your builder updates have resolved this:

$ greplogs --dashboard -l -e 'exit status 16962' --since=2021-01-01 | findflakes -paths
First observed 0981724 17 Nov 18:14 2021 (1704 commits ago)
Last observed 1ab8273 03 Feb 18:23 2022 (98 commits ago)
0.29% chance failure is still happening
5.8% failure probability (94 of 1607 commits)

@bcmills
Copy link
Member

bcmills commented Feb 15, 2022

@aclements, are you also fetching subrepos? This failure mode is definitely lower, but I'm still seeing it in 2 out of 3 failures after the configuration change:

greplogs --dashboard -md -l -e \\Anetbsd-arm64 -E . --since=2022-02-05

2022-02-11T22:01:05-badba35/netbsd-arm64-bsiegert (runtime_test.TestShrinkStackDuringBlockedSend deadlock?)
2022-02-10T15:16:21-f4118a5-0a6cf87/netbsd-arm64-bsiegert

/var/gobuilder/buildlet/go/pkg/tool/netbsd_arm64/vet: exit status 16962

2022-02-09T21:30:12-5430f30-0a6cf87/netbsd-arm64-bsiegert

go build google.golang.org/protobuf/types/descriptorpb: /var/gobuilder/buildlet/go/pkg/tool/netbsd_arm64/compile: exit status 16962

@bcmills
Copy link
Member

bcmills commented Feb 15, 2022

Ah, 0a6cf87 is the head of release-branch.go1.16. So probably the root cause was fixed in 1.17.

@bcmills bcmills closed this as completed Feb 15, 2022
@bcmills
Copy link
Member

bcmills commented Feb 24, 2022

Hrm. Here are two on release-branch.go1.17:

greplogs --dashboard -md -l -e 'exit status 16962' --since=2022-02-16

2022-02-23T14:43:16-4f21f7a-7d8fa65/netbsd-arm64-bsiegert
2022-02-18T00:51:21-bbc52f6-7d8fa65/netbsd-arm64-bsiegert

So it seems like the root cause was not fixed in 1.17. 🤔

@bcmills bcmills reopened this Mar 30, 2022
@zoulasc
Copy link
Contributor

zoulasc commented Mar 30, 2022

The error part comes from:
func (c *Cmd) Run() error { if err := c.Start(); err != nil { return err } return c.Wait() }
Calling c.Run() returns %v 0x4242 for "error" which does not appear to be a valid return code from c.Wait(). I think that the process fails to start instead. Can someone instrument the the Run command first and see if my theory is correct, and then instrument c.Start() if that's true and see where it fails?

@bcmills
Copy link
Member

bcmills commented May 5, 2022

@zoulasc
Copy link
Contributor

zoulasc commented May 6, 2022

Sorry, I am too busy to try my theory (from above). Perhaps next month.

@bcmills
Copy link
Member

bcmills commented May 19, 2022

@bcmills
Copy link
Member

bcmills commented May 19, 2022

Ah, no. That large crop is on release-branch.go1.17.

Should we just turn this builder off for that branch?

@bsiegert
Copy link
Contributor

bsiegert commented May 19, 2022

Yes, that seems like a good idea.

jproberts pushed a commit to jproberts/go that referenced this issue Jun 21, 2022
Instead of exiting with status 16962 when we fail to call SYS_setcontext
in sigreturn, reference a null pointer and crash. Hopefully, this will
enable grabbing a core dump to debug.

Updates golang#42422

Change-Id: If02c14a0a37084351f3f00db3dc9766cb68ae4b8
Reviewed-on: https://go-review.googlesource.com/c/go/+/375834
Reviewed-by: Tobias Klauser <tobias.klauser@gmail.com>
Trust: Tobias Klauser <tobias.klauser@gmail.com>
Trust: Benny Siegert <bsiegert@gmail.com>
@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Jul 7, 2022
@bcmills
Copy link
Member

bcmills commented Oct 10, 2022

This is recurring on the main-branch builder now that we're consistently bootstrapping with Go 1.17.13 on most builders.

@bsiegert, should we configure this builder with GoBootstrap: "none" in order to avoid this failure mode? (Does the builder have a bootstrap toolchain installed that is newer than 1.17.13?)

greplogs -l -e 'exit status 16962' --since=2022-05-19 --details

2022-10-07T18:28:11-c7cc2b9/netbsd-arm64-bsiegert
2022-10-07T18:16:10-97760ed/netbsd-arm64-bsiegert
2022-10-07T16:48:38-5ca0cd3/netbsd-arm64-bsiegert
2022-10-06T21:22:22-2837ffe/netbsd-arm64-bsiegert
2022-10-06T14:46:33-9dfadf9/netbsd-arm64-bsiegert
2022-10-01T01:52:16-0cbe304/netbsd-arm64-bsiegert
2022-09-30T20:09:46-4444f85/netbsd-arm64-bsiegert
2022-09-30T11:22:39-6616573/netbsd-arm64-bsiegert
2022-09-30T05:48:38-d7dbe01/netbsd-arm64-bsiegert
2022-09-29T23:49:15-1a6af5f/netbsd-arm64-bsiegert
2022-09-29T23:48:45-5613de9/netbsd-arm64-bsiegert
2022-09-29T22:59:14-b8ed88d/netbsd-arm64-bsiegert
2022-09-29T18:20:11-36f046d/netbsd-arm64-bsiegert
2022-09-29T09:04:59-090ec61/netbsd-arm64-bsiegert
2022-09-29T09:04:58-58093e8/netbsd-arm64-bsiegert
2022-09-28T20:37:59-f1d281f/netbsd-arm64-bsiegert
2022-09-28T11:47:16-7a74829/netbsd-arm64-bsiegert
2022-09-28T10:36:02-dbe56ff/netbsd-arm64-bsiegert
2022-09-28T03:44:27-a59e614/netbsd-arm64-bsiegert
2022-09-27T23:19:02-b2137e7/netbsd-arm64-bsiegert
2022-09-27T21:14:06-3b5188e/netbsd-arm64-bsiegert
2022-09-27T21:14:03-605148c/netbsd-arm64-bsiegert
2022-09-27T20:36:31-e8afb29/netbsd-arm64-bsiegert
2022-09-27T16:50:30-7d157fd/netbsd-arm64-bsiegert
2022-09-27T16:08:16-c929a5b/netbsd-arm64-bsiegert
2022-09-27T16:01:59-d9d2ef9/netbsd-arm64-bsiegert
2022-09-27T15:53:49-789e798/netbsd-arm64-bsiegert
2022-09-26T21:33:33-4c5a6f9/netbsd-arm64-bsiegert
2022-09-26T20:36:02-4da2bc2/netbsd-arm64-bsiegert
2022-09-26T16:42:07-95a5e20/netbsd-arm64-bsiegert
2022-09-22T23:46:28-ddaf682/netbsd-arm64-bsiegert
2022-09-20T15:50:36-0bfa9f0/netbsd-arm64-bsiegert
2022-09-19T21:57:04-f1b7b2f/netbsd-arm64-bsiegert
2022-09-19T18:59:50-fd82718/netbsd-arm64-bsiegert
2022-09-16T00:17:05-805f174/netbsd-arm64-bsiegert
2022-09-15T21:12:43-3a067b2/netbsd-arm64-bsiegert
2022-09-15T21:11:27-dedce99/netbsd-arm64-bsiegert
2022-09-15T21:07:21-f6436c6/netbsd-arm64-bsiegert
2022-09-14T15:19:55-a7e9a37/netbsd-arm64-bsiegert
2022-09-14T14:52:52-d60e51e/netbsd-arm64-bsiegert
2022-09-14T14:30:53-72e07db/netbsd-arm64-bsiegert
2022-09-13T17:39:52-09f3ff1/netbsd-arm64-bsiegert
2022-09-09T18:21:40-9dd9174/netbsd-arm64-bsiegert
2022-09-09T15:34:46-86477e5/netbsd-arm64-bsiegert
2022-09-09T15:30:06-f53b211/netbsd-arm64-bsiegert
2022-09-09T15:30:04-1b19698/netbsd-arm64-bsiegert
2022-09-08T14:33:31-596b0d0/netbsd-arm64-bsiegert
2022-09-07T16:39:44-027ff3f/netbsd-arm64-bsiegert
2022-09-07T13:59:41-88149ed/netbsd-arm64-bsiegert
2022-09-07T06:18:03-e2d8ed2/netbsd-arm64-bsiegert
2022-09-07T05:38:50-b6a6847/netbsd-arm64-bsiegert
2022-09-07T05:37:53-c011270/netbsd-arm64-bsiegert
2022-09-07T01:28:55-1429a97/netbsd-arm64-bsiegert
2022-09-07T01:27:54-db9e549/netbsd-arm64-bsiegert
2022-09-06T23:07:55-29af494/netbsd-arm64-bsiegert
2022-09-06T22:23:43-c83e1b6/netbsd-arm64-bsiegert
2022-09-06T17:36:47-e0e0c8f/netbsd-arm64-bsiegert
2022-09-06T17:05:15-fbf452a/netbsd-arm64-bsiegert
2022-09-05T08:07:47-357b922/netbsd-arm64-bsiegert
2022-09-05T04:17:02-be26aa7/netbsd-arm64-bsiegert
2022-09-02T17:59:27-0cf996a/netbsd-arm64-bsiegert
2022-09-02T02:15:21-157aae6/netbsd-arm64-bsiegert
2022-09-02T02:08:58-042f051/netbsd-arm64-bsiegert
2022-09-01T23:17:30-bbaf0a5/netbsd-arm64-bsiegert
2022-09-01T00:42:27-64b260d/netbsd-arm64-bsiegert
2022-08-30T19:42:56-8a3d167/netbsd-arm64-bsiegert
2022-08-30T00:37:15-47f8db3/netbsd-arm64-bsiegert
2022-08-29T14:52:44-a1c9783/netbsd-arm64-bsiegert
2022-08-29T14:48:48-863d57c/netbsd-arm64-bsiegert
2022-08-26T19:15:02-897ad2f/netbsd-arm64-bsiegert
2022-08-26T18:28:14-83b083e-bf812b3/netbsd-arm64-bsiegert
2022-08-26T17:44:36-297e3de/netbsd-arm64-bsiegert
2022-08-26T14:32:13-d7a3fa1/netbsd-arm64-bsiegert
2022-08-23T18:54:32-ef22285/netbsd-arm64-bsiegert
2022-08-23T18:13:48-aa6a7fa/netbsd-arm64-bsiegert
2022-08-22T18:50:54-e21c1f7/netbsd-arm64-bsiegert
2022-08-22T14:48:53-6bdca82/netbsd-arm64-bsiegert
2022-08-19T18:40:16-b6ab218-2375ef8/netbsd-arm64-bsiegert
2022-08-19T18:39:57-b6ab218-27ced96/netbsd-arm64-bsiegert
2022-08-19T18:39:57-8227340-27ced96/netbsd-arm64-bsiegert
2022-08-19T18:39:57-0c548b8-27ced96/netbsd-arm64-bsiegert
2022-08-18T03:12:27-b15c399/netbsd-arm64-bsiegert
2022-08-17T16:48:18-b11b4b4/netbsd-arm64-bsiegert
2022-08-17T04:10:17-2c46cc8/netbsd-arm64-bsiegert
2022-08-16T17:18:19-805305e/netbsd-arm64-bsiegert
2022-08-16T16:27:35-a45bbea/netbsd-arm64-bsiegert
2022-08-16T01:54:23-5e7697b/netbsd-arm64-bsiegert
2022-08-15T18:58:51-95c125a/netbsd-arm64-bsiegert
2022-08-15T18:57:35-05ff045/netbsd-arm64-bsiegert
2022-08-12T15:47:04-57c1edc/netbsd-arm64-bsiegert
2022-08-12T00:03:39-d88d91e/netbsd-arm64-bsiegert
2022-08-11T19:05:54-62654df/netbsd-arm64-bsiegert
2022-08-10T02:13:53-efe5929/netbsd-arm64-bsiegert
2022-08-09T18:11:37-469a87d/netbsd-arm64-bsiegert
2022-08-09T05:59:43-5639fca/netbsd-arm64-bsiegert
2022-08-05T15:19:31-b12366b/netbsd-arm64-bsiegert
2022-08-05T15:04:35-8f68843/netbsd-arm64-bsiegert
2022-08-02T18:51:38-54a0ab3/netbsd-arm64-bsiegert
2022-08-01T20:48:56-c7f1191-15da892/netbsd-arm64-bsiegert
2022-08-01T18:43:39-3af8cad-15da892/netbsd-arm64-bsiegert
2022-07-29T14:06:18-dd944dc-703c8ab/netbsd-arm64-bsiegert
2022-07-28T20:06:00-8ea5687-d9242f7/netbsd-arm64-bsiegert
2022-07-28T15:35:11-98bfcd1-d9242f7/netbsd-arm64-bsiegert
2022-07-26T14:16:52-8c9f86f-d9242f7/netbsd-arm64-bsiegert
2022-07-25T10:54:40-4151a5a-c25b12f/netbsd-arm64-bsiegert
2022-07-22T17:25:33-c25b12f/netbsd-arm64-bsiegert
2022-07-13T21:08:41-9b6c018-1ed3c12/netbsd-arm64-bsiegert
2022-07-12T16:56:46-6e6f313-cd54600/netbsd-arm64-bsiegert
2022-07-08T19:48:04-d6c099e-ae2dfcc/netbsd-arm64-bsiegert
2022-07-06T20:57:57-84e58bf-ae2dfcc/netbsd-arm64-bsiegert
2022-07-06T20:57:57-36430f4-ae2dfcc/netbsd-arm64-bsiegert
2022-07-06T19:33:10-9ef614f/netbsd-arm64-bsiegert
2022-07-01T14:51:09-f042799-8d2935a/netbsd-arm64-bsiegert
2022-07-01T14:13:36-93bf1fc-8d2935a/netbsd-arm64-bsiegert
2022-06-29T15:46:14-651a8d8/netbsd-arm64-bsiegert
2022-06-29T15:46:14-02e64fa-651a8d8/netbsd-arm64-bsiegert
2022-06-01T15:40:28-6b760fc-26cdea3/netbsd-arm64-bsiegert
2022-05-31T14:14:38-2bb78cf-909881d/netbsd-arm64-bsiegert
2022-05-25T19:23:53-e846f3f/netbsd-arm64-bsiegert

(119 matching logs)

@gopherbot
Copy link

gopherbot commented Oct 13, 2022

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "netbsd" && goarch == "arm64" && `exit status 16962`
2022-10-12 19:16 netbsd-arm64-bsiegert go@4a4de141 (log)
go build cmd/link/internal/loader: /var/gobuilder/buildlet/go/pkg/tool/netbsd_arm64/compile: exit status 16962
go tool dist: FAILED: /var/gobuilder/buildlet/go/pkg/tool/netbsd_arm64/go_bootstrap install cmd/asm cmd/cgo cmd/compile cmd/link: exit status 1
2022-10-12 23:00 netbsd-arm64-bsiegert go@19095e10 (log)
go build cmd/link/internal/ld: /var/gobuilder/buildlet/go/pkg/tool/netbsd_arm64/compile: exit status 16962
go tool dist: FAILED: /var/gobuilder/buildlet/go/pkg/tool/netbsd_arm64/go_bootstrap install cmd/asm cmd/cgo cmd/compile cmd/link: exit status 1

watchflakes

@bsiegert
Copy link
Contributor

bsiegert commented Oct 13, 2022

@rsc
Copy link
Contributor

rsc commented Oct 13, 2022

If the problem is Go 1.17.13, then we can have the coordinator install something newer.

@bsiegert
Copy link
Contributor

bsiegert commented Oct 13, 2022

@gopherbot
Copy link

gopherbot commented Oct 13, 2022

Change https://go.dev/cl/442855 mentions this issue: dashboard: set default GoBootstrap version to 1.18.7

@gopherbot
Copy link

gopherbot commented Oct 13, 2022

Change https://go.dev/cl/442787 mentions this issue: dashboard: update netbsd, openbsd builders to Go 1.19.2

gopherbot pushed a commit to golang/build that referenced this issue Oct 14, 2022
The NetBSD builders are failing in a way that only manifests with Go 1.17.
The OpenBSD builders are also very flaky.
Update both operating systems to Go 1.19.2 in hopes of eliminating those flakes.

For golang/go#42422.

Change-Id: I10a736c12b38924f6fed5269c2cde21c16de4887
Reviewed-on: https://go-review.googlesource.com/c/build/+/442787
Run-TryBot: Russ Cox <rsc@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
@gopherbot
Copy link

gopherbot commented Oct 17, 2022

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "netbsd" && goarch == "arm64" && `exit status 16962`
2022-10-12 20:23 netbsd-arm64-bsiegert go@987f94fa (log)
go build cmd/compile/internal/ssa: /var/gobuilder/buildlet/go/pkg/tool/netbsd_arm64/compile: exit status 16962
go tool dist: FAILED: /var/gobuilder/buildlet/go/pkg/tool/netbsd_arm64/go_bootstrap install cmd/asm cmd/cgo cmd/compile cmd/link: exit status 1
2022-10-13 15:31 netbsd-arm64-bsiegert go@4a0ce469 (log)
go build math/big: /var/gobuilder/buildlet/go/pkg/tool/netbsd_arm64/compile: exit status 16962
go tool dist: FAILED: /var/gobuilder/buildlet/go/pkg/tool/netbsd_arm64/go_bootstrap install cmd/asm cmd/cgo cmd/compile cmd/link: exit status 1
2022-10-13 20:22 netbsd-arm64-bsiegert go@005c78d8 (log)
go build cmd/compile/internal/typecheck: /var/gobuilder/buildlet/go/pkg/tool/netbsd_arm64/compile: exit status 16962
go tool dist: FAILED: /var/gobuilder/buildlet/go/pkg/tool/netbsd_arm64/go_bootstrap install cmd/asm cmd/cgo cmd/compile cmd/link: exit status 1
2022-10-13 21:14 netbsd-arm64-bsiegert go@c79b2009 (log)
go build cmd/internal/obj/x86: /var/gobuilder/buildlet/go/pkg/tool/netbsd_arm64/compile: exit status 16962
go build cmd/cgo: /var/gobuilder/buildlet/go/pkg/tool/netbsd_arm64/compile: exit status 16962
go tool dist: FAILED: /var/gobuilder/buildlet/go/pkg/tool/netbsd_arm64/go_bootstrap install cmd/asm cmd/cgo cmd/compile cmd/link: exit status 1
2022-10-14 12:48 netbsd-arm64-bsiegert go@a4b4717f (log)
go build runtime/pprof: /var/gobuilder/buildlet/go1.4/pkg/tool/netbsd_arm64/compile: exit status 16962
go tool dist: FAILED: /var/gobuilder/buildlet/go1.4/bin/go install -gcflags=-l -tags=math_big_pure_go compiler_bootstrap purego bootstrap/cmd/...: exit status 1

build failed: make script failed: exit status 2

watchflakes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
arch-arm64 compiler/runtime Issues related to the Go compiler and/or runtime. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-NetBSD
Projects
Status: Triage Backlog
Status: No status
Development

No branches or pull requests

8 participants