-
Notifications
You must be signed in to change notification settings - Fork 17.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
all: plan9-arm builder failing frequently with a variety of errors #49338
Comments
For most of plan9-arm failures, the cause is filesystem flakiness on the builder machines (a cluster of 3 x raspberry pi 4). I've just reconfigured the build script to stop using ramfs (which has a locking bug) for /tmp, and stop using the ad hoc directory cache (which still seems to have some flaws). This should bypass most of the failures but will make the test suite a lot slower to run (by go.17 it had already regressed to about 2 hours even with these filesystem tweaks). There are some other flakes of the "shouldn't be possible" variety, which are are rare and perplexing and impossible to replicate. They are occurring in general go runtime code, not specific to plan 9. There's a slight whiff of possible memory cache coherence trouble in some of these: It would be good to have a server-class ARM platform capable of running plan 9 as a more reliable builder. I don't know of one. |
Change https://golang.org/cl/362975 mentions this issue: |
The plan9 builders have relatively slow filesystems, and the website repo is relatively filesystem-intensive — many of its tests walk and process large trees of website content. Instead of trying to precisely tune the x/website tests to pass on the plan9 builders, let's skip this repo on them. For golang/go#49338. Change-Id: Ieaf50fb6955a6b39012a23ff72e61c1c31e4ce7c Reviewed-on: https://go-review.googlesource.com/c/build/+/362975 Trust: Bryan C. Mills <bcmills@google.com> Run-TryBot: Bryan C. Mills <bcmills@google.com> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Carlos Amedee <carlos@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Many of the failure modes seem to have cleared up around 11 Nov.; it's not clear to me what fixed them. One of the failures visible on the dashboard today is #49653; the other is the |
Change https://golang.org/cl/369018 mentions this issue: |
To prevent interleaving of output when 'go build' compiles several packages in parallel, the output mutex in the Builder struct must be locked around any calls to Builder.Print which could generate arbitrary amounts of text (ie more than is guaranteed to be written atomically to a pipe). Fixes #49987 For #49338 Change-Id: I7947df57667deeff3f03f231824298d823f8a943 Reviewed-on: https://go-review.googlesource.com/c/go/+/369018 Reviewed-by: Bryan Mills <bcmills@google.com> Run-TryBot: Bryan Mills <bcmills@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Trust: Russ Cox <rsc@golang.org>
Change https://golang.org/cl/380414 mentions this issue: |
This test has timed out several times recently on this builder. The test is pretty slow even on Linux, and may be especially filesystem-intensive. For golang/go#49338 Change-Id: Ife2bb399b10f369f815055bb0ad44bb007f606b7 Reviewed-on: https://go-review.googlesource.com/c/text/+/380414 Trust: Bryan Mills <bcmills@google.com> Run-TryBot: Bryan Mills <bcmills@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>
A sampling from the past week or so. Undiagnosed failures in bold.
2022-01-27T05:30:27-be5769c-a991d9d/plan9-arm
That's stuck running 2022-01-27T05:30:27-bbe1937-a991d9d/plan9-arm (#50857)
2022-01-24T16:42:11-2cc1836-19d819d/plan9-arm (#50775, mitigated)
That's running a 2022-01-21T21:27:57-e7c9de2-9eba5ff/plan9-arm (#46520, not specific to |
Change https://go.dev/cl/408697 mentions this issue: |
For golang/go#52653. Updates golang/go#49338. Change-Id: Ib23be56bd79bce8b3168cb40064916a2a91dbb88 Reviewed-on: https://go-review.googlesource.com/c/build/+/408697 Auto-Submit: Bryan Mills <bcmills@google.com> Reviewed-by: Alex Rakoczy <alex@golang.org> Run-TryBot: Bryan Mills <bcmills@google.com> TryBot-Result: Gopher Robot <gobot@golang.org>
This test has timed out several times recently on this builder. The test is pretty slow even on Linux, and may be especially filesystem-intensive. For golang/go#49338 Change-Id: Ife2bb399b10f369f815055bb0ad44bb007f606b7 Reviewed-on: https://go-review.googlesource.com/c/text/+/380414 Trust: Bryan Mills <bcmills@google.com> Run-TryBot: Bryan Mills <bcmills@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>
This comment was marked as resolved.
This comment was marked as resolved.
Found new dashboard test flakes for:
2022-12-05 22:01 plan9-arm go@185e1a7b os/exec.TestExtraFilesRace (log)
|
Found new dashboard test flakes for:
2022-12-06 20:08 plan9-arm net@1e63c2f0 go@a4a86c7b x/net/http2.TestTransportUsesGoAwayDebugError_Body (log)
|
Found new dashboard test flakes for:
2022-12-22 15:34 plan9-arm benchmarks@03c43f28 go@f721fa3b (log)
|
Found new dashboard test flakes for:
2022-12-01 20:58 plan9-arm go@cc1771e2 net/http.TestOmitHTTP2Vet (log)
2022-12-12 22:32 plan9-arm crypto@23edec0b go@357ea858 (log)
2022-12-12 23:10 plan9-arm go@b16e94d1 (log)
2022-12-13 21:00 plan9-arm benchmarks@555ee2b5 go@357ea858 (log)
2022-12-15 00:54 plan9-arm tools@5899b6a2 go@ea14d1b6 (log)2022-12-23 17:16 plan9-arm go@38cfb3be net.TestPipe (log)
2022-12-29 17:47 plan9-arm go@642fd5f7 net/http.TestServerReadTimeout (log)
|
Found new dashboard test flakes for:
2023-01-20 21:10 plan9-arm benchmarks@33cb3306 go@27b6ace2 (log)
|
Found new dashboard test flakes for:
2023-01-05 16:42 plan9-arm net@f8411da7 go@cc82867f x/net/http2/h2c.TestPropagation (log)
2023-01-19 20:43 plan9-arm go@f89badf6 (log)
2023-01-19 20:46 plan9-arm go@e08642ca (log)2023-01-20 02:42 plan9-arm tools@a7f033af go@c641d48a x/tools/gopls/internal/lsp/filecache.TestConcurrency (log)
2023-01-20 17:52 plan9-arm go@1011ccad os.TestDirFS (log)
2023-01-20 19:38 plan9-arm go@f07910bd os.TestDirFS (log)
2023-01-20 22:42 plan9-arm go@0518e33f os.TestWriteAtInAppendMode (log)
2023-01-20 22:42 plan9-arm go@0518e33f os.TestOpenFileLimit (log)
2023-01-20 22:42 plan9-arm go@0518e33f os.TestDirFSRootDir (log)
2023-01-20 22:42 plan9-arm go@0518e33f os.TestReaddirSmallSeek (log)
2023-01-20 22:42 plan9-arm go@0518e33f os.TestDirFS (log)
2023-01-20 22:42 plan9-arm go@0518e33f os.TestReadDir (log)
2023-01-20 22:42 plan9-arm go@0518e33f os.TestReadFile (log)
2023-01-20 22:42 plan9-arm go@0518e33f os.TestFileReaddir (log)
2023-01-20 22:42 plan9-arm go@0518e33f os.TestFileReaddirnames (log)
2023-01-20 22:42 plan9-arm go@0518e33f os.TestFileReadDir (log)
2023-01-23 18:50 plan9-arm go@b90971b2 os.TestDirFS (log)
|
Found new dashboard test flakes for:
2023-01-24 19:29 plan9-arm go@083d94f6 (log)
|
Found new dashboard test flakes for:
2023-01-20 09:27 plan9-arm go@1bef0707 os.TestDirFS (log)
2023-01-25 16:38 plan9-arm go@1d3088ef os.TestFileReadDir (log)
2023-01-25 18:50 plan9-arm go@3c7884a2 (log)2023-01-26 01:06 plan9-arm go@89de906a os.TestDirFS (log)
2023-01-26 19:45 plan9-arm go@627f1286 os.TestFileReadDir (log)
2023-01-26 21:33 plan9-arm net@296f09aa go@dd44d981 x/net/http2/h2c.TestPropagation (log)
2023-01-26 21:54 plan9-arm go@ea6d4b0e os.TestWriteAtInAppendMode (log)
2023-01-26 21:54 plan9-arm go@ea6d4b0e os.TestOpenFileLimit (log)
2023-01-26 21:54 plan9-arm go@ea6d4b0e os.TestReadDir (log)
2023-01-26 21:54 plan9-arm go@ea6d4b0e os.TestReadFile (log)
2023-01-26 21:54 plan9-arm go@ea6d4b0e os.TestDirFSRootDir (log)
2023-01-26 21:54 plan9-arm go@ea6d4b0e os.TestDirFS (log)
2023-01-26 21:54 plan9-arm go@ea6d4b0e os.TestReaddirSmallSeek (log)
2023-01-26 21:54 plan9-arm go@ea6d4b0e os.TestFileReadDir (log)
2023-01-26 21:54 plan9-arm go@ea6d4b0e os.TestFileReaddir (log)
2023-01-26 21:54 plan9-arm go@ea6d4b0e os.TestFileReaddirnames (log)
2023-01-26 21:54 plan9-arm go@ea6d4b0e os/exec (log)
2023-01-27 03:32 plan9-arm go@4e3abee2 os.TestDirFS (log)
2023-01-30 16:32 plan9-arm go@87860873 net/http.TestServerReadTimeout (log)
2023-01-30 20:09 plan9-arm go@45bee5e7 os.TestReadFile (log)
2023-01-31 19:43 plan9-arm go@b89a840d (log)
|
Found new dashboard test flakes for:
2023-02-02 23:42 plan9-arm tools@81111180 go@a9648761 x/tools/cmd/bundle.TestBundle (log)
|
Found new dashboard test flakes for:
2025-01-31 17:38 go1.22-plan9-arm release-branch.go1.22@0cc45e7c sync/atomic.TestValueCompareAndSwapConcurrent [ABORT] (log)
|
Found new dashboard test flakes for:
2025-01-16 20:07 go1.23-plan9-arm release-branch.go1.23@d04e3cbc cmd/link/internal/ld.TestMemProfileCheck/with_memprofile_http_pprof (log)
2025-01-16 20:07 go1.23-plan9-arm release-branch.go1.23@d04e3cbc runtime/internal/math (log)
2025-01-17 15:41 go1.23-plan9-arm release-branch.go1.23@9cbbf5e0 context.ExampleAfterFunc_connection [ABORT] (log)
2025-01-17 15:41 go1.23-plan9-arm release-branch.go1.23@9cbbf5e0 runtime.TestChanSendBarrier [ABORT] (log)
2025-01-17 16:30 go1.23-plan9-arm release-branch.go1.23@ab44565b context.ExampleAfterFunc_connection [ABORT] (log)
2025-01-17 16:30 go1.23-plan9-arm release-branch.go1.23@ab44565b runtime.TestFakeTime [ABORT] (log)
|
Found new dashboard test flakes for:
2025-01-31 16:04 x_tools-gotip-plan9-arm tools@a721d4ca go@ce7ea0a6 x/tools/cmd/deadcode.Test/testdata/jsonflag.txtar/L3 (log)
2025-01-31 16:04 x_tools-gotip-plan9-arm tools@a721d4ca go@ce7ea0a6 x/tools/internal/refactor/inline.TestData/embed.txtar (log)
|
Found new dashboard test flakes for:
2025-01-31 19:16 x_tools-go1.24-plan9-arm tools@5ffcf75f release-branch.go1.24@4241f582 x/tools/go/ssa.TestInit (log)
|
Found new dashboard test flakes for:
2025-01-30 21:39 go1.24-plan9-arm release-branch.go1.24@ce7ea0a6 cmd/objdump.TestDisasmCode (log)
2025-01-30 21:39 go1.24-plan9-arm release-branch.go1.24@ce7ea0a6 cmd/vet.TestVet/unsafeptr (log)
2025-01-30 21:39 go1.24-plan9-arm release-branch.go1.24@ce7ea0a6 compress/zlib (log)
|
Found new dashboard test flakes for:
2025-01-31 18:49 x_tools-go1.24-plan9-arm tools@dace8c8b release-branch.go1.24@4241f582 x/tools/gopls/internal/cmd.TestCodeLens [ABORT] (log)
2025-01-31 18:49 x_tools-go1.24-plan9-arm tools@dace8c8b release-branch.go1.24@4241f582 x/tools/internal/refactor/inline.TestData/crosspkg.txtar [ABORT] (log)
|
Found new dashboard test flakes for:
2025-01-31 18:32 x_tools-gotip-plan9-arm tools@b7813756 go@37f27fbe x/tools/gopls/internal/cmd.TestCodeLens (log)
2025-01-31 18:32 x_tools-gotip-plan9-arm tools@b7813756 go@37f27fbe x/tools/gopls/internal/cmd.TestImplementations (log)
|
Found new dashboard test flakes for:
2025-01-31 18:32 x_tools-go1.23-plan9-arm tools@b7813756 release-branch.go1.23@6644ed63 x/tools/go/ssa/ssautil.TestCreateProgram (log)
2025-01-31 18:32 x_tools-go1.23-plan9-arm tools@b7813756 release-branch.go1.23@6644ed63 x/tools/gopls/internal/cmd.TestCodeLens [ABORT] (log)
|
Found new dashboard test flakes for:
2025-01-29 14:30 go1.24-plan9-arm release-branch.go1.24@4f48ad5c cmd/compile/internal/test.TestIntendedInlining (log)
|
Found new dashboard test flakes for:
2025-01-31 18:19 x_tools-go1.24-plan9-arm tools@bbe00fb5 release-branch.go1.24@4241f582 x/tools/gopls/internal/debug.TestTemplates [ABORT] (log)
2025-01-31 18:19 x_tools-go1.24-plan9-arm tools@bbe00fb5 release-branch.go1.24@4241f582 x/tools/internal/refactor/inline.TestData/crosspkg.txtar (log)
|
Found new dashboard test flakes for:
2025-01-29 14:29 go1.24-plan9-arm release-branch.go1.24@1f58ad5d cmd/api.TestIssue64958 [ABORT] (log)
|
Found new dashboard test flakes for:
2025-01-31 17:09 x_tools-gotip-plan9-arm tools@058d583f go@37f27fbe x/tools/gopls/internal/cmd.TestHighlight (log)
|
Found new dashboard test flakes for:
2025-01-29 02:20 go1.24-plan9-arm release-branch.go1.24@90ec9996 cmd/compile/internal/test.TestPGOHash (log)
2025-01-29 02:20 go1.24-plan9-arm release-branch.go1.24@90ec9996 cmd/compile/internal/test.TestScanfRemoval (log)
2025-01-29 02:20 go1.24-plan9-arm release-branch.go1.24@90ec9996 cmd/link.TestCheckLinkname/ok.go (log)
|
Found new dashboard test flakes for:
2025-01-28 18:54 go1.24-plan9-arm release-branch.go1.24@78e6f2a1 cmd/link.TestStrictDup (log)
2025-01-28 18:54 go1.24-plan9-arm release-branch.go1.24@78e6f2a1 go/types.TestIssue13898 (log)
|
Found new dashboard test flakes for:
2025-01-28 18:44 go1.24-plan9-arm release-branch.go1.24@4ebd5bf8 go/types.TestStdTest [ABORT] (log)
2025-01-28 18:44 go1.24-plan9-arm release-branch.go1.24@4ebd5bf8 internal/trace.TestMMU [ABORT] (log)
2025-01-28 18:44 go1.24-plan9-arm release-branch.go1.24@4ebd5bf8 net/http.TestOmitHTTP2Vet [ABORT] (log)
|
Found new dashboard test flakes for:
2025-01-27 18:46 go1.24-plan9-arm release-branch.go1.24@f70aa382 cmd/addr2line.TestAddr2Line (log)
2025-01-27 18:46 go1.24-plan9-arm release-branch.go1.24@f70aa382 cmd/compile/internal/importer.TestImportedTypes (log)
2025-01-27 18:46 go1.24-plan9-arm release-branch.go1.24@f70aa382 cmd/compile/internal/test.TestPGOIntendedInliningShiftedLines (log)
|
Found new dashboard test flakes for:
2025-01-27 17:58 go1.24-plan9-arm release-branch.go1.24@475e0834 cmd/link/internal/ld.TestRISCVTrampolines (log)
2025-01-27 17:58 go1.24-plan9-arm release-branch.go1.24@475e0834 context.ExampleAfterFunc_connection [ABORT] (log)
|
Found new dashboard test flakes for:
2025-01-27 17:30 go1.24-plan9-arm release-branch.go1.24@e2e700f8 cmd/link/internal/ld.TestMemProfileCheck/with_memprofile_runtime_pprof_writeheap (log)
2025-01-27 17:30 go1.24-plan9-arm release-branch.go1.24@e2e700f8 time (log)
|
Found new dashboard test flakes for:
2025-01-31 16:42 gotip-plan9-arm go@77d20838 cmd/link.TestStrictDup (log)
2025-01-31 16:42 gotip-plan9-arm go@77d20838 compress/gzip (log)
2025-01-31 16:42 gotip-plan9-arm go@77d20838 compress/lzw (log)
2025-01-31 16:50 gotip-plan9-arm go@37f27fbe cmd/cover.TestCoverWithToolExec/HtmlUnformatted (log)
2025-01-31 16:50 gotip-plan9-arm go@37f27fbe compress/lzw (log)
|
Found new dashboard test flakes for:
2025-01-31 19:16 x_tools-go1.23-plan9-arm tools@5ffcf75f release-branch.go1.23@6644ed63 x/tools/go/ssa.TestObjValueLookup (log)
2025-01-31 19:16 x_tools-go1.23-plan9-arm tools@51f179ca release-branch.go1.23@6644ed63 x/tools/go/analysis/passes/pkgfact.Test [ABORT] (log)
2025-01-31 21:26 x_tools-go1.23-plan9-arm tools@b3bde130 release-branch.go1.23@6644ed63 x/tools/gopls/internal/lsprpc.TestClientLogging (log)
2025-02-03 16:14 x_tools-go1.23-plan9-arm tools@da3a6b29 release-branch.go1.23@6644ed63 x/tools/gopls/internal/cmd.TestCallHierarchy (log)
2025-02-03 16:14 x_tools-go1.23-plan9-arm tools@da3a6b29 release-branch.go1.23@6644ed63 x/tools/gopls/internal/cmd.TestCodeLens (log)
|
Found new dashboard test flakes for:
2025-02-03 16:14 x_tools-go1.24-plan9-arm tools@da3a6b29 release-branch.go1.24@4241f582 x/tools/cmd/deadcode.Test/testdata/whylive.txtar/L19 [ABORT] (log)
2025-02-03 18:11 x_tools-go1.23-plan9-arm tools@0abda08b release-branch.go1.23@6644ed63 x/tools/gopls/internal/cmd.TestCodeLens (log)
2025-02-03 18:11 x_tools-go1.23-plan9-arm tools@0abda08b release-branch.go1.23@6644ed63 x/tools/gopls/internal/cmd.TestWorkspaceSymbol (log)
|
Found new dashboard test flakes for:
2025-02-03 16:44 x_tools-go1.24-plan9-arm tools@269282df release-branch.go1.24@4241f582 x/tools/go/analysis/passes/asmdecl.Test/linux/riscv64 [ABORT] (log)
|
Found new dashboard test flakes for:
2025-02-03 20:11 gotip-plan9-arm go@4160ebf2 cmd/link.TestIssue28429 (log)
2025-02-03 20:11 gotip-plan9-arm go@4160ebf2 cmd/link/internal/ld.TestMemProfileCheck/with_memprofile_runtime_pprof_writeheap (log)
|
Found new dashboard test flakes for:
2025-02-03 18:11 x_tools-gotip-plan9-arm tools@0abda08b go@530c8292 x/tools/go/analysis/passes/shadow.Test (log)
|
Found new dashboard test flakes for:
2025-02-03 20:11 gotip-plan9-arm go@dcebb675 cmd/link/internal/ld.TestMemProfileCheck/with_memprofile_http_pprof (log)
|
Found new dashboard test flakes for:
2025-02-04 16:39 go1.22-plan9-arm release-branch.go1.22@5817e650 runtime.TestBigGOMAXPROCS [ABORT] (log)
|
Found new dashboard test flakes for:
2025-02-04 14:41 x_tools-go1.23-plan9-arm tools@e9f7be9f release-branch.go1.23@a991f9c3 x/tools/gopls/internal/cmd.TestCodeAction (log)
|
Found new dashboard test flakes for:
2025-02-04 18:25 x_tools-go1.24-plan9-arm tools@70a7d863 release-branch.go1.24@4241f582 x/tools/cmd/stringer.TestMultifileStringer/test_package_only (log)
|
Found new dashboard test flakes for:
2025-02-03 22:04 gotip-plan9-arm go@ad7b46ee compress/gzip (log)
2025-02-03 22:04 gotip-plan9-arm go@ad7b46ee runtime.TestFakeTime (log)
2025-02-03 23:44 gotip-plan9-arm go@b07b20fb cmd/compile/internal/test.TestIntendedInlining (log)
2025-02-03 23:44 gotip-plan9-arm go@b07b20fb cmd/link/internal/ld.TestDeadcode/ifacemethod2 (log)
2025-02-04 21:10 gotip-plan9-arm go@5c2b5e02 go/ast (log)
|
Found new dashboard test flakes for:
2025-02-04 17:54 x_tools-go1.23-plan9-arm tools@0556adba release-branch.go1.23@a991f9c3 x/tools/go/ssa.TestEnclosingFunction/Multiple_explicit_init_functions [ABORT] (log)
2025-02-04 17:54 x_tools-go1.23-plan9-arm tools@0556adba release-branch.go1.23@a991f9c3 x/tools/refactor/eg.Test/testdata/h.txtar [ABORT] (log)
2025-02-04 17:54 x_tools-go1.24-plan9-arm tools@0556adba release-branch.go1.24@4241f582 x/tools/go/analysis/analysistest.TestTheTest [ABORT] (log)
|
The one remaining
plan9
builder is failing a significant fraction of build attempts.Many of the failures follow specific known patterns (notably #49337, #46526, and #41952). However, many do not.
@millerresearch, @0intro, @fhs: is there someone who can investigate these failures and bring the
plan9
port back up to par? With the0intro
builders missing (#49327, #49328), there is no longer anyplan9
builder consistently passing tests.greplogs --dashboard -md -l -e \\Aplan9. -E . --since=2021-10-01
2021-11-04T01:44:54-d9bb5f6/plan9-arm
2021-11-03T23:48:28-283373d/plan9-arm
2021-11-03T21:45:33-4a4e1f2-d3f5dd5/plan9-arm
2021-11-03T21:45:33-2c98350-d3f5dd5/plan9-arm
2021-11-03T19:32:33-74f99d0/plan9-arm
2021-11-03T18:37:22-cfd016d/plan9-arm
2021-11-03T15:59:11-5f0e5d2/plan9-arm
2021-11-03T09:23:34-6458b2e/plan9-arm
2021-11-03T05:00:00-519c0a2/plan9-arm
2021-11-03T00:49:50-a4b2c57/plan9-arm
2021-11-03T00:21:24-8f0ca7d/plan9-arm
2021-11-03T00:07:38-49a0063/plan9-arm
2021-11-03T00:06:52-32d2752/plan9-arm
2021-11-02T23:44:58-a0f373c/plan9-arm
2021-11-02T23:05:17-714668c-eb63cbd/plan9-arm
2021-11-02T21:18:39-60fd3ed/plan9-arm
2021-11-02T20:59:34-79024cf/plan9-arm
2021-11-02T19:56:02-c45c32b/plan9-arm
2021-11-02T19:37:42-1011e26/plan9-arm
2021-11-02T18:35:29-433ba58/plan9-arm
2021-11-02T18:35:22-0b37bd9/plan9-arm
2021-11-02T18:31:53-79c5240/plan9-arm
2021-11-02T18:24:18-f7a95d2/plan9-arm
2021-11-02T15:33:01-4d06874-da7173a/plan9-arm
2021-11-02T13:43:24-ee08195-4d7bf41/plan9-arm
2021-11-02T13:43:24-4ff95d6-4d7bf41/plan9-arm
2021-11-02T06:25:39-4ff95d6-088bb4b/plan9-arm
2021-11-02T06:21:44-3ee426a/plan9-arm
2021-11-02T06:20:21-c1ea339/plan9-arm
2021-11-02T06:19:33-4633b2d/plan9-arm
2021-11-02T05:35:11-6f327f7/plan9-arm
2021-11-02T03:55:19-4ff95d6-6f1e9a9/plan9-arm
2021-11-02T03:54:24-3c61cb3/plan9-arm
2021-11-02T03:53:56-b0472aa/plan9-arm
2021-11-02T03:52:33-9268565/plan9-arm
2021-11-01T22:55:50-02e5913/plan9-arm
2021-11-01T19:42:30-4ff95d6-5f1bcd1/plan9-arm
2021-11-01T18:53:06-b5a5b7b/plan9-arm
2021-11-01T16:31:02-03971e3-2bcf1c0/plan9-arm
2021-11-01T15:55:25-e2e910e/plan9-arm
2021-11-01T15:55:20-d2b5121/plan9-arm
2021-11-01T14:45:28-82f902a/plan9-arm
2021-11-01T13:12:37-4056934/plan9-arm
2021-11-01T13:08:16-732db40/plan9-arm
2021-11-01T02:47:30-fde4cc2/plan9-arm
2021-10-31T18:39:05-89c5270/plan9-arm
2021-10-31T18:13:09-fd09e88/plan9-arm
2021-10-31T17:52:41-3fa9ee4/plan9-arm
2021-10-30T18:30:34-6113dac/plan9-arm
2021-10-30T16:46:47-d1dceaf/plan9-arm
2021-10-30T12:32:06-351c04c-5d6d9f5/plan9-arm
2021-10-29T23:25:18-5d6d9f5/plan9-arm
2021-10-29T22:29:31-c812b97/plan9-arm
2021-10-29T22:27:26-3571ab5/plan9-arm
2021-10-29T19:35:47-5ec139f/plan9-arm
2021-10-29T18:56:29-6afdf01/plan9-arm
2021-10-29T18:35:29-9da6415/plan9-arm
2021-10-29T18:23:03-413672f/plan9-arm
2021-10-29T17:44:52-353d5b6/plan9-arm
2021-10-29T04:21:07-103d89b-f288526/plan9-arm
2021-10-29T02:16:47-2c7cdec/plan9-arm
2021-10-29T02:11:42-33c392f/plan9-arm
2021-10-29T00:04:41-d6a9af8-af05d8b/plan9-arm
2021-10-28T20:50:34-d0751b1/plan9-arm
2021-10-28T20:50:34-103d89b-d0751b1/plan9-arm
2021-10-28T20:43:39-b954024/plan9-arm
2021-10-28T20:43:02-e741e2f/plan9-arm
2021-10-28T18:48:25-e7eb6f6-f229e70/plan9-arm
2021-10-28T18:10:59-ed475fe/plan9-arm
2021-10-28T18:10:50-2bc8ed8/plan9-arm
2021-10-28T17:57:57-2666262/plan9-arm
2021-10-28T17:52:59-3124968/plan9-arm
2021-10-28T02:35:22-056dfe6/plan9-arm
2021-10-28T01:15:26-b2fe2eb/plan9-arm
2021-10-28T00:11:24-a91d0b6/plan9-arm
2021-10-27T22:17:35-79ff663/plan9-arm
2021-10-27T21:37:54-749f6e9/plan9-arm
2021-10-27T21:35:48-a3c4ac0/plan9-arm
2021-10-27T21:34:20-4fefd43/plan9-arm
2021-10-27T20:25:06-bbc0595/plan9-arm
2021-10-27T20:24:34-56dcf97/plan9-arm
2021-10-27T20:03:17-68bd512/plan9-arm
2021-10-27T19:34:49-ffd2284/plan9-arm
2021-10-27T19:18:20-514ebae/plan9-arm
2021-10-27T18:24:50-259735f/plan9-amd64-0intro
2021-10-27T18:24:50-259735f/plan9-arm
2021-10-27T16:59:43-c0ac39c/plan9-arm
2021-10-27T13:12:49-cfb5321/plan9-arm
2021-10-27T08:50:27-e1e2965-bdefb77/plan9-arm
2021-10-27T08:50:27-bdefb77/plan9-arm
2021-10-26T23:12:17-13eccaa/plan9-arm
2021-10-26T22:05:53-80be4a4/plan9-386-0intro
2021-10-26T19:04:53-9626607-1dc77a3/plan9-arm
2021-10-26T19:04:53-03971e3-1dc77a3/plan9-arm
2021-10-26T18:33:39-244f92e-76cef81/plan9-386-0intro
2021-10-26T17:10:45-d418f37-76cef81/plan9-386-0intro
2021-10-26T17:10:45-76cef81/plan9-386-0intro
2021-10-26T17:10:45-76cef81/plan9-amd64-0intro
2021-10-26T16:00:26-94870a3-1b2362b/plan9-arm
2021-10-26T14:05:47-a2b8c18/plan9-arm
2021-10-26T11:58:05-1e2820a/plan9-arm
2021-10-26T05:05:24-23fdd7f/plan9-arm
2021-10-26T01:22:47-f916b54-adfb85b/plan9-arm
2021-10-26T01:22:47-22652b4-adfb85b/plan9-arm
2021-10-26T01:07:41-41f7407/plan9-arm
2021-10-25T23:28:03-f916b54-e9eb66d/plan9-arm
2021-10-25T23:28:03-e9eb66d/plan9-arm
2021-10-25T21:03:33-60c3069/plan9-arm
2021-10-25T18:46:45-f47335e/plan9-arm
2021-10-25T17:02:16-52b10ab/plan9-arm
2021-10-25T16:07:11-fc8b4ca-c6e82e5/plan9-arm
2021-10-23T20:23:35-c96bc14-8dbf3e9/plan9-386-0intro
2021-10-23T20:23:35-8dbf3e9/plan9-arm
2021-10-23T16:18:19-d418f37-16318df/plan9-386-0intro
2021-10-23T16:18:19-c96bc14-16318df/plan9-386-0intro
2021-10-23T16:18:19-c96bc14-16318df/plan9-amd64-0intro
2021-10-23T16:18:19-affba50-16318df/plan9-386-0intro
2021-10-23T16:18:19-16318df/plan9-386-0intro
2021-10-23T16:18:19-16318df/plan9-amd64-0intro
2021-10-23T16:18:19-16318df/plan9-arm
2021-10-23T16:18:15-12e4404/plan9-arm
2021-10-23T16:18:09-c526cf8/plan9-arm
2021-10-23T16:18:04-2d205eb/plan9-arm
2021-10-23T16:17:58-3cd28ba/plan9-arm
2021-10-23T12:44:47-b0f7eb6/plan9-arm
2021-10-22T21:48:34-ae4d67c/plan9-arm
2021-10-22T21:12:07-76dd01f/plan9-arm
2021-10-22T14:05:31-ff8115d/plan9-arm
2021-10-21T17:31:42-392bb06/plan9-arm
2021-10-21T17:23:58-02a3666/plan9-arm
2021-10-21T13:05:07-8b9c0d1/plan9-arm
2021-10-21T02:35:47-9b675d0-fca46d0/plan9-arm
2021-10-20T20:39:19-d419a80/plan9-arm
2021-10-20T20:38:59-75b73d6/plan9-arm
2021-10-20T20:38:47-3ec8d4b/plan9-arm
2021-10-20T19:30:13-cbf1d01-3ff39c5/plan9-arm
2021-10-20T19:30:13-3ff39c5/plan9-arm
2021-10-20T19:29:12-4320949/plan9-arm
2021-10-20T18:48:51-cbf1d01-4e565f7/plan9-arm
2021-10-20T05:28:45-b091189/plan9-arm
2021-10-19T23:27:41-4ea6123-2be5b84/plan9-arm
2021-10-19T22:47:35-9fa8551/plan9-arm
2021-10-19T18:56:08-07e5527/plan9-arm
2021-10-19T18:43:12-d944984/plan9-arm
2021-10-19T18:43:08-1d63052/plan9-arm
2021-10-19T18:22:59-6c3cd5d/plan9-arm
2021-10-19T18:18:37-9820602/plan9-arm
2021-10-19T17:12:53-404f84d/plan9-arm
2021-10-19T17:01:42-3a07ab7/plan9-arm
2021-10-19T16:19:27-98f6e03-f92a358/plan9-arm
2021-10-19T08:44:22-1e49210/plan9-arm
2021-10-19T07:45:46-281050f-ee92daa/plan9-arm
2021-10-18T22:56:07-eba0e86/plan9-arm
2021-10-18T22:55:05-7397178/plan9-arm
2021-10-18T19:46:27-98f6e03-394a1ad/plan9-arm
2021-10-18T16:32:38-543a513/plan9-arm
2021-10-18T15:38:40-74acbaf/plan9-arm
2021-10-16T16:27:40-c96bc14-640a49b/plan9-386-0intro
2021-10-16T16:27:40-98f6e03-640a49b/plan9-arm
2021-10-16T16:27:40-640a49b/plan9-arm
2021-10-16T16:27:40-4f30a5c-640a49b/plan9-386-0intro
2021-10-16T14:50:21-c96bc14-680caf1/plan9-386-0intro
2021-10-16T14:50:21-98f6e03-680caf1/plan9-386-0intro
2021-10-16T14:50:21-4f30a5c-680caf1/plan9-386-0intro
2021-10-15T21:46:06-ed1c8db/plan9-386-0intro
2021-10-15T21:46:06-ed1c8db/plan9-amd64-0intro
2021-10-15T21:46:06-c96bc14-ed1c8db/plan9-amd64-0intro
2021-10-15T21:46:06-4f30a5c-ed1c8db/plan9-amd64-0intro
2021-10-15T21:44:07-4a7975e/plan9-386-0intro
2021-10-15T20:29:48-85cbdda/plan9-arm
2021-10-15T20:29:36-a17b2e8/plan9-386-0intro
2021-10-15T19:54:44-22951fb/plan9-arm
2021-10-15T19:03:33-98f6e03-cfe6763/plan9-arm
2021-10-15T18:07:49-98f6e03-8331f25/plan9-arm
2021-10-15T02:45:22-288d0b3-3da0ff8/plan9-arm
2021-10-15T02:02:50-ce04ca3-3da0ff8/plan9-arm
2021-10-15T02:02:50-3da0ff8/plan9-386-0intro
2021-10-15T00:16:20-ce04ca3-9e8ed86/plan9-arm
2021-10-14T23:32:09-3a269dc-9e8ed86/plan9-arm
2021-10-14T22:15:24-281050f-b59467e/plan9-arm
2021-10-14T18:44:37-a37bebc/plan9-arm
2021-10-14T18:44:29-a8c5a99/plan9-arm
2021-10-14T15:17:03-fbdf830/plan9-arm
2021-10-14T13:09:28-88e80df-24e798e/plan9-arm
2021-10-14T13:09:28-24e798e/plan9-arm
2021-10-14T12:38:31-88e80df-2feb2cc/plan9-arm
2021-10-14T07:18:59-1349c6e/plan9-arm
2021-10-14T04:18:44-acac939-9e4dc6f/plan9-arm
2021-10-14T01:51:22-276fb27/plan9-arm
2021-10-13T22:21:08-ac2ed98-b90d258/plan9-arm
2021-10-13T20:31:25-b90d258/plan9-arm
2021-10-13T18:15:48-40f82f8/plan9-amd64-0intro
2021-10-13T18:15:48-40f82f8/plan9-arm
2021-10-13T18:15:48-281050f-40f82f8/plan9-arm
2021-10-13T17:02:43-b5904f3/plan9-arm
2021-10-13T16:39:23-3986e5c/plan9-arm
2021-10-13T16:36:59-69041c7/plan9-arm
2021-10-13T15:11:16-0454d73/plan9-arm
2021-10-13T03:12:54-24389d4-4fb2e1c/plan9-arm
2021-10-12T22:02:39-732f6fa/plan9-arm
2021-10-12T21:30:45-0b930fb-3283d1a/plan9-arm
2021-10-12T21:15:50-3d051ba/plan9-arm
2021-10-12T20:53:52-ac6d706/plan9-arm
2021-10-12T20:20:41-d032b2b/plan9-arm
2021-10-12T18:30:24-281050f-6e0adde/plan9-arm
2021-10-12T12:24:09-c8768b9-4679670/plan9-arm
2021-10-12T06:55:50-c83725b-d887d3b/plan9-arm
2021-10-11T20:46:14-7023535/plan9-arm
2021-10-11T20:46:14-18fa840-7023535/plan9-arm
2021-10-11T19:20:12-18fa840-65ffee6/plan9-arm
2021-10-11T15:48:29-2478b6f/plan9-arm
2021-10-11T12:53:11-3b03d30-34f7b1f/plan9-386-0intro
2021-10-11T12:53:11-34f7b1f/plan9-arm
2021-10-11T12:42:06-59b2f51/plan9-arm
2021-10-10T15:11:26-d2d21d9/plan9-386-0intro
2021-10-10T15:11:26-d2d21d9/plan9-amd64-0intro
2021-10-10T15:11:26-d2d21d9/plan9-arm
2021-10-10T15:11:26-3b03d30-d2d21d9/plan9-386-0intro
2021-10-10T15:11:26-1af23bd-d2d21d9/plan9-386-0intro
2021-10-10T13:37:52-1af23bd-b99abf3/plan9-arm
2021-10-09T23:29:26-b99abf3/plan9-386-0intro
2021-10-09T23:29:26-b99abf3/plan9-arm
2021-10-08T22:21:41-ee04797-74abcab/plan9-arm
2021-10-08T19:41:16-c5188f2-7fcf9a1/plan9-arm
2021-10-08T19:41:16-7fcf9a1/plan9-386-0intro
2021-10-08T19:30:58-f430bda/plan9-arm
2021-10-08T19:19:09-b9e1e1b/plan9-386-0intro
2021-10-08T18:07:25-d480b5c/plan9-386-0intro
2021-10-08T17:58:41-16a3cef/plan9-arm
2021-10-08T17:32:31-d7ba1d2/plan9-386-0intro
2021-10-08T17:32:25-a7d3a0e/plan9-arm
2021-10-08T15:27:20-b3a4b2d/plan9-arm
2021-10-08T14:08:12-5b9206f/plan9-386-0intro
2021-10-08T14:08:12-5b9206f/plan9-arm
2021-10-07T22:19:58-1ceb723/plan9-arm
2021-10-07T21:39:17-6b4cf2b/plan9-arm
2021-10-07T20:37:43-d477ef3-ef2ebbe/plan9-arm
2021-10-07T19:56:19-4ab3c10/plan9-arm
2021-10-07T18:58:33-7286502/plan9-arm
2021-10-07T18:09:53-4607ebc/plan9-arm
2021-10-07T16:38:24-be571a3/plan9-arm
2021-10-07T14:37:45-ebeab63/plan9-arm
2021-10-07T14:37:36-375a1fb/plan9-arm
2021-10-07T02:30:47-2c98350-39bbf08/plan9-arm
2021-10-06T23:26:57-c8dd89e/plan9-arm
2021-10-06T23:26:57-86761ae-c8dd89e/plan9-arm
2021-10-06T21:17:08-2e107b4/plan9-arm
2021-10-06T21:06:48-197b328/plan9-arm
2021-10-06T15:53:04-4d8db00/plan9-arm
2021-10-05T23:58:29-ac60900/plan9-arm
2021-10-05T23:36:43-e82ed0c/plan9-arm
2021-10-05T22:42:11-96fface/plan9-arm
2021-10-05T20:52:58-0d6561b/plan9-arm
2021-10-05T20:35:54-990c9c6/plan9-arm
2021-10-05T19:15:03-ae83301/plan9-arm
2021-10-05T18:31:14-5d946f1/plan9-arm
2021-10-05T17:10:45-2dc2755-5140ad1/plan9-arm
2021-10-05T15:47:57-017ffcd/plan9-arm
2021-10-05T15:47:07-cfd74f7/plan9-arm
2021-10-05T01:30:56-d4b1ae0-123393a/plan9-386-0intro
2021-10-04T22:59:44-78c2529/plan9-arm
2021-10-04T22:59:13-199ec42/plan9-arm
2021-10-04T22:54:58-7c79e8e/plan9-arm
2021-10-04T22:00:53-d9952ff/plan9-386-0intro
2021-10-04T22:00:19-739328c/plan9-arm
2021-10-04T21:10:16-db89b5a-f0db7ea/plan9-arm
2021-10-04T18:15:09-ccaa907-9f8d558/plan9-arm
2021-10-04T17:17:11-cedda3a-cc5e3de/plan9-386-0intro
2021-10-04T17:17:11-ccaa907-cc5e3de/plan9-386-0intro
2021-10-04T17:17:11-cc5e3de/plan9-amd64-0intro
2021-10-04T15:50:43-9432320/plan9-386-0intro
2021-10-04T15:50:43-9432320/plan9-amd64-0intro
2021-10-02T16:05:55-0ebff1a-a7fe161/plan9-arm
2021-10-02T13:15:42-0ebff1a-7bf7bbc/plan9-arm
2021-10-01T23:51:16-c129af9/plan9-arm
2021-10-01T21:26:43-0ebff1a-0d65c27/plan9-arm
2021-10-01T19:30:53-13c407c-0d65c27/plan9-arm
2021-10-01T18:18:46-0d65c27/plan9-arm
2021-10-01T17:18:20-5279e53/plan9-arm
2021-10-01T16:45:09-58ad7eb-f19b2d5/plan9-arm
2021-10-01T16:17:08-f19b2d5/plan9-arm
2021-10-01T16:01:09-558b9da-8d494b0/plan9-arm
2021-10-01T15:59:38-e5ad363/plan9-arm
2021-10-01T14:06:36-96d3ba8/plan9-arm
The text was updated successfully, but these errors were encountered: