-
Notifications
You must be signed in to change notification settings - Fork 17.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
runtime: allocated pages below zero on openbsd/riscv64 #63385
Comments
Hm, unfortunately this doesn't seem like it would be a kernel/mmap problem. This is a check that's failing when one or more pages are freed back to the page heap. What's happening is that some of the bookkeeping maintained per 4 MiB aligned address range indicates that the region is totally free, so it doesn't make sense to try and free memory back to it. I think it implies some kind of corruption of the data structure, or a data race. It could also mean that when the check isn't firing, some pages are getting allocated twice (which could lead to other memory corruption symptoms). It may just be reproducing in If this is straightforward to reproduce, my first thought would be to pepper debuglog calls around the various allocs and frees to the page heap and make sure nothing is being allocated (and thus later possibly freed) twice. If you haven't used debuglog before, it's a very handy low-overhead tool for identifying what actually happened in the runtime. Because it's low overhead, it maintains a low chance of perturbing bugs involving race conditions and tends to reproduce real thread orderings. You can enable it by passing |
Found new dashboard test flakes for:
2023-09-28 17:55 openbsd-riscv64-jsing sys@2964e1e4 go@3a69dcdc go/types [build] (log)
2023-09-28 17:55 openbsd-riscv64-jsing sys@2964e1e4 go@68a12a80 runtime [build] (log)
2023-10-04 10:08 openbsd-riscv64-jsing go@0074125c encoding/gob.TestLargeSlice (log)
2023-10-04 15:11 openbsd-riscv64-jsing go@a9036396 runtime.TestFakeTime (log)
2023-10-04 16:05 openbsd-riscv64-jsing go@0c64ebce cmd/link.TestMachOBuildVersion (log)
2023-10-04 16:50 openbsd-riscv64-jsing go@3a69dcdc cmd/go.TestGoListExport (log)
2023-10-04 22:08 openbsd-riscv64-jsing net@5d5a036a go@bc150700 runtime [build] (log)
|
Found new dashboard test flakes for:
2023-10-04 22:08 openbsd-riscv64-jsing net@5d5a036a go@008dabcd cmd/link/internal/ld [build] (log)
|
Found new dashboard test flakes for:
2023-10-05 19:45 openbsd-riscv64-jsing go@10da3b64 cmd/link/internal/ld [build] (log)
2023-10-06 15:42 openbsd-riscv64-jsing go@28f4ea16 cmd/compile/internal/ssagen [build] (log)
|
Found new dashboard test flakes for:
2023-10-05 16:33 openbsd-riscv64-jsing net@88194ad8 go@10da3b64 cmd/compile/internal/ssa [build] (log)
2023-10-05 16:33 openbsd-riscv64-jsing net@88194ad8 go@008dabcd runtime [build] (log)
|
Found new dashboard test flakes for:
2023-09-28 17:55 openbsd-riscv64-jsing sys@2964e1e4 go@10da3b64 runtime [build] (log)
|
Found new dashboard test flakes for:
2023-10-06 21:01 openbsd-riscv64-jsing go@be3d5fb6 runtime [build] (log)
|
Found new dashboard test flakes for:
2023-10-06 20:57 openbsd-riscv64-jsing go@b72bbaeb cmd/go.TestScript (log)
|
Found new dashboard test flakes for:
2023-10-05 16:33 openbsd-riscv64-jsing net@88194ad8 go@8b6e0e6e runtime [build] (log)
|
Found new dashboard test flakes for:
2023-10-06 20:48 openbsd-riscv64-jsing go@ad76a98d cmd/compile/internal/ssa [build] (log)
|
Found new dashboard test flakes for:
2023-10-05 16:33 openbsd-riscv64-jsing net@88194ad8 go@b01cb72e net/http [build] (log)
|
Found new dashboard test flakes for:
2023-10-06 17:02 openbsd-riscv64-jsing go@b455e239 runtime.TestArenaCollision (log)
2023-10-06 17:02 openbsd-riscv64-jsing go@b455e239 runtime.TestArenaCollision (log)
|
Found new dashboard test flakes for:
2023-10-06 20:50 openbsd-riscv64-jsing sys@8ccaaf02 go@561bf045 runtime [build] (log)
2023-10-06 20:50 openbsd-riscv64-jsing sys@8ccaaf02 go@f711892a runtime [build] (log)
2023-10-06 20:50 openbsd-riscv64-jsing sys@8ccaaf02 go@6e8caefc runtime [build] (log)
|
Found new dashboard test flakes for:
2023-10-07 12:31 openbsd-riscv64-jsing go@f711892a runtime [build] (log)
2023-10-08 23:15 openbsd-riscv64-jsing go@2744155d runtime [build] (log)
|
Found new dashboard test flakes for:
2023-10-05 16:33 openbsd-riscv64-jsing net@88194ad8 go@262b8050 go/types [build] (log)
|
Found new dashboard test flakes for:
2023-10-06 20:50 openbsd-riscv64-jsing sys@8ccaaf02 go@b201cf4a (log)
|
Found new dashboard test flakes for:
2023-10-05 16:33 openbsd-riscv64-jsing net@88194ad8 go@5955a03e runtime [build] (log)
|
Found new dashboard test flakes for:
2023-10-10 17:14 openbsd-riscv64-jsing go@7e1713e9 encoding/gob.TestLargeSlice (log)
|
Found new dashboard test flakes for:
2023-10-10 15:45 openbsd-riscv64-jsing net@b225e7ca go@e7015c93 runtime [build] (log)
|
Found new dashboard test flakes for:
2023-10-06 20:50 openbsd-riscv64-jsing sys@8ccaaf02 go@e0948d82 cmd/compile/internal/ssa [build] (log)
2023-10-06 20:50 openbsd-riscv64-jsing sys@8ccaaf02 go@778880b0 runtime [build] (log)
2023-10-09 16:00 openbsd-riscv64-jsing go@7fcc626b cmd/compile/internal/ssa [build] (log)
2023-10-09 17:48 openbsd-riscv64-jsing go@e6990b76 encoding/gob.TestLargeSlice (log)
2023-10-10 00:36 openbsd-riscv64-jsing sys@1d9f0b6d go@a5943e9d runtime [build] (log)
2023-10-10 00:36 openbsd-riscv64-jsing sys@1d9f0b6d go@e12bdc80 cmd/compile/internal/ssa [build] (log)
2023-10-10 15:45 openbsd-riscv64-jsing net@b225e7ca go@f34964a5 go/types [build] (log)
2023-10-10 16:32 openbsd-riscv64-jsing go@f34964a5 cmd/compile/internal/ssa [build] (log)
2023-10-10 20:47 openbsd-riscv64-jsing go@7c446bab runtime [build] (log)
2023-10-10 22:48 openbsd-riscv64-jsing go@dacf1f1e cmd/compile/internal/ssa [build] (log)
2023-10-11 12:48 openbsd-riscv64-jsing go@e12bdc80 cmd/compile/internal/ssa [build] (log)
2023-10-11 16:52 openbsd-riscv64-jsing net@92728b3b go@655155d0 runtime [build] (log)
2023-10-11 16:55 openbsd-riscv64-jsing go@47a71d1e runtime.TestArenaCollision (log)
2023-10-11 20:20 openbsd-riscv64-jsing go@09aada24 encoding/gob.TestLargeSlice (log)
2023-10-11 21:54 openbsd-riscv64-jsing sys@1bfbee0e go@3de6033d runtime [build] (log)
2023-10-11 21:58 openbsd-riscv64-jsing net@d23d9bc5 go@c729dc18 cmd/compile/internal/ssa [build] (log)
2023-10-12 14:39 openbsd-riscv64-jsing go@ade730a9 regexp/syntax (log)
2023-10-12 18:09 openbsd-riscv64-jsing go@657c885f runtime/pprof.TestGoroutineCounts (log)
2023-10-14 21:46 openbsd-riscv64-jsing go@cc47df01 runtime.TestArenaCollision (log)
2023-10-14 22:30 openbsd-riscv64-jsing go@bc9dc8d4 net/http [build] (log)
2023-10-17 15:06 openbsd-riscv64-jsing go@aa05674b runtime [build] (log)
2023-10-17 18:43 openbsd-riscv64-jsing go@860c2557 runtime [build] (log)
2023-10-17 20:32 openbsd-riscv64-jsing go@a30967f4 runtime.TestArenaCollision (log)
2023-10-17 20:36 openbsd-riscv64-jsing go@ab2e3ce7 cmd/compile/internal/types2 [build] (log)
2023-10-18 00:24 openbsd-riscv64-jsing go@416bc85f cmd/compile/internal/ssa [build] (log)
2023-10-18 14:06 openbsd-riscv64-jsing go@045ce51b net/http [build] (log)
2023-10-18 14:46 openbsd-riscv64-jsing go@ce25ad60 encoding/gob.TestLargeSlice (log)
2023-10-18 14:46 openbsd-riscv64-jsing go@66287d55 net/http.TestOmitHTTP2Vet (log)
2023-10-18 15:03 openbsd-riscv64-jsing go@144252d2 regexp.TestMatch (log)
2023-10-18 17:56 openbsd-riscv64-jsing go@91bb19a5 runtime [build] (log)
2023-10-18 18:04 openbsd-riscv64-jsing go@3550a848 runtime.TestArenaCollision (log)
2023-10-18 18:04 openbsd-riscv64-jsing go@061d77cb runtime.TestArenaCollision (log)
2023-10-18 19:30 openbsd-riscv64-jsing go@b78aa6c2 go/types [build] (log)
2023-10-18 19:36 openbsd-riscv64-jsing go@5d0a277f net/http [build] (log)
2023-10-18 19:37 openbsd-riscv64-jsing go@16606351 runtime [build] (log)
2023-10-18 20:00 openbsd-riscv64-jsing go@1d400cdf encoding/gob.TestLargeSlice (log)
2023-10-19 12:02 openbsd-riscv64-jsing go@28f1bf61 cmd/go.TestGoListExport (log)
2023-10-19 12:23 openbsd-riscv64-jsing go@ab5bd159 runtime.TestFakeTime (log)
|
Found new dashboard test flakes for:
2023-10-24 20:28 openbsd-riscv64-jsing go@b5f87b54 runtime.TestArenaCollision (log)
|
Found new dashboard test flakes for:
2023-10-23 09:04 openbsd-riscv64-jsing go@6ba6e72e encoding/json.TestUnmarshalMaxDepth (log)
2023-10-23 09:05 openbsd-riscv64-jsing sys@1e638101 go@2d9e8ded runtime [build] (log)
2023-10-23 09:06 openbsd-riscv64-jsing go@bc2124da bootstrap/cmd/compile/internal/ssagen [build] (log)
2023-10-23 19:41 openbsd-riscv64-jsing go@e46e8610 runtime [build] (log)
2023-10-23 20:47 openbsd-riscv64-jsing go@f9c54f9c encoding/gob.TestLargeSlice (log)
2023-10-23 22:24 openbsd-riscv64-jsing net@48a59773 go@25ad137c runtime [build] (log)
2023-10-23 22:24 openbsd-riscv64-jsing net@48a59773 go@983d90e1 cmd/compile/internal/types2 [build] (log)
2023-10-24 13:47 openbsd-riscv64-jsing go@6f87db50 cmd/compile/internal/ssa [build] (log)
2023-10-24 17:49 openbsd-riscv64-jsing go@25ad137c runtime [build] (log)
2023-10-24 18:12 openbsd-riscv64-jsing go@9162c4be runtime [build] (log)
2023-10-24 20:24 openbsd-riscv64-jsing go@2d9e8ded runtime.TestArenaCollision (log)
|
Found new dashboard test flakes for:
2023-10-25 19:25 openbsd-riscv64-jsing go@57322b3c runtime [build] (log)
|
Found new dashboard test flakes for:
2023-10-25 18:07 openbsd-riscv64-jsing net@4c7a5b64 go@e7908ab9 cmd/compile/internal/ssa [build] (log)
|
Found new dashboard test flakes for:
2023-12-20 18:38 openbsd-riscv64-jsing go@adec22b9 runtime.TestArenaCollision (log)
2023-12-21 00:15 openbsd-riscv64-jsing go@f6509cf5 runtime.TestArenaCollision (log)
|
Found new dashboard test flakes for:
2023-12-19 21:36 openbsd-riscv64-jsing net@f12db26b go@0b568040 runtime [build] (log)
|
Found new dashboard test flakes for:
2023-12-18 19:42 openbsd-riscv64-jsing go@08bec0db cmd/internal/obj/x86.TestVexEvexPCrelative (log)
|
Found new dashboard test flakes for:
2023-12-05 20:38 openbsd-riscv64-jsing net@c1b6eee3 go@5686b650 runtime [build] (log)
|
Found new dashboard test flakes for:
2023-12-05 17:05 openbsd-riscv64-jsing go@16d3040a runtime [build] (log)
2023-12-05 17:17 openbsd-riscv64-jsing go@285ef16b encoding/gob (log)
2023-12-05 18:48 openbsd-riscv64-jsing go@3d2645f3 cmd/compile/internal/ssa [build] (log)
|
Found new dashboard test flakes for:
2023-11-29 21:54 openbsd-riscv64-jsing net@f812076c go@3d2645f3 runtime [build] (log)
|
Found new dashboard test flakes for:
2023-11-29 21:54 openbsd-riscv64-jsing net@f812076c go@285ef16b runtime [build] (log)
|
Found new dashboard test flakes for:
2023-12-13 16:20 openbsd-riscv64-jsing go@f2d243db runtime [build] (log)
|
Found new dashboard test flakes for:
2023-12-01 17:42 openbsd-riscv64-jsing sys@5ff87d7b go@450f5d90 bootstrap/cmd/compile/internal/types2 [build] (log)
|
Found new dashboard test flakes for:
2023-12-08 16:07 openbsd-riscv64-jsing net@577e44a5 go@e70d3433 runtime [build] (log)
|
Found new dashboard test flakes for:
2023-12-08 16:07 openbsd-riscv64-jsing net@577e44a5 go@100651b6 runtime [build] (log)
|
Found new dashboard test flakes for:
2023-12-08 16:07 openbsd-riscv64-jsing net@577e44a5 go@59275e26 cmd/compile/internal/ssa [build] (log)
2023-12-08 16:07 openbsd-riscv64-jsing net@577e44a5 go@20a03fc7 runtime [build] (log)
2023-12-19 21:36 openbsd-riscv64-jsing net@f12db26b go@1dddd83c runtime [build] (log)
2023-12-19 21:36 openbsd-riscv64-jsing net@f12db26b go@36a2463e runtime [build] (log)
|
Change https://go.dev/cl/553135 mentions this issue: |
@mknyszek many thanks for the pointers (especially re debuglog!) - I managed to find some time to investigate further and finally tracked the issue down to a bug that was introduced in 91f8630. In summary, the min index value is incorrectly updated, which results in an in-use part of the chunk backing store being remapped and zeroed. Thankfully building the runtime test and then running The bug effectively boils down to an incorrect
During the second I'm still somewhat perplexed by the fact that this only ever seems to be hit on openbsd/riscv64 - it would make sense if it was all 64 bit OpenBSD architectures, since they randomise memory allocations, which could lead to this allocation pattern. |
Found new dashboard test flakes for:
2023-12-01 17:42 openbsd-riscv64-jsing sys@5ff87d7b go@34416d7f bootstrap/cmd/link/internal/ld [build] (log)
|
Found new dashboard test flakes for:
2023-12-01 17:42 openbsd-riscv64-jsing sys@5ff87d7b go@af5d544b runtime [build] (log)
|
Found new dashboard test flakes for:
2023-12-01 17:42 openbsd-riscv64-jsing sys@5ff87d7b go@285ef16b bootstrap/cmd/compile/internal/ssa [build] (log)
|
Found new dashboard test flakes for:
2023-12-27 15:34 openbsd-riscv64-jsing go@988b718f encoding/gob (log)
|
Found new dashboard test flakes for:
2023-12-01 17:42 openbsd-riscv64-jsing sys@5ff87d7b go@6d7b3c8c bootstrap/cmd/link/internal/ld [build] (log)
|
The backing store for the scavengeIndex chunks slice is allocated on demand as page allocation occurs. When pageAlloc.grow is called, a range is allocated from a reserved region, before scavengeIndex.grow is called to ensure that the chunks needed to manage this new range have a valid backing store. The valid region for chunks is recorded as the index min and index max. Any changes need to take the existing valid range into consideration and ensure that a contiguous valid range is maintained. However, a bug in the min index handling can currently lead to an existing part of the chunk slice backing store being zeroed via remapping. Initially, there is no backing store allocated and both min and max are zero. As soon as an allocation occurs max will be non-zero, however it is still valid for min to be zero depending on the base addresses of the page allocations. A sequence like the following will trigger the bug: 1. A page allocation occurs requiring chunks [0, 512) (after rounding) - a sysMap occurs for the backing store, min is set to 0 and max is set to 512. 2. A page allocation occurs requiring chunks [512, 1024) - another sysMap occurs for this part of the backing store, max is set to 1024, however min is incorrectly set to 512, since haveMin == 0 (validly). 3. Another page allocation occurs requiring chunks [0, 512) - since min is currently 512 a sysMap occurs for the already mapped and inuse part of the backing store from [0, 512), zeroing the chunk data. Correct this by only updating min when either haveMax == 0 (the uninitialised case) or when needMin < haveMin (the case where the new backing store range is actually below the current allocation). Remove the unnecessary haveMax == 0 check for updating max, as the existing needMax > haveMax case already covers this. Fixes golang#63385 Change-Id: I9deed74c4ffa187c98286fe7110e5d735e81f35f Reviewed-on: https://go-review.googlesource.com/c/go/+/553135 Reviewed-by: Michael Knyszek <mknyszek@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Michael Pratt <mpratt@google.com> Run-TryBot: Joel Sing <joel@sing.id.au>
The openbsd/riscv64 port is regularly failing with errors like:
https://build.golang.org/log/bb95405e34149982611770caf632d58e9bc50ff4
https://build.golang.org/log/e5fb11bcae4d4dab31c13f7e271852e39ae5a6df
This seems particularly reproducible (although still intermittent) via the
TestArenaCollision
test:inUse
always appears to be zero, whilenpages
is some small positive value.It would seem that this is specific to openbsd/riscv64 and possibly even a kernel/mmap related problem.
/cc @mknyszek any pointers in tracking this down would be greatly appreciated.
The text was updated successfully, but these errors were encountered: