syscall: memory corruption on OpenBSD/amd64 and NetBSD/amd64,386 when forking #34988
Comments
|
I missed that 1.13.3 was also released yesterday. Currently updating to that and will report whether this is still an issue. |
|
This looks like cmd/go crashing while building the test, not the test itself. |
|
@jrick maybe you meant this in your original post, but I just want to be clear. Does this reproduce with Go 1.12.X or older versions of Go? Since we have a reasonable reproducer, the next step to me would be to just bisect what went into Go 1.13, if we know it isn't reproducing in Go 1.12. I genuinely have no idea what this could be. I thought at first that it could be scavenging related but that's highly unlikely for a number of reasons. I won't rule it out yet, though. |
|
I haven't tested 1.12.x but will follow up testing that next. Currently hammering this test with 1.13.3 and so far it has not failed, but my application built with 1.13.3 still fails with SIGBUS (could be unrelated). |
|
@mknyszek it still hasn't failed on 1.13.3 (running close to an hour now) but quickly failed on 1.12.12. |
|
1.13.3 finally errored after an hour. more errors from 1.13.3: |
dmitshur
added
NeedsInvestigation
|
This remains a problem in 1.13.5, so it's not addressed by the recent fixes to the go tool. |
|
This may be fork/exec related. This program exhibits similar crashes on OpenBSD 6.7 and Go 1.14.3. crash trace: https://gist.github.com/jrick/8d6ef72796a772668b891310a18dd805 Synchronizing the os/exec call with an additional mutex appears to remove the crash. |
ianlancetaylor
changed the title
|
Thanks for the stack trace. That looks very much like a forked child process is changing the memory seen by the parent process. Which should of course be impossible. Specifically it seems that |
|
I'm seeing another strange thing in addition to that crash. Sometimes the program will run forever, spinning cpu, but appears to be deadlocked because none of the pids of those true processes are ever changing. Here's the trace after sending sigquit: https://gist.github.com/jrick/74aaa63624961145b7bc7b9518da75e1 |
|
I am currently testing with this OpenBSD kernel patch to the virtual memory system: https://marc.info/?l=openbsd-tech&m=160008279223088&w=2 however these crashes still persist. Another interesting data point: so far it appears that this only reproduces on amd ryzen cpus, and not any intel ones. |
|
https://build.golang.org/log/3f45171bc52a0a86435abb9f795c0e8a45c4a0b0 looks similar: |
|
https://storage.googleapis.com/go-build-log/abee19ae/openbsd-amd64-68_0f13ec3d.log (a TryBot) looks like it could plausibly be from a fork syscall. |
|
I'm not sure when this changed but since returning to this issue I haven't been able to reproduce with my minimal test case again on the same hardware with OpenBSD 7.0-current and Go 1.17.3. I suspect it's due to some OpenBSD fix if the 6.8 builders are still hitting this. (also 6.8 is no longer a supported OpenBSD version; i don't think it makes much sense to continue testing with it) |
|
spoke too soon: |
|
and it took far longer than 1.17.3 but a very similar crash (in scanstack) still occurs with |
|
I can also reproduce crashes on netbsd-386 and netbsd-amd64 with #34988 (comment) on AMD, of the form: as well as #49453 |
|
Some observations I've made (from netbsd-amd64): The crashes still seem to occur with GOMAXPROCS=1, however Go still has some background threads in this case. Disabling sysmon and GC makes this program truly single-threaded: Once the program is truly single-threaded, the crashes disappear. Setting GOMAXPROCS=2 with this patch brings the crashes back. Here is a slightly simplified reproducer: package main
import (
"os/exec"
"runtime"
)
func main() {
go func() {
for {
err := exec.Command("/usr/bin/true").Run()
if err != nil {
panic(err)
}
}
}()
for {
runtime.Gosched()
}
}This version has only a single forker, but crashes about as quickly. The (cc @aclements @mknyszek) |
|
More observations:
I've simplified that repro even further:
package main
import (
//"runtime"
"syscall"
)
func fork() int32
func main() {
go func() {
for {
pid := fork()
syscall.Syscall6(syscall.SYS_WAIT4, uintptr(pid), 0, 0, 0, 0, 0)
//syscall.RawSyscall6(syscall.SYS_WAIT4, uintptr(pid), 0, 0, 0, 0, 0)
}
}()
for {
syscall.Syscall(syscall.SYS_GETPID, 0, 0, 0)
//runtime.Gosched()
}
}
The key parts here:
The crashes I get with this look like (source): This is complaining that the assertion The one case I've caught in GDB looks like (stopped just inside the failing branch): From the assembly, Of course, I can't really tell if that memory location read as zero, or if the register was cleared after the load somehow. |
prattmic
changed the title
|
I've filed a bug with NetBSD to get some more help looking at this. Still awaiting the problem ID for my bug report. |
|
Hopefully we're chasing the same bug as on OpenBSD/amd64. I've been running the single-forker reproducer from #34988 (comment) for several hours now and it has not crashed yet (though my own reproducers were taking just as long). I haven't yet tested OpenBSD/386 at all. |
|
@coypoop and I have been trying to investigate this too in NetBSD/amd64. We've been using the With both reproducers, the crashes are very quick, under a minute, short enough we haven't bothered to measure. We tried finding a C program that does the same thing -- create 100 threads, fork and wait4, verify some invariants on the stack and in thread-local storage with We tried setting GOGC=off, in case it was some activity in the garbage collector that somehow triggered the bug, but the same crashes persisted. We tried removing the nontemporal stores in memclr and memmove, just using REP STOS and REP MOVS, in case Intel and AMD have any different semantics/bugs about nontemporal stores and store fences (which are used in Go only in memclr and memmove as far as I can tell), but the same crashes persisted. (We made sure to use STOSQ/MOVSQ when aligned and STOSB/MOVSB only when unaligned, and to do forward/backward copies as appropriate for overlapping regions in memmove; the tests passed with our changes.) Prompted by https://gitweb.dragonflybsd.org/dragonfly.git/commitdiff/1a923442efb8ded428689c89162631d739cf0548 we hypothesized that perhaps the fs segment is not always being set correctly on return to the Go userland program on AMD CPUs after syscalls and/or context switches. We tried changing the way the fsbase and fs segment selector are set in the NetBSD kernel (avoiding null segment selectors), but the crashes persisted. We tried instrumenting Go to check for fs corruption -- verifying it against gs (by querying the OS for _lwp_self() in entersyscall and entersyscallblock, storing it in g, and verifying it in exitsyscall) and against sp (by pushing leaq %fs:0 before every syscall instruction, and then popping and comparing after syscall) -- but none of these diagnostics bore any fruit; the same crashes seemed to persist. We dug into entersyscall and found that |
|
@riastradh i can confirm this test panics frequently in the child process on openbsd. it leaves an increasing number of zombie processes around as well. edit: misread the ps output, the lingering processes are sleeping, not zombies |
This comment has been minimized.
This comment has been minimized.
|
FWIW, the NetBSD issue I filed is at http://gnats.netbsd.org/56535. @bokunodev the issue you are describing is not related to this bug. You perhaps accidentally replied to this bug, or need to file a new issue. |
|
After a week of noodling with timing and random delay, managed to arrive at this Go reproducer, where the failing code is simpler (does not involve G corruption). This is not clean; it's less than 90 minutes old. The observed failure is that the forking parent goroutine does a store to |
|
And fork.s, a minor edit that perhaps matters: |
|
I managed to reproduce this on NetBSD with a C program!
#include <pthread.h>
#include <stdbool.h>
#include <stdint.h>
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <sys/mman.h>
#include <sys/syscall.h>
#include <sys/wait.h>
#include <unistd.h>
void __attribute((noinline)) spin(uint64_t loops) {
for (volatile uint64_t i = 0; i < loops; i++) {
}
}
struct thing {
uint64_t b;
uint32_t c; // Making this (plus 'sink' below) uint64_t may make repro take longer?
};
volatile struct thing* page;
volatile uint32_t sink;
int ready;
void* thread(void* arg) {
__atomic_store_n(&ready, 1, __ATOMIC_SEQ_CST);
while (1) {
// Spin not strictly required, but it speeds up repro in my case.
spin(40*1000);
// Atomic not required, this works too:
// page->c = sink;
__atomic_store_n(&page->c, sink, __ATOMIC_SEQ_CST);
sink++;
}
}
int main(void) {
page = mmap(NULL, 4096, PROT_READ | PROT_WRITE, MAP_ANONYMOUS | MAP_PRIVATE, -1, 0);
if (page == MAP_FAILED) {
perror("mmap");
return 1;
}
pthread_t thread_id;
int ret = pthread_create(&thread_id, NULL, &thread, NULL);
if (ret != 0) {
perror("pthread_create");
return 1;
}
// Wait for child thread to start.
//
// This is not required to repro, but eliminates racing fork+thread create as
// a possibility.
while (!__atomic_load_n(&ready, __ATOMIC_SEQ_CST)) {
}
int64_t i = 0;
while (1) {
i++;
if (i % 10000 == 0) {
printf("Loop %d...\n", i);
}
page->b = 102;
// Does not work with libc fork(). libc fork() is significantly slower,
// which may be the problem.
uint64_t pid = syscall(SYS_fork);
if (pid == 0) {
/* Child */
_exit(0);
}
/* Parent */
/* spin(40*1000); may speed up repro. */
page->b = 2;
uint64_t pb = page->b;
if (pb != 2) {
printf("Corruption! pb, page->b = %lu, %lu\n", pb, page->b);
_exit(1);
}
int status;
ret = waitpid(pid, &status, 0);
if (ret < 0) {
perror("waitpid");
return 1;
}
if (WEXITSTATUS(status) != 0) {
printf("Bad child status %#x\n", status);
return 1;
}
}
}Build and run: $ cc -pthread forkstress.c
$ ./a.out & ./a.out & ./a.out & ./a.out
Loop 10000...
Corruption! pb, page->b = 102, 2
Loop 10000...
Loop 10000...
Loop 10000...
Corruption! pb, page->b = 102, 2
Loop 20000...
Loop 20000...
Corruption! pb, page->b = 102, 2Notes:
The summarized behavior we see is:
All while another thread is spinning writing to |
|
Running the netbsd C reproducer on openbsd with the kernel patch below to allow the syscalls from the program's text segment, I'm not able to hit the corruption. (maybe now that i've commented, the corruption will happen again?) |
|
@jrick your comment reminds me that I don't need to use assembly to make a direct syscall in C, I can just use I also haven't had luck reproducing with this program on openbsd-386. FWIW, it runs much slower than netbsd-amd64. The latter gets to 10k loops in a few seconds, the former takes minutes to do that same. I'm not sure if this is a difference in OpenBSD's fork, or a 386 difference. Edit: seems to be mostly 386. openbsd-amd64 is faster (though not quite as fast as netbsd). |
|
Change https://golang.org/cl/372355 mentions this issue: |
|
Change https://golang.org/cl/370874 mentions this issue: |
|
We found a similar mutiprocessor COW bug in NetBSD (same as https://reviews.freebsd.org/D14347) but I'm pretty sure it's not the same bug as this one, and although the fix for the similar bug empirically seems to suppress the symptoms of this one, I am pretty sure it's not a correct fix. The crux of the similar bug is: The bug is that the new page may be exposed writably to thread B while thread C still has TLB entries for the old page, since multiprocessor TLB invalidation isn't simultaneous on all CPUs at once and TLB updates aren't ordered with respect to atomic operations. The fix is either to expose the new page nonwritably to all CPUs first, and then expose it writably to all CPUs (as FreeBSD does now); or (simpler but maybe slower) to prohibit all access to the old page on all CPUs first when doing copy-on-write, and then expose the new page writably to all CPUs -- either way, B can begin to write to it only after C has witnessed the new page. We haven't been able to reproduce either problem with the following NetBSD kernel patch: However, I'm pretty sure the golang issue can't be the same issue and I suspect that this change only papers over the symptoms. The difference is that with the golang bug, the same thread writes to We've tried verifying which CPU it's on across the two reads using _lwp_ctl but haven't observed any migration in the event of the crash. We've also tried measuring the time spent in executing the lines by surrounding them in New code: forkstress.c.txt On a couple different CPUs (one advertised as 3.4 GHz, one as 3.6 GHz, to be taken with a grain of salt owing to dynamic frequency scaling, of course), with the results scaled by the TSC frequency, we got about 10ns, 130ns, 40ns, and 50ns. These latencies are enough to cover a couple dozen or so CPU instructions, a few L1 or L2 cache fetches, and maybe a load (e.g., a page table entry) from main memory (but in 10ns -- which we observed at -O0! -- even that seems unlikely). Certainly even 130ns is nowhere near enough time for the OS's page fault handler to run and do the copy-on-write; it looks like the COW fault must happen in the other thread. So exactly what is happening when executing that leads to %rsi holding 102 instead of 2 is unclear, but there are very tight limits to what the CPU could possibly be doing. I'm struggling to come up with hypotheses that don't involve some microarchitectural bug like a buffered store getting reordered around invlpg. (To rule that hypothesis out, we tried inserting |
|
We also tried pinning the threads to logical CPUs so they never run as multiple hyperthreads on the same core, or so that they always run as multiple hyperthreads on the same core, but the only difference this might have made was how long it took the problem to trigger. (Of course, when pinned to a single logical CPU and time-shared there was no problem!) |
|
I ran 4 copies of forkstress in the background yesterday and overnight , and this morning it's still chugging along NetBSD arm64 9.99.92 NetBSD 9.99.92 aarch64 evbarm Raspberry Pi 4 / 8GB w/ 0.5T SSD A few lines of output this morning, below. If we normalize by dividing those number by 10,000, the biggest difference is: It would appear there is no problem with aarch64 NetBSD 9.99.92 (-current as of a few days ago) -Mike Loop 43690000... p.s. 73 / 4399 = 1.7 % It's unclear whether this 'non-uniformity' of allocating processor time among the RPi's 4 processors is a 'bug' or not. |
FWIW, I did this as well when the repro was still in Go, and I never saw a CPU migration there either. |
can't be written to while any thread can see the original version of the page via a not-yet-flushed stale TLB entry: pmaps can indicate they do this correctly by defining __HAVE_PMAP_MPSAFE_ENTER_COW; uvm will force the initial CoW fault to be read-only otherwise. Set that on amd64 and fix the problem case in pmap_enter() by putting a read-only mapping in place, shooting the TLB entry, then fixing it to the final read-write entry so this thread can continue without re-faulting. reported by jsing@ from golang/go#34988 assisted by discussion in https://reviews.freebsd.org/D14347 tweaks from jsing@ and kettenis@ ok jsing@ mpi@ kettenis@
|
Two more occurrences on the |
|
...I wonder if the failures on (CC @golang/runtime @golang/solaris @golang/aix @golang/netbsd @golang/openbsd) |
|
I tried the linked C program on aix with a couple modifications. I had to modify it to use |
|
https://storage.googleapis.com/go-build-log/4697874e/openbsd-amd64-68_4c637ab4.log (another |
|
Curiously, I'm not aware of any of these failures on the |
|
OpenBSD applied the following change: Based on a cursory skim (without having thought much about the details of the change), I suspect that this change is designed to fix another copy-on-write bug just like https://reviews.freebsd.org/D14347 and like I described in #34988 (comment). However, I have no theory for how the FreeBSD fix or the NetBSD fix could affect the problem we detected here, because that copy-on-write bug -- and the fix in FreeBSD and NetBSD -- makes sense only if the TLB IPI handler runs between the store and load of the memory location at issue, whereas our measurements on NetBSD indicate that that's physically implausible for the issue in this thread. In NetBSD, we have not yet committed a fix for that COW bug because our draft fixes (like the one in #34988 (comment)) seem to have the side effect of suppressing the issue in this thread. I'm still hoping to hear from AMD with an idea about what could be going wrong in the 10ns window we observed but they haven't gotten back to me yet. However, if we don't hear anything before NetBSD 10.0 is ready we'll probably just apply the unrelated-COW fix. |
What version of Go are you using (
go version)?Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (
go env)?go envOutputWhat did you do?
I observed these issues in one of my applications, and assumed it was a race or invalid unsafe.Pointer usage or some other fault of the application code. When the 1.13.2 release dropped yesterday I built it from source and observed a similar issue running the regression tests. The failed regression test does not look related to the memory corruption, but I can reproduce the problem by repeatedly running the test in a loop:
It can take several minutes to observe the issue but here are some of the captured panics and fatal runtime errors:
https://gist.githubusercontent.com/jrick/f8b21ecbfbe516e1282b757d1bfe4165/raw/6cf0efb9ba47ba869f98817ce945971f2dff47d6/gistfile1.txt
https://gist.githubusercontent.com/jrick/9a54c085b918aa32910f4ece84e5aa21/raw/91ec29275c2eb1be49f62ad8a01a5317ad168c94/gistfile1.txt
https://gist.githubusercontent.com/jrick/8faf088593331c104cc0da0adb3f24da/raw/7c92e7e7d60d426b2156fd1bdff42e0717b708f1/gistfile1.txt
https://gist.githubusercontent.com/jrick/4645316444c12cd815fb71874f6bdfc4/raw/bffac2a448b07242a538b77a2823c9db34b6ef6f/gistfile1.txt
https://gist.githubusercontent.com/jrick/3843b180670811069319e4122d32507a/raw/0d1f897aa25d91307b04ae951f1b260f33246b61/gistfile1.txt
https://gist.githubusercontent.com/jrick/99b7171c5a49b4b069edf06884ad8e17/raw/740c7b9e8fa64d9ad149fd2669df94e89c466927/gistfile1.txt
Additionally, I observed
go runhanging (no runtime failure due to deadlock) and it had to be killed with SIGABRT to get a trace: https://gist.githubusercontent.com/jrick/d4ae1e4355a7ac42f1910b7bb10a1297/raw/54e408c51a01444abda76dc32ac55c2dd217822b/gistfile1.txtIt may not matter which regression test is run as the errors also occur in run.go.
The text was updated successfully, but these errors were encountered: