build: FreeBSD failed builds in AWS / works fine on physical hardware

[rhavenn]

I'm running FreeBSD port builds on a physical machine and in an AWS ec2 instance. The same exact environment fails to build Go in AWS. My FreeBSD jail versions are the same (11.1-p10) and the only package requirements are go14 and pkg. All other items come from the base OS (clang, etc..)

Go 1.10.2 is what I'm trying to build. It works fine across multiple poudriere build jails on the physical box and fails for all my poudriere build jails on the virtual box. Occasionally it will just hang for hours, but CPU usage is nill. Usually, it crashes.

My dmesg info for the physical box:

FreeBSD 11.1-RELEASE-p10 #0: Tue May 8 05:21:56 UTC 2018
root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64
FreeBSD clang version 4.0.0 (tags/RELEASE_400/final 297347) (based on LLVM 4.0.0)
VT(vga): resolution 640x480
CPU: Intel(R) Xeon(R) CPU E5-2420 v2 @ 2.20GHz (2200.05-MHz K8-class CPU)
Origin="GenuineIntel" Id=0x306e4 Family=0x6 Model=0x3e Stepping=4
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0x7fbee3ff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
AMD Features2=0x1
Structured Extended Features=0x281<FSGSBASE,SMEP,ERMS>
XSAVE Features=0x1
VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
TSC: P-state invariant, performance statistics
real memory = 25769803776 (24576 MB)
avail memory = 24902356992 (23748 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table:

My dmesg info for the AWS instance:
FreeBSD 11.1-RELEASE-p10 #0: Tue May 8 05:21:56 UTC 2018
root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64
FreeBSD clang version 4.0.0 (tags/RELEASE_400/final 297347) (based on LLVM 4.0.0)
VT(vga): text 80x25
XEN: Hypervisor version 4.2 detected.
CPU: Intel(R) Xeon(R) CPU E5-2686 v4 @ 2.30GHz (2300.05-MHz K8-class CPU)
Origin="GenuineIntel" Id=0x406f1 Family=0x6 Model=0x4f Stepping=1
Features=0x1783fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,HTT>
Features2=0xfffa3203<SSE3,PCLMULQDQ,SSSE3,FMA,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV>
AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
AMD Features2=0x21<LAHF,ABM>
Structured Extended Features=0x7a9<FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID>
XSAVE Features=0x1
Hypervisor: Origin = "XenVMMXenVMM"
real memory = 17179869184 (16384 MB)
avail memory = 16603602944 (15834 MB)
Event timer "LAPIC" quality 100
ACPI APIC Table:

go14 builds okay and runs, but when running the all.bash build script for 1.10.2 I get the following:

root@server-srv-PHP7-default:/wrkdirs/usr/ports/lang/go/work/go/src # ./all.bash
-ap: not found
go: not found
Building Go cmd/dist using /usr/local/go14/.
Building Go toolchain1 using /usr/local/go14/.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
fatal error: unexpected signal during runtime execution
[signal 0xb code=0x1 addr=0xa0 pc=0x473050]

runtime stack:
runtime.gothrow(0x5edb70, 0x2a)
/usr/local/go14/src/runtime/panic.go:503 +0x8e
runtime.sigpanic()
/usr/local/go14/src/runtime/sigpanic_unix.go:14 +0x5e
globrunqget(0xc208012000, 0x0, 0x800000299)
/usr/local/go14/src/runtime/proc.c:3245 +0xf0
findrunnable(0xc208012000)
/usr/local/go14/src/runtime/proc.c:1400 +0xcf
schedule()
/usr/local/go14/src/runtime/proc.c:1575 +0x151
runtime.park_m(0xc2080006c0)
/usr/local/go14/src/runtime/proc.c:1654 +0x113
runtime.mcall(0x46da94)
/usr/local/go14/src/runtime/asm_amd64.s:186 +0x5a

goroutine 1 [chan receive]:
main.install(0x5b4b10, 0x6)
/wrkdirs/usr/ports/lang/go/work/go/src/cmd/dist/build.go:567 +0x55
main.cmdbootstrap()
/wrkdirs/usr/ports/lang/go/work/go/src/cmd/dist/build.go:1178 +0x913
main.xmain()
/wrkdirs/usr/ports/lang/go/work/go/src/cmd/dist/main.go:178 +0x1a7
main.main()
/wrkdirs/usr/ports/lang/go/work/go/src/cmd/dist/main.go:161 +0x5ad

goroutine 6 [select]:
main.bghelper()
/wrkdirs/usr/ports/lang/go/work/go/src/cmd/dist/util.go:138 +0x148
created by main.bginit
/wrkdirs/usr/ports/lang/go/work/go/src/cmd/dist/util.go:131 +0x58

goroutine 7 [select]:
main.bghelper()
/wrkdirs/usr/ports/lang/go/work/go/src/cmd/dist/util.go:138 +0x148
created by main.bginit
/wrkdirs/usr/ports/lang/go/work/go/src/cmd/dist/util.go:131 +0x58

goroutine 8 [select]:
main.bghelper()
/wrkdirs/usr/ports/lang/go/work/go/src/cmd/dist/util.go:138 +0x148
created by main.bginit
/wrkdirs/usr/ports/lang/go/work/go/src/cmd/dist/util.go:131 +0x58

goroutine 9 [select]:
main.bghelper()
/wrkdirs/usr/ports/lang/go/work/go/src/cmd/dist/util.go:138 +0x148
created by main.bginit
/wrkdirs/usr/ports/lang/go/work/go/src/cmd/dist/util.go:131 +0x58

goroutine 26 [chan receive]:
main.install(0xc20a8222e1, 0x13)
/wrkdirs/usr/ports/lang/go/work/go/src/cmd/dist/build.go:567 +0x55
main.runInstall(0x5b4b10, 0x6, 0xc20803a180)
/wrkdirs/usr/ports/lang/go/work/go/src/cmd/dist/build.go:760 +0x24b3
created by main.startInstall
/wrkdirs/usr/ports/lang/go/work/go/src/cmd/dist/build.go:576 +0x114

goroutine 34 [chan receive]:
main.install(0xc208138a61, 0x16)
/wrkdirs/usr/ports/lang/go/work/go/src/cmd/dist/build.go:567 +0x55
main.runInstall(0xc20a8222e1, 0x13, 0xc20803a5a0)
/wrkdirs/usr/ports/lang/go/work/go/src/cmd/dist/build.go:760 +0x24b3
created by main.startInstall
/wrkdirs/usr/ports/lang/go/work/go/src/cmd/dist/build.go:576 +0x114

goroutine 36 [chan receive]:
main.install(0xc208138101, 0x14)
/wrkdirs/usr/ports/lang/go/work/go/src/cmd/dist/build.go:567 +0x55
main.runInstall(0xc20a822321, 0x15, 0xc20803a660)
/wrkdirs/usr/ports/lang/go/work/go/src/cmd/dist/build.go:760 +0x24b3
created by main.startInstall
/wrkdirs/usr/ports/lang/go/work/go/src/cmd/dist/build.go:576 +0x114

goroutine 38 [chan receive]:
main.install(0xc2081383a1, 0x14)
/wrkdirs/usr/ports/lang/go/work/go/src/cmd/dist/build.go:567 +0x55
main.runInstall(0xc20a822361, 0x16, 0xc20803a720)
/wrkdirs/usr/ports/lang/go/work/go/src/cmd/dist/build.go:760 +0x24b3
created by main.startInstall
/wrkdirs/usr/ports/lang/go/work/go/src/cmd/dist/build.go:576 +0x114

goroutine 39 [runnable]:
syscall.Syscall(0x3, 0x4, 0xc20a737348, 0x8, 0x0, 0x8, 0x0)
/usr/local/go14/src/syscall/asm_freebsd_amd64.s:21 +0x5
syscall.readlen(0x4, 0xc20a737348, 0x8, 0x9, 0x0, 0x0)
/usr/local/go14/src/syscall/zsyscall_freebsd_amd64.go:1298 +0x57
syscall.forkExec(0xc20a84a050, 0x41, 0xc20a768000, 0x8, 0x8, 0xc20a737550, 0x13598, 0x0, 0x0)
/usr/local/go14/src/syscall/exec_unix.go:200 +0x5d0
syscall.StartProcess(0xc20a84a050, 0x41, 0xc20a768000, 0x8, 0x8, 0xc20a737550, 0x2, 0x4, 0x0, 0x0)
/usr/local/go14/src/syscall/exec_unix.go:238 +0x72
os.startProcess(0xc20a84a050, 0x41, 0xc20a768000, 0x8, 0x8, 0xc20a84a0a0, 0x58ff20, 0x0, 0x0)
/usr/local/go14/src/os/exec_posix.go:45 +0x47b
os.StartProcess(0xc20a84a050, 0x41, 0xc20a768000, 0x8, 0x8, 0xc20a84a0a0, 0x2, 0x0, 0x0)
/usr/local/go14/src/os/doc.go:24 +0x72
os/exec.(*Cmd).Start(0xc20a802000, 0x0, 0x0)
/usr/local/go14/src/os/exec/exec.go:316 +0x92e
os/exec.(*Cmd).Run(0xc20a802000, 0x0, 0x0)
/usr/local/go14/src/os/exec/exec.go:243 +0x3a
main.run(0xc20a80a7c0, 0x3a, 0x3, 0xc20a6321c0, 0x8, 0xe, 0x0, 0x0)
/wrkdirs/usr/ports/lang/go/work/go/src/cmd/dist/util.go:89 +0x328
main.runInstall(0xc20a822381, 0x13, 0xc20803a780)
/wrkdirs/usr/ports/lang/go/work/go/src/cmd/dist/build.go:795 +0x340b
created by main.startInstall
/wrkdirs/usr/ports/lang/go/work/go/src/cmd/dist/build.go:576 +0x114

goroutine 40 [runnable]:
syscall.Syscall(0x5, 0xc208032550, 0x100000, 0x0, 0x5, 0x0, 0x0)
/usr/local/go14/src/syscall/asm_freebsd_amd64.s:21 +0x5
syscall.Open(0x5bc610, 0x9, 0x100000, 0xc200000000, 0x0, 0x0, 0x0)
/usr/local/go14/src/syscall/zsyscall_freebsd_amd64.go:819 +0xaa
os.OpenFile(0x5bc610, 0x9, 0x0, 0x0, 0x59f420, 0x0, 0x0)
/usr/local/go14/src/os/file_unix.go:78 +0x6f
os.Open(0x5bc610, 0x9, 0x80067f840, 0x0, 0x0)
/usr/local/go14/src/os/file.go:239 +0x55
os/exec.(*Cmd).stdin(0xc20a802280, 0x0, 0x0, 0x0)
/usr/local/go14/src/os/exec/exec.go:159 +0xab
os/exec.(*Cmd).Start(0xc20a802280, 0x0, 0x0)
/usr/local/go14/src/os/exec/exec.go:300 +0x345
os/exec.(*Cmd).Run(0xc20a802280, 0x0, 0x0)
/usr/local/go14/src/os/exec/exec.go:243 +0x3a
main.run(0xc20a80aa40, 0x3d, 0x3, 0xc20a6322a0, 0x8, 0xe, 0x0, 0x0)
/wrkdirs/usr/ports/lang/go/work/go/src/cmd/dist/util.go:89 +0x328
main.runInstall(0xc20a8223a1, 0x16, 0xc20803a7e0)
/wrkdirs/usr/ports/lang/go/work/go/src/cmd/dist/build.go:795 +0x340b
created by main.startInstall
/wrkdirs/usr/ports/lang/go/work/go/src/cmd/dist/build.go:576 +0x114

goroutine 41 [chan receive]:
main.install(0xc2081387c1, 0x14)
/wrkdirs/usr/ports/lang/go/work/go/src/cmd/dist/build.go:567 +0x55
main.runInstall(0xc20a8223c1, 0x18, 0xc20803a840)
/wrkdirs/usr/ports/lang/go/work/go/src/cmd/dist/build.go:760 +0x24b3
created by main.startInstall
/wrkdirs/usr/ports/lang/go/work/go/src/cmd/dist/build.go:576 +0x114

goroutine 42 [chan receive]:
main.install(0xc208138aa1, 0x14)
/wrkdirs/usr/ports/lang/go/work/go/src/cmd/dist/build.go:567 +0x55
main.runInstall(0xc20a8223e1, 0x13, 0xc20803a8a0)
/wrkdirs/usr/ports/lang/go/work/go/src/cmd/dist/build.go:760 +0x24b3
created by main.startInstall
/wrkdirs/usr/ports/lang/go/work/go/src/cmd/dist/build.go:576 +0x114

goroutine 44 [chan receive]:
main.install(0xc208138fa1, 0x14)
/wrkdirs/usr/ports/lang/go/work/go/src/cmd/dist/build.go:567 +0x55
main.runInstall(0xc20a822421, 0x14, 0xc20803a960)
/wrkdirs/usr/ports/lang/go/work/go/src/cmd/dist/build.go:760 +0x24b3
created by main.startInstall
/wrkdirs/usr/ports/lang/go/work/go/src/cmd/dist/build.go:576 +0x114

goroutine 45 [chan receive]:
main.install(0xc2081390e1, 0x14)
/wrkdirs/usr/ports/lang/go/work/go/src/cmd/dist/build.go:567 +0x55
main.runInstall(0xc20a822441, 0x13, 0xc20803a9c0)
/wrkdirs/usr/ports/lang/go/work/go/src/cmd/dist/build.go:760 +0x24b3
created by main.startInstall
/wrkdirs/usr/ports/lang/go/work/go/src/cmd/dist/build.go:576 +0x114

goroutine 46 [chan receive]:
main.install(0xc208139341, 0x14)
/wrkdirs/usr/ports/lang/go/work/go/src/cmd/dist/build.go:567 +0x55
main.runInstall(0xc20a822461, 0x14, 0xc20803aa20)
/wrkdirs/usr/ports/lang/go/work/go/src/cmd/dist/build.go:760 +0x24b3
created by main.startInstall
/wrkdirs/usr/ports/lang/go/work/go/src/cmd/dist/build.go:576 +0x114

goroutine 49 [chan receive]:
main.install(0xc2081397a1, 0x14)
/wrkdirs/usr/ports/lang/go/work/go/src/cmd/dist/build.go:567 +0x55
main.runInstall(0xc20a8224c1, 0x13, 0xc20803ab40)
/wrkdirs/usr/ports/lang/go/work/go/src/cmd/dist/build.go:760 +0x24b3
created by main.startInstall
/wrkdirs/usr/ports/lang/go/work/go/src/cmd/dist/build.go:576 +0x114

goroutine 50 [syscall]:
syscall.Syscall(0x3, 0x7, 0xc20a447348, 0x8, 0x0, 0x0, 0x0)
/usr/local/go14/src/syscall/asm_freebsd_amd64.s:21 +0x5
syscall.readlen(0x7, 0xc20a447348, 0x8, 0x10, 0x0, 0x0)
/usr/local/go14/src/syscall/zsyscall_freebsd_amd64.go:1298 +0x57
syscall.forkExec(0xc20a84a370, 0x41, 0xc20a7b44b0, 0xf, 0xf, 0xc20a447550, 0x13dd8, 0x0, 0x0)
/usr/local/go14/src/syscall/exec_unix.go:200 +0x5d0
syscall.StartProcess(0xc20a84a370, 0x41, 0xc20a7b44b0, 0xf, 0xf, 0xc20a447550, 0x2, 0x4, 0x0, 0x0)
/usr/local/go14/src/syscall/exec_unix.go:238 +0x72
os.startProcess(0xc20a84a370, 0x41, 0xc20a7b44b0, 0xf, 0xf, 0xc20a84a3c0, 0x58ff20, 0x0, 0x0)
/usr/local/go14/src/os/exec_posix.go:45 +0x47b
os.StartProcess(0xc20a84a370, 0x41, 0xc20a7b44b0, 0xf, 0xf, 0xc20a84a3c0, 0x2, 0x0, 0x0)
/usr/local/go14/src/os/doc.go:24 +0x72
os/exec.(*Cmd).Start(0xc20a8023c0, 0x0, 0x0)
/usr/local/go14/src/os/exec/exec.go:316 +0x92e
os/exec.(*Cmd).Run(0xc20a8023c0, 0x0, 0x0)
/usr/local/go14/src/os/exec/exec.go:243 +0x3a
main.run(0xc20819e340, 0x3b, 0x3, 0xc20a7b43c0, 0xf, 0xf, 0x0, 0x0)
/wrkdirs/usr/ports/lang/go/work/go/src/cmd/dist/util.go:89 +0x328
main.runInstall(0xc20a8224e1, 0x14, 0xc20803aba0)
/wrkdirs/usr/ports/lang/go/work/go/src/cmd/dist/build.go:795 +0x340b
created by main.startInstall
/wrkdirs/usr/ports/lang/go/work/go/src/cmd/dist/build.go:576 +0x114

[ianlancetaylor]

This looks like a problem running Go 1.4 to build Go 1.10.2. Precisely where did you get the Go 1.4 sources that you are using?

[rhavenn]

Go 1.4 is built as a dependency by the FreeBSD build structure. It pulls it as source from http://golang.org/dl/ . I can try forcing a package rebuild of the 1.4.2, but the source tar ball matches the SHA hash from there.

[rhavenn]

Sorry, 1.4.3 is what it's using.

[rhavenn]

Not that it's the best example, but inside the jail I was able to use go14/bin/go to run, build and execute a "hello world", so the 1.4.3 binary is working at a very basic level.

[ianlancetaylor]

I don't know that this is the problem, but for your 1.4 sources you should currently be using go1.4-bootstrap-20171003.tar.gz, not the 1.4.3 release. The 20171003 file includes various fixes that we've found necessary.

[rhavenn]

Yeah, I was checking on those bootstrap files. The port doesn't use them currently. I tried with the 201705 one, but now notice the docs point at 20171003. If I modify the ports to build against that I do get 1.4-bootstrap to build, but the 1.10.2 now fails. It seems to fail in different places which is wierd. I listed 3 outputs below:

Building Go cmd/dist using /usr/local/go14.
Building Go toolchain1 using /usr/local/go14.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
fatal error: fatal error: span has no free stacks
unexpected signal during runtime execution
fatal error: unlock of unlocked lock

runtime stack:
runtime.throw(0x6f82e9, 0x17)
/usr/local/go/src/runtime/panic.go:616 +0x81
runtime.stackpoolalloc(0x888401, 0xc4204e9c40)
/usr/local/go/src/runtime/stack.go:199 +0x173
runtime.stackcacherefill(0x80084e458, 0x1)
/usr/local/go/src/runtime/stack.go:260 +0x44
runtime.stackalloc(0x1000, 0x0, 0xc4204e9e30)
/usr/local/go/src/runtime/stack.go:362 +0x151
runtime.copystack(0xc420496a80, 0x1000, 0xc420255801)
/usr/local/go/src/runtime/stack.go:840 +0x7d
runtime.newstack()
/usr/local/go/src/runtime/stack.go:1063 +0x30f
runtime.morestack()
/usr/local/go/src/runtime/asm_amd64.s:480 +0x89

goroutine 36 [copystack]:
cmd/go/internal/work.(*Builder).build(0xc420125900, 0xc4201db680, 0x5a2463, 0xc420125970)
/usr/local/go/src/cmd/go/internal/work/exec.go:300 +0x51cf fp=0xc420033e70 sp=0xc420033e68 pc=0x5e98ef
cmd/go/internal/work.(*Builder).Do.func1(0xc4201db680)
/usr/local/go/src/cmd/go/internal/work/exec.go:106 +0x72 fp=0xc420033ef8 sp=0xc420033e70 pc=0x611f42
cmd/go/internal/work.(*Builder).Do.func2(0xc4201a5040, 0xc420125900, 0xc420569ca0)
/usr/local/go/src/cmd/go/internal/work/exec.go:164 +0xbb fp=0xc420033fc8 sp=0xc420033ef8 pc=0x61222b
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc420033fd0 sp=0xc420033fc8 pc=0x4535f1
created by cmd/go/internal/work.(*Builder).Do
/usr/local/go/src/cmd/go/internal/work/exec.go:151 +0x38c

goroutine 1 [semacquire]:
sync.runtime_Semacquire(0xc4201a504c)
/usr/local/go/src/runtime/sema.go:56 +0x39
sync.(*WaitGroup).Wait(0xc4201a5040)
/usr/local/go/src/sync/waitgroup.go:129 +0x72
cmd/go/internal/work.(*Builder).Do(0xc420125900, 0xc4205e6000)
/usr/local/go/src/cmd/go/internal/work/exec.go:173 +0x3b0
cmd/go/internal/work.InstallPackages(0xc42000e0e0, 0x4, 0x4, 0x0)
/usr/local/go/src/cmd/go/internal/work/build.go:482 +0xb25
cmd/go/internal/work.runInstall(0x864dc0, 0xc42000e0e0, 0x4, 0x4)
/usr/local/go/src/cmd/go/internal/work/build.go:413 +0x49
main.main()
/usr/local/go/src/cmd/go/main.go:140 +0x7e1

goroutine 5 [syscall]:
os/signal.signal_recv(0x0)
/usr/local/go/src/runtime/sigqueue.go:139 +0xa6
os/signal.loop()
/usr/local/go/src/os/signal/signal_unix.go:22 +0x22
created by os/signal.init.0
/usr/local/go/src/os/signal/signal_unix.go:28 +0x41

goroutine 33 [runnable]:
syscall.Syscall(0x3, 0xd, 0xc420042860, 0x8, 0x0, 0x8, 0x0)
/usr/local/go/src/syscall/asm_unix_amd64.s:19 +0x5
syscall.readlen(0xd, 0xc420042860, 0x8, 0x3, 0xc4202fe000, 0x3f)
/usr/local/go/src/syscall/zsyscall_freebsd_amd64.go:1263 +0x56
syscall.forkExec(0xc420606000, 0x41, 0xc42060a020, 0x2, 0x2, 0xc4200429d8, 0x20, 0x0, 0xc420490020)
/usr/local/go/src/syscall/exec_unix.go:203 +0x419
syscall.StartProcess(0xc420606000, 0x41, 0xc42060a020, 0x2, 0x2, 0xc4200429d8, 0x2, 0x4, 0x0, 0x0)
/usr/local/go/src/syscall/exec_unix.go:241 +0x64
os.startProcess(0xc420606000, 0x41, 0xc42060a020, 0x2, 0x2, 0xc420042b80, 0xc42060e800, 0x3e, 0x3e)
/usr/local/go/src/os/exec_posix.go:46 +0x1f1
os.StartProcess(0xc420606000, 0x41, 0xc42060a020, 0x2, 0x2, 0xc420042b80, 0x0, 0x0, 0xc420042b20)
/usr/local/go/src/os/exec.go:102 +0x7c
os/exec.(*Cmd).Start(0xc42060c000, 0x0, 0x68)
/usr/local/go/src/os/exec/exec.go:379 +0x4e6
os/exec.(*Cmd).Run(0xc42060c000, 0xc4200b6150, 0xc42060e000)
/usr/local/go/src/os/exec/exec.go:302 +0x2b
cmd/go/internal/work.(*Builder).toolID(0xc420125900, 0x6f19bc, 0x7, 0x2c, 0xc420043380)
/usr/local/go/src/cmd/go/internal/work/buildid.go:183 +0x321
cmd/go/internal/work.(*Builder).buildActionID(0xc420125900, 0xc4201db400, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/cmd/go/internal/work/exec.go:219 +0xf69
cmd/go/internal/work.(*Builder).build(0xc420125900, 0xc4201db400, 0x0, 0x0)
/usr/local/go/src/cmd/go/internal/work/exec.go:304 +0xb0
cmd/go/internal/work.(*Builder).Do.func1(0xc4201db400)
/usr/local/go/src/cmd/go/internal/work/exec.go:106 +0x72
cmd/go/internal/work.(*Builder).Do.func2(0xc4201a5040, 0xc420125900, 0xc420569ca0)
/usr/local/go/src/cmd/go/internal/work/exec.go:164 +0xbb
created by cmd/go/internal/work.(*Builder).Do
/usr/local/go/src/cmd/go/internal/work/exec.go:151 +0x38c

goroutine 34 [syscall]:
syscall.Syscall(0xbc, 0xc42030a050, 0xc42030c038, 0x0, 0x0, 0xc42030a000, 0xc4204e4ab0)
/usr/local/go/src/syscall/asm_unix_amd64.s:19 +0x5
syscall.Stat(0xc42030a000, 0x41, 0xc42030c038, 0x41, 0xc42030a000)
/usr/local/go/src/syscall/zsyscall_freebsd_amd64.go:1102 +0x97
os.statNolog(0xc42030a000, 0x41, 0x2, 0x2, 0xc42030a000, 0x41)
/usr/local/go/src/os/stat_unix.go:31 +0x5c
os.Stat(0xc42030a000, 0x41, 0x2, 0xc42030a000, 0x41, 0x646cac1a59ef0a01)
/usr/local/go/src/os/stat.go:13 +0x4d
cmd/go/internal/base.Tool(0x6f19bc, 0x7, 0x6f19bc, 0x7)
/usr/local/go/src/cmd/go/internal/base/tool.go:38 +0xe4
cmd/go/internal/work.(*Builder).toolID(0xc420125900, 0x6f19bc, 0x7, 0x2c, 0xc4204e5380)
/usr/local/go/src/cmd/go/internal/work/buildid.go:177 +0xe5
cmd/go/internal/work.(*Builder).buildActionID(0xc420125900, 0xc4201db680, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/cmd/go/internal/work/exec.go:219 +0xf69
cmd/go/internal/work.(*Builder).build(0xc420125900, 0xc4201db680, 0x0, 0x0)
/usr/local/go/src/cmd/go/internal/work/exec.go:304 +0xb0
cmd/go/internal/work.(*Builder).Do.func1(0xc4201db680)
/usr/local/go/src/cmd/go/internal/work/exec.go:106 +0x72
cmd/go/internal/work.(*Builder).Do.func2(0xc4201a5040, 0xc420125900, 0xc420569ca0)
/usr/local/go/src/cmd/go/internal/work/exec.go:164 +0xbb
created by cmd/go/internal/work.(*Builder).Do
/usr/local/go/src/cmd/go/internal/work/exec.go:151 +0x38c

goroutine 35 [runnable]:
syscall.Syscall(0xbc, 0xc4206061e0, 0xc4206080e8, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/syscall/asm_unix_amd64.s:19 +0x5
syscall.Stat(0xc420606190, 0x41, 0xc4206080e8, 0x41, 0xc420606190)
/usr/local/go/src/syscall/zsyscall_freebsd_amd64.go:1102 +0x97
os.statNolog(0xc420606190, 0x41, 0x2, 0x2, 0xc420606190, 0x41)
/usr/local/go/src/os/stat_unix.go:31 +0x5c
os.Stat(0xc420606190, 0x41, 0x2, 0xc420606190, 0x41, 0xb271e0c58477efe2)
/usr/local/go/src/os/stat.go:13 +0x4d
cmd/go/internal/base.Tool(0x6f19bc, 0x7, 0x6f19bc, 0x7)
/usr/local/go/src/cmd/go/internal/base/tool.go:38 +0xe4
cmd/go/internal/work.(*Builder).toolID(0xc420125900, 0x6f19bc, 0x7, 0x2c, 0xc420077380)
/usr/local/go/src/cmd/go/internal/work/buildid.go:177 +0xe5
cmd/go/internal/work.(*Builder).buildActionID(0xc420125900, 0xc4201dbcc0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/cmd/go/internal/work/exec.go:219 +0xf69
cmd/go/internal/work.(*Builder).build(0xc420125900, 0xc4201dbcc0, 0x0, 0x0)
/usr/local/go/src/cmd/go/internal/work/exec.go:304 +0xb0
cmd/go/internal/work.(*Builder).Do.func1(0xc4201dbcc0)
/usr/local/go/src/cmd/go/internal/work/exec.go:106 +0x72
cmd/go/internal/work.(*Builder).Do.func2(0xc4201a5040, 0xc420125900, 0xc420569ca0)
/usr/local/go/src/cmd/go/internal/work/exec.go:164 +0xbb
created by cmd/go/internal/work.(*Builder).Do
/usr/local/go/src/cmd/go/internal/work/exec.go:151 +0x38c
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x42d7f8]

runtime stack:
runtime.throw(0x6f8414, 0x17)
/usr/local/go/src/runtime/panic.go:616 +0x81
runtime.unlock(0x888458)
/usr/local/go/src/runtime/lock_futex.go:109 +0xbb
runtime.printunlock()
/usr/local/go/src/runtime/print.go:80 +0x59
runtime.throw(0x6fee8c, 0x2a)
/usr/local/go/src/runtime/panic.go:610 +0x4e
runtime.sigpanic()
/usr/local/go/src/runtime/signal_unix.go:372 +0x28e
runtime.runSafePointFn()
/usr/local/go/src/runtime/proc.go:1459 +0x58
runtime.systemstack(0x0)
/usr/local/go/src/runtime/asm_amd64.s:409 +0x79
runtime.mstart()
/usr/local/go/src/runtime/proc.go:1175
go tool dist: FAILED: /wrkdirs/usr/ports/lang/go/work/go/pkg/tool/freebsd_amd64/go_bootstrap install -gcflags=all= -ldflags=all= -i cmd/asm cmd/cgo cmd/compile cmd/link: exit status 2
*** Error code 2

Stop.

=====================
If it run the all.bash manually I get an error as well:

Building Go cmd/dist using /usr/local/go14/.
Building Go toolchain1 using /usr/local/go14/.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Segmentation fault (core dumped)

============================
or the next time I run it:

-ap: not found
go: not found
Building Go cmd/dist using /usr/local/go14/.
Building Go toolchain1 using /usr/local/go14/.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
acquirep: p->m=842356128768(8) p->status=1
fatal error: acquirep: invalid p state
runtime: newstack called from g=0xc4205de180
m=0xc420032400 m->curg=0x0 m->g0=0xc420000900 m->gsignal=0xc420000780
runtime: unexpected return pc for syscall.Syscall6 called from 0x7141a0
stack: frame={sp:0xc420042a38, fp:0xc420042a40} stack=[0xc420042000,0xc420044000)
000000c420042938: 0000000000487b3c <internal/poll.(*FD).destroy+124> 000000000000000e
000000c420042948: 0000000800a50b18 000000000041125f <runtime.(*mspan).nextFreeIndex+111>
000000c420042958: 0000000800a5abc0 0000000000000008
000000c420042968: 0000000000000100 0000000000000040
000000c420042978: 000000c4200429b0 000000c4200429d8
000000c420042988: 000000000040f386 <runtime.(*mcache).nextFree+198> 0000000800a5abc0
000000c420042998: 0000000000000075 000000c4200429d0
000000c4200429a8: 0000000000431569 <runtime.exitsyscall+425> 0000000000000006
000000c4200429b8: 0000000800a5abc0 000000000044dee0 <runtime.(*mcache).nextFree.func1+0>
000000c4200429c8: 000000080084f000 000000c420042a06
000000c4200429d8: 000000c420042a78 000000000040f819 <runtime.mallocgc+761>
000000c4200429e8: 000000c42043eea0 0000000000000020
000000c4200429f8: 0000000000000018 00000000006c1200
000000c420042a08: 000000c4203f4001 0001000000000000
000000c420042a18: 0000000000000018 0000000000000018
000000c420042a28: 000000c42043eea0 000000000042a63a <runtime.gopark+282>
000000c420042a38: <00000000007141a0 >0000000000000214
000000c420042a48: 000000c420042a88 000000000042a6ee <runtime.goparkunlock+94>
000000c420042a58: 00000000007141a8 000000c4202841d8
000000c420042a68: 00000000006f3bac 000000000000000c
000000c420042a78: 0000000000000017 0000000000000003
000000c420042a88: 000000c420042b20 0000000000403e32 <runtime.chanrecv+754>
000000c420042a98: 000000c4202841d8 00000000006f3bac
000000c420042aa8: 000000000000000c 000000000048d517 <os.(*Process).wait+119>
000000c420042ab8: 0000000000000003 000000c42043eea0
000000c420042ac8: 0000000000000000 000000c4204e1c80
000000c420042ad8: 000000c4205ba360 000000c4205de180
000000c420042ae8: 0000000000000000 000000c4205de208
000000c420042af8: 000000c4204e1cd8 000000c4205de2b8
000000c420042b08: 000000c4202841d8 000000c4205ce254
000000c420042b18: 000000c4203f4000 000000c420042b50
000000c420042b28: 0000000000403aeb <runtime.chanrecv1+43> 000000c420284180
000000c420042b38: 000000c420042bb8
syscall.Syscall6(0x214, 0xc420042a88, 0x42a6ee, 0x7141a8, 0xc4202841d8, 0x6f3bac, 0xc, 0x17, 0x3, 0xc420042b20)
/wrkdirs/usr/ports/lang/go/work/go/src/syscall/asm_unix_amd64.sfatal error: :42 +0x5
created by cmd/go/internal/work.(*Builder).Do
/wrkdirs/usr/ports/lang/go/work/go/src/cmd/go/internal/work/exec.go:151 +0x38c

unexpected signal during runtime executionpanic during panic
[signal SIGSEGV: segmentation violationfatal error: unexpected signal during runtime execution
stack trace unavailable
go tool dist: FAILED: /wrkdirs/usr/ports/lang/go/work/go/pkg/tool/freebsd_amd64/go_bootstrap install -gcflags=all= -ldflags=all= -i cmd/asm cmd/cgo cmd/compile cmd/link: exit status 4

[ianlancetaylor]

No clue, sorry. I think you are going to have to debug this. Happy to answer questions.

[bradfitz]

Our FreeBSD port was pretty stable, I thought, so that runtime error looks concerning. @aclements, @randall77, see anything interesting there?

[rhavenn]

I even pulled a SVN 11.1-RELEASE (no patches) into a test poudriere jail and it stills falls over randomly. It will either just spin on the ./cmd/dist/dist bootstrap -a for hours or crash in random places. At this point I'm thinking it's something Amazon did to the VM or host patch problems vs. the jail itself. It works perfectly fine over 3 different poudriere jails on the physical server I have and has been stable for months. I'm also having trouble with rust compiling on my AWS instance vs. my physical box, so it might not just be Go.

11.1-RELEASE-p6 worked fine for Go and Rust on 04/06 and then I updated to -p9 on 5/6 and it's been failing since for both Go and Rust.

So, either an issue with the host Spectre / Meltdown patches and AWS EC2 instances or something. I just rolled my host back to -p6 and we'll see what happens.

[rhavenn]

After rolling my host back from 11.1-p10 to 11.1-p6 I've successfully built lang/go 3 times in the 11.1-RELEASE temp jail I created. So, I'm guessing something in the Spectre / Meltdown patches is causing it.

[rhavenn]

Since this doesn't seem to be a Go issue I'll close this. Submitted a FreeBSD bug:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=228620