runtime: test failures on linux-arm, freebsd-386-12_0, and linux-s390x-ibm at CL 171758

[bcmills]

As of CL 171758, the linux-arm builder is consistently failing the ../test phase in fixedbugs/issue27518b.go, with the following error:

##### ../test
# go run run.go -- fixedbugs/issue27518b.go
exit status 1
runtime: nelems=128 nalloc=2 previous allocCount=1 nfreed=65535
fatal error: sweep increased allocation count

goroutine 1 [running]:
runtime.throw(0x88eac, 0x20)
	/workdir/go/src/runtime/panic.go:758 +0x5c fp=0x426690 sp=0x42667c pc=0x37044
runtime.(*mspan).sweep(0xb6f75878, 0x0, 0x1f)
	/workdir/go/src/runtime/mgcsweep.go:328 +0x888 fp=0x42671c sp=0x426690 pc=0x2df14
runtime.sweepone(0xd1e3c)
	/workdir/go/src/runtime/mgcsweep.go:136 +0x2d4 fp=0x426764 sp=0x42671c pc=0x2d3b8
runtime.GC()
	/workdir/go/src/runtime/mgc.go:1076 +0x1c0 fp=0x426784 sp=0x426764 pc=0x24014
main.gc(0x426700)
	/workdir/go/test/fixedbugs/issue27518b.go:36 +0x14 fp=0x426788 sp=0x426784 pc=0x61af0
main.main()
	/workdir/go/test/fixedbugs/issue27518b.go:53 +0x84 fp=0x4267a4 sp=0x426788 pc=0x61bf4
runtime.main()
	/workdir/go/src/runtime/proc.go:203 +0x208 fp=0x4267e4 sp=0x4267a4 pc=0x389f0
runtime.goexit()
	/workdir/go/src/runtime/asm_arm.s:868 +0x4 fp=0x4267e4 sp=0x4267e4 pc=0x5d8f0
exit status 2

FAIL	fixedbugs/issue27518b.go	1.908s

(https://build.golang.org/log/fc5cdd37ce154f561cb55d74243500c9432365f0)

It appears that there may be a bad interaction between stack-allocated defer records and the garbage collector.

CC @randall77 @aclements @mdempsky @josharian @mknyszek

[bcmills]

The freebsd-386-12_0 builder started failing consistently at the same CL, but in the syscall test and with a totally different failure mode.

--- FAIL: TestGetdirentries (0.04s)
    --- FAIL: TestGetdirentries/n=10 (0.00s)
        getdirentries_test.go:83: names don't match
             names: ["." ".." "file000" "file001" "file002" "file003" "file004" "file005" "file006" "file007" "file008" "file009"]
            names2: []
    --- FAIL: TestGetdirentries/n=1000 (0.04s)
        getdirentries_test.go:83: names don't match
             names: ["." ".." "file000" "file001" "file002" "file003" "file004" "file005" "file006" "file007" "file008" "file009" "file010" "file011" "file012" "file013" "file014" "file015" "file016" "file017" "file018" "file019" "file020" "file021" "file022" "file023" "file024" "file025" "file026" "file027" "file028" "file029" "file030" "file031" "file032" "file033" "file034" "file035" "file036" "file037" "file038" "file039" "file040" "file041" "file042" "file043" "file044" "file045" "file046" "file047" "file048" "file049" "file050" "file051" "file052" "file053" "file054" "file055" "file056" "file057" "file058" "file059" "file060" "file061" "file062" "file063" "file064" "file065" "file066" "file067" "file068" "file069" "file070" "file071" "file072" "file073" "file074" "file075" "file076" "file077" "file078" "file079" "file080" "file081" "file082" "file083" "file084" "file085" "file086" "file087" "file088" "file089" "file090" "file091" "file092" "file093" "file094" "file095" "file096" "file097" "file098" "file099" "file100" "file101" "file102" "file103" "file104" "file105" "file106" "file107" "file108" "file109" "file110" "file111" "file112" "file113" "file114" "file115" "file116" "file117" "file118" "file119" "file120" "file121" "file122" "file123" "file124" "file125" "file126" "file127" "file128" "file129" "file130" "file131" "file132" "file133" "file134" "file135" "file136" "file137" "file138" "file139" "file140" "file141" "file142" "file143" "file144" "file145" "file146" "file147" "file148" "file149" "file150" "file151" "file152" "file153" "file154" "file155" "file156" "file157" "file158" "file159" "file160" "file161" "file162" "file163" "file164" "file165" "file166" "file167" "file168" "file169" "file170" "file171" "file172" "file173" "file174" "file175" "file176" "file177" "file178" "file179" "file180" "file181" "file182" "file183" "file184" "file185" "file186" "file187" "file188" "file189" "file190" "file191" "file192" "file193" "file194" "file195" "file196" "file197" "file198" "file199" "file200" "file201" "file202" "file203" "file204" "file205" "file206" "file207" "file208" "file209" "file210" "file211" "file212" "file213" "file214" "file215" "file216" "file217" "file218" "file219" "file220" "file221" "file222" "file223" "file224" "file225" "file226" "file227" "file228" "file229" "file230" "file231" "file232" "file233" "file234" "file235" "file236" "file237" "file238" "file239" "file240" "file241" "file242" "file243" "file244" "file245" "file246" "file247" "file248" "file249" "file250" "file251" "file252" "file253" "file254" "file255" "file256" "file257" "file258" "file259" "file260" "file261" "file262" "file263" "file264" "file265" "file266" "file267" "file268" "file269" "file270" "file271" "file272" "file273" "file274" "file275" "file276" "file277" "file278" "file279" "file280" "file281" "file282" "file283" "file284" "file285" "file286" "file287" "file288" "file289" "file290" "file291" "file292" "file293" "file294" "file295" "file296" "file297" "file298" "file299" "file300" "file301" "file302" "file303" "file304" "file305" "file306" "file307" "file308" "file309" "file310" "file311" "file312" "file313" "file314" "file315" "file316" "file317" "file318" "file319" "file320" "file321" "file322" "file323" "file324" "file325" "file326" "file327" "file328" "file329" "file330" "file331" "file332" "file333" "file334" "file335" "file336" "file337" "file338" "file339" "file340" "file341" "file342" "file343" "file344" "file345" "file346" "file347" "file348" "file349" "file350" "file351" "file352" "file353" "file354" "file355" "file356" "file357" "file358" "file359" "file360" "file361" "file362" "file363" "file364" "file365" "file366" "file367" "file368" "file369" "file370" "file371" "file372" "file373" "file374" "file375" "file376" "file377" "file378" "file379" "file380" "file381" "file382" "file383" "file384" "file385" "file386" "file387" "file388" "file389" "file390" "file391" "file392" "file393" "file394" "file395" "file396" "file397" "file398" "file399" "file400" "file401" "file402" "file403" "file404" "file405" "file406" "file407" "file408" "file409" "file410" "file411" "file412" "file413" "file414" "file415" "file416" "file417" "file418" "file419" "file420" "file421" "file422" "file423" "file424" "file425" "file426" "file427" "file428" "file429" "file430" "file431" "file432" "file433" "file434" "file435" "file436" "file437" "file438" "file439" "file440" "file441" "file442" "file443" "file444" "file445" "file446" "file447" "file448" "file449" "file450" "file451" "file452" "file453" "file454" "file455" "file456" "file457" "file458" "file459" "file460" "file461" "file462" "file463" "file464" "file465" "file466" "file467" "file468" "file469" "file470" "file471" "file472" "file473" "file474" "file475" "file476" "file477" "file478" "file479" "file480" "file481" "file482" "file483" "file484" "file485" "file486" "file487" "file488" "file489" "file490" "file491" "file492" "file493" "file494" "file495" "file496" "file497" "file498" "file499" "file500" "file501" "file502" "file503" "file504" "file505" "file506" "file507" "file508" "file509" "file510" "file511" "file512" "file513" "file514" "file515" "file516" "file517" "file518" "file519" "file520" "file521" "file522" "file523" "file524" "file525" "file526" "file527" "file528" "file529" "file530" "file531" "file532" "file533" "file534" "file535" "file536" "file537" "file538" "file539" "file540" "file541" "file542" "file543" "file544" "file545" "file546" "file547" "file548" "file549" "file550" "file551" "file552" "file553" "file554" "file555" "file556" "file557" "file558" "file559" "file560" "file561" "file562" "file563" "file564" "file565" "file566" "file567" "file568" "file569" "file570" "file571" "file572" "file573" "file574" "file575" "file576" "file577" "file578" "file579" "file580" "file581" "file582" "file583" "file584" "file585" "file586" "file587" "file588" "file589" "file590" "file591" "file592" "file593" "file594" "file595" "file596" "file597" "file598" "file599" "file600" "file601" "file602" "file603" "file604" "file605" "file606" "file607" "file608" "file609" "file610" "file611" "file612" "file613" "file614" "file615" "file616" "file617" "file618" "file619" "file620" "file621" "file622" "file623" "file624" "file625" "file626" "file627" "file628" "file629" "file630" "file631" "file632" "file633" "file634" "file635" "file636" "file637" "file638" "file639" "file640" "file641" "file642" "file643" "file644" "file645" "file646" "file647" "file648" "file649" "file650" "file651" "file652" "file653" "file654" "file655" "file656" "file657" "file658" "file659" "file660" "file661" "file662" "file663" "file664" "file665" "file666" "file667" "file668" "file669" "file670" "file671" "file672" "file673" "file674" "file675" "file676" "file677" "file678" "file679" "file680" "file681" "file682" "file683" "file684" "file685" "file686" "file687" "file688" "file689" "file690" "file691" "file692" "file693" "file694" "file695" "file696" "file697" "file698" "file699" "file700" "file701" "file702" "file703" "file704" "file705" "file706" "file707" "file708" "file709" "file710" "file711" "file712" "file713" "file714" "file715" "file716" "file717" "file718" "file719" "file720" "file721" "file722" "file723" "file724" "file725" "file726" "file727" "file728" "file729" "file730" "file731" "file732" "file733" "file734" "file735" "file736" "file737" "file738" "file739" "file740" "file741" "file742" "file743" "file744" "file745" "file746" "file747" "file748" "file749" "file750" "file751" "file752" "file753" "file754" "file755" "file756" "file757" "file758" "file759" "file760" "file761" "file762" "file763" "file764" "file765" "file766" "file767" "file768" "file769" "file770" "file771" "file772" "file773" "file774" "file775" "file776" "file777" "file778" "file779" "file780" "file781" "file782" "file783" "file784" "file785" "file786" "file787" "file788" "file789" "file790" "file791" "file792" "file793" "file794" "file795" "file796" "file797" "file798" "file799" "file800" "file801" "file802" "file803" "file804" "file805" "file806" "file807" "file808" "file809" "file810" "file811" "file812" "file813" "file814" "file815" "file816" "file817" "file818" "file819" "file820" "file821" "file822" "file823" "file824" "file825" "file826" "file827" "file828" "file829" "file830" "file831" "file832" "file833" "file834" "file835" "file836" "file837" "file838" "file839" "file840" "file841" "file842" "file843" "file844" "file845" "file846" "file847" "file848" "file849" "file850" "file851" "file852" "file853" "file854" "file855" "file856" "file857" "file858" "file859" "file860" "file861" "file862" "file863" "file864" "file865" "file866" "file867" "file868" "file869" "file870" "file871" "file872" "file873" "file874" "file875" "file876" "file877" "file878" "file879" "file880" "file881" "file882" "file883" "file884" "file885" "file886" "file887" "file888" "file889" "file890" "file891" "file892" "file893" "file894" "file895" "file896" "file897" "file898" "file899" "file900" "file901" "file902" "file903" "file904" "file905" "file906" "file907" "file908" "file909" "file910" "file911" "file912" "file913" "file914" "file915" "file916" "file917" "file918" "file919" "file920" "file921" "file922" "file923" "file924" "file925" "file926" "file927" "file928" "file929" "file930" "file931" "file932" "file933" "file934" "file935" "file936" "file937" "file938" "file939" "file940" "file941" "file942" "file943" "file944" "file945" "file946" "file947" "file948" "file949" "file950" "file951" "file952" "file953" "file954" "file955" "file956" "file957" "file958" "file959" "file960" "file961" "file962" "file963" "file964" "file965" "file966" "file967" "file968" "file969" "file970" "file971" "file972" "file973" "file974" "file975" "file976" "file977" "file978" "file979" "file980" "file981" "file982" "file983" "file984" "file985" "file986" "file987" "file988" "file989" "file990" "file991" "file992" "file993" "file994" "file995" "file996" "file997" "file998" "file999"]
            names2: []
FAIL
FAIL	syscall	0.061s

(https://build.golang.org/log/150949ad287d3008b2d92ba74e61bd5b8795f515)

[bcmills]

linux-s390x-ibm also started failing at that CL, and this failure mode is a doozy: it doesn't even make it through bootstrapping.

linux-s390x-ibm at fff4f599fe1c21e411a99de5c9b3777d06ce0ce6

:: Running /data/golang/workdir/go/src/make.bash with args ["/data/golang/workdir/go/src/make.bash"] and env ["LC_CTYPE=en_US.UTF-8" "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" "HOME=/root" "LOGNAME=root" "USER=root" "SHELL=/bin/sh" "META_BUILDLET_BINARY_URL=https://storage.googleapis.com/go-builder-data/buildlet.linux-s390x" "WORKDIR=/data/golang/workdir" "GOROOT_BOOTSTRAP=/data/golang/workdir/go1.4" "GO_BUILDER_NAME=linux-s390x-ibm" "GOROOT_BOOTSTRAP=/var/buildlet/go-linux-s390x-bootstrap" "GOBIN=" "TMPDIR=/data/golang/workdir/tmp" "GOCACHE=/data/golang/workdir/gocache"] in dir /data/golang/workdir/go/src

Building Go cmd/dist using /var/buildlet/go-linux-s390x-bootstrap.
Building Go toolchain1 using /var/buildlet/go-linux-s390x-bootstrap.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
Building Go toolchain3 using go_bootstrap and Go toolchain2.
HASH[build runtime/internal/sys]
HASH[build runtime/internal/sys]: "devel fff4f599fe1c21e411a99de5c9b3777d06ce0ce6"
HASH[build runtime/internal/sys]: "compile\n"
HASH[build runtime/internal/sys]: "goos linux goarch s390x\n"
HASH[build runtime/internal/sys]: "import \"runtime/internal/sys\"\n"
HASH[build runtime/internal/sys]: "omitdebug false standard true local false prefix \"\"\n"
HASH[build runtime/internal/sys]: "modinfo \"\"\n"
HASH[build runtime/internal/sys]: "compile Ju8I4bJGBKqsO5TJa6uN [] []\n"
HASH[build runtime/internal/sys]: "=\n"
HASH /data/golang/workdir/go/src/runtime/internal/sys/arch.go: d9b0b7e72538d421b2607acaba60ca49f20ef584b3d1d191c6729e35fbb8101d
HASH[build runtime/internal/sys]: "file arch.go 2bC35yU41CGyYHrKumDK\n"
HASH /data/golang/workdir/go/src/runtime/internal/sys/arch_s390x.go: 1860cf11a7c72ce0db5a7a73b14c52004d97c6af6d513540ece8d890e3f3ec16
HASH[build runtime/internal/sys]: "file arch_s390x.go GGDPEafHLODbWnpzsUxS\n"
HASH /data/golang/workdir/go/src/runtime/internal/sys/intrinsics.go: 8b469a461e1d983706e0b3635715ce70691adc5db7c4e067b88cc59f40cd66f4
HASH[build runtime/internal/sys]: "file intrinsics.go i0aaRh4dmDcG4LNjVxXO\n"
HASH /data/golang/workdir/go/src/runtime/internal/sys/stubs.go: 23b3e5c631b086fe7a2dec4bf044600e034bf6a8eeb25e0a19efc4ce6311423d
HASH[build runtime/internal/sys]: "file stubs.go I7PlxjGwhv56LexL8ERg\n"
HASH /data/golang/workdir/go/src/runtime/internal/sys/sys.go: 55e021891200a7e6a5c371c8a1ab71b6c15aeb16ea6c1b192185d17df8c8b18f
HASH[build runtime/internal/sys]: "file sys.go VeAhiRIAp-alw3HIoatx\n"
HASH /data/golang/workdir/go/src/runtime/internal/sys/zgoarch_s390x.go: a0333e25c1ec5662104b2b1e8128b274a0218ace80c3a905210fe49223d663a6
HASH[build runtime/internal/sys]: "file zgoarch_s390x.go oDM-JcHsVmIQSysegSiy\n"
HASH /data/golang/workdir/go/src/runtime/internal/sys/zgoos_linux.go: 806c088d7491b4560a28a5af86a52b459ebbf155ea455af873baa0bf697355e4
HASH[build runtime/internal/sys]: "file zgoos_linux.go gGwIjXSRtFYKKKWvhqUr\n"
HASH /data/golang/workdir/go/src/runtime/internal/sys/zversion.go: 447c7f41ab18d0b6265c2a94f93269075a518bad8df751d07fc7ee9eab0e2275
HASH[build runtime/internal/sys]: "file zversion.go RHx_QasY0LYmXCqU-TJp\n"
HASH[build runtime/internal/sys]: e8ab1b435529fa39c8c480e61b2271a65ebd4b10f852a1ef18f3516817609de5
runtime/internal/sys true
go tool dist: unexpected stale targets reported by /data/golang/workdir/go/pkg/tool/linux_s390x/go_bootstrap list -gcflags="" -ldflags="" for [cmd/asm cmd/cgo cmd/compile cmd/link runtime/internal/sys]:
	STALE cmd/asm: stale dependency: internal/cpu
	STALE cmd/cgo: stale dependency: internal/cpu
	STALE cmd/compile: stale dependency: internal/cpu
	STALE cmd/link: stale dependency: internal/cpu
	STALE runtime/internal/sys: build ID mismatch

[bcmills]

Given the number and diversity of failing builders, I think we should roll back CL 171758 until it can be fixed.

[gopherbot]

Change https://golang.org/cl/180761 mentions this issue: Revert "cmd/compile,runtime: allocate defer records on the stack"