tests fail on ppc64le

[carlwgeorge]

Description

The containerd builds in Fedora have started to fail on just the ppc64le architecture.

• https://koji.fedoraproject.org/koji/taskinfo?taskID=32749298
• https://kojipkgs.fedoraproject.org/work/tasks/9337/32749337/build.log

Back in January this same version of containerd (1.2.1) passed all the tests on all architectures, including ppc64le.

• https://koji.fedoraproject.org/koji/taskinfo?taskID=31813189
• https://kojipkgs.fedoraproject.org/packages/containerd/1.2.1/1.fc30/data/logs/ppc64le/build.log

The big thing that sticks out to me as different between the builds is that the build that passed was using golang-1.11.4-1.fc30.ppc64le, but now the buildroot has moved on to golang-1.12-0.beta2.2.fc30.1.ppc64le.

I also did a scratch build with 1.2.3 to see if this had already been resolved, but I got the same error.

• https://koji.fedoraproject.org/koji/taskinfo?taskID=32749106
• https://kojipkgs.fedoraproject.org/work/tasks/9108/32749108/build.log

Steps to reproduce the issue:

1. go test within the metadata directory on ppc64le architecture

Describe the results you received:

Testing: "/builddir/build/BUILD/containerd-1.2.1/_build/src/github.com/containerd/containerd/metadata"
+ GOPATH=/builddir/build/BUILD/containerd-1.2.1/_build:/usr/share/gocode
+ go test -buildmode pie -compiler gc -ldflags '-extldflags '\''-Wl,-z,relro -Wl,--as-needed  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld '\'''
unexpected fault address 0x1249289f0
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0x1249289f0 pc=0x124e59130]
goroutine 5 [running]:
runtime.throw(0x1253e7528, 0x5)
	/usr/lib/golang/src/runtime/panic.go:617 +0x68 fp=0xc0001b50f0 sp=0xc0001b50b0 pc=0x124e7b988
runtime.sigpanic()
	/usr/lib/golang/src/runtime/signal_unix.go:397 +0x464 fp=0xc0001b5130 sp=0xc0001b50f0 pc=0x124e947c4
runtime.mapiterinit(0x1249289b0, 0xc0001cc2d0, 0xc0001b5328)
	/usr/lib/golang/src/runtime/map.go:823 +0xa0 fp=0xc0001b5180 sp=0xc0001b5150 pc=0x124e59130
github.com/containerd/containerd/vendor/go.etcd.io/bbolt.(*Bucket).spill(0xc00006d4c0, 0xc0001cc200, 0xc0001b5530)
	/builddir/build/BUILD/containerd-1.2.1/_build/src/github.com/containerd/containerd/vendor/go.etcd.io/bbolt/bucket.go:526 +0x80 fp=0xc0001b5388 sp=0xc0001b5180 pc=0x125055bd0
github.com/containerd/containerd/vendor/go.etcd.io/bbolt.(*Bucket).spill(0xc00006d440, 0xc0001cc200, 0xc0001b5738)
	/builddir/build/BUILD/containerd-1.2.1/_build/src/github.com/containerd/containerd/vendor/go.etcd.io/bbolt/bucket.go:535 +0x308 fp=0xc0001b5590 sp=0xc0001b5388 pc=0x125055e58
github.com/containerd/containerd/vendor/go.etcd.io/bbolt.(*Bucket).spill(0xc00006d3c0, 0xc0001cc100, 0xc0001b5940)
	/builddir/build/BUILD/containerd-1.2.1/_build/src/github.com/containerd/containerd/vendor/go.etcd.io/bbolt/bucket.go:535 +0x308 fp=0xc0001b5798 sp=0xc0001b5590 pc=0x125055e58
github.com/containerd/containerd/vendor/go.etcd.io/bbolt.(*Bucket).spill(0xc00006d340, 0xc0001cc100, 0xc0001b5b48)
	/builddir/build/BUILD/containerd-1.2.1/_build/src/github.com/containerd/containerd/vendor/go.etcd.io/bbolt/bucket.go:535 +0x308 fp=0xc0001b59a0 sp=0xc0001b5798 pc=0x125055e58
github.com/containerd/containerd/vendor/go.etcd.io/bbolt.(*Bucket).spill(0xc0001b81d8, 0x730d88b, 0x125c0a520)
	/builddir/build/BUILD/containerd-1.2.1/_build/src/github.com/containerd/containerd/vendor/go.etcd.io/bbolt/bucket.go:535 +0x308 fp=0xc0001b5ba8 sp=0xc0001b59a0 pc=0x125055e58
github.com/containerd/containerd/vendor/go.etcd.io/bbolt.(*Tx).Commit(0xc0001b81c0, 0x0, 0x0)
	/builddir/build/BUILD/containerd-1.2.1/_build/src/github.com/containerd/containerd/vendor/go.etcd.io/bbolt/tx.go:160 +0xd0 fp=0xc0001b5ce8 sp=0xc0001b5ba8 pc=0x125063270
github.com/containerd/containerd/vendor/go.etcd.io/bbolt.(*DB).Update(0xc000156000, 0xc0001b5ed8, 0x0, 0x0)
	/builddir/build/BUILD/containerd-1.2.1/_build/src/github.com/containerd/containerd/vendor/go.etcd.io/bbolt/db.go:677 +0x108 fp=0xc0001b5d48 sp=0xc0001b5ce8 pc=0x12505af88
github.com/containerd/containerd/metadata.TestContainersList(0xc0000c2400)
	/builddir/build/BUILD/containerd-1.2.1/_build/src/github.com/containerd/containerd/metadata/containers_test.go:74 +0x4e0 fp=0xc0001b5f70 sp=0xc0001b5d48 pc=0x1253b3290
testing.tRunner(0xc0000c2400, 0x12576e090)
	/usr/lib/golang/src/testing/testing.go:862 +0xdc fp=0xc0001b5fb0 sp=0xc0001b5f70 pc=0x124f4cb5c
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_ppc64x.s:856 +0x4 fp=0xc0001b5fb0 sp=0xc0001b5fb0 pc=0x124eaeaa4
created by testing.(*T).Run
	/usr/lib/golang/src/testing/testing.go:913 +0x304
goroutine 1 [chan receive]:
testing.(*T).Run(0xc0000c2400, 0x1253f100c, 0x12, 0x12576e090, 0x125bc7f00)
	/usr/lib/golang/src/testing/testing.go:914 +0x320
testing.runTests.func1(0xc0000c2300)
	/usr/lib/golang/src/testing/testing.go:1154 +0x8c
testing.tRunner(0xc0000c2300, 0xc0000a9dd8)
	/usr/lib/golang/src/testing/testing.go:862 +0xdc
testing.runTests(0xc00014f6a0, 0x125bc94c0, 0x10, 0x10, 0x0)
	/usr/lib/golang/src/testing/testing.go:1152 +0x2a0
testing.(*M).Run(0xc00000c300, 0x0)
	/usr/lib/golang/src/testing/testing.go:1069 +0x174
main.main()
	_testmain.go:74 +0x150
exit status 2
FAIL	github.com/containerd/containerd/metadata	0.128s

Describe the results you expected:

Testing: "/builddir/build/BUILD/containerd-1.2.1/_build/src/github.com/containerd/containerd/metadata"
+ GOPATH=/builddir/build/BUILD/containerd-1.2.1/_build:/usr/share/gocode
+ go test -buildmode pie -compiler gc -ldflags '-extldflags '\''-Wl,-z,relro -Wl,--as-needed  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld '\'''
PASS
ok  	github.com/containerd/containerd/metadata	27.029s

Output of containerd --version:

containerd github.com/containerd/containerd 1.2.1

(same error with 1.2.3 as well)

Extra info

If ppc64le or golang 1.12 are not (yet) supported by the project, I'm fine with that, I just wanted to at least share the info. I have the option of disabling the tests on just ppc64le to get the build to pass, if needed.

See also #2996.

[thaJeztah]

Yes, we were running into segfaults in moby CI as well on powerpc; see #2896 (comment)

/cc @estesp

[estesp]

Thanks for the report. @tophj-ibm and @clnperez : any idea if there is a known issue with Golang 1.12 on ppc64le? if not, can we get someone to look into the above stack trace/issue?

[mkumatag]

@laboger any idea what is happening here?

[mkumatag]

I tried the steps mentioned and facing a problem only with build option -buildmode pie. If no option supplied then tests are going through.

Steps to reproduce this issues:

root@ip9-114-192-113:~# docker run -ti golang:rc-stretch
root@361515a017ed:/go#
root@361515a017ed:/go# go version
go version go1.12rc1 linux/ppc64le
root@361515a017ed:~# mkdir -p containerd_ws/src/github.com/containerd
root@361515a017ed:~# cd containerd_ws/src/github.com/containerd/
root@361515a017ed:~/containerd_ws/src/github.com/containerd# git clone https://github.com/containerd/containerd.git
Cloning into 'containerd'...
remote: Enumerating objects: 9, done.
remote: Counting objects: 100% (9/9), done.
remote: Compressing objects: 100% (9/9), done.
remote: Total 38383 (delta 2), reused 0 (delta 0), pack-reused 38374
Receiving objects: 100% (38383/38383), 49.63 MiB | 24.02 MiB/s, done.
Resolving deltas: 100% (22575/22575), done.
root@361515a017ed:~/containerd_ws/src/github.com/containerd#
root@361515a017ed:~/containerd_ws/src/github.com/containerd/containerd# cd metadata/
root@361515a017ed:~/containerd_ws/src/github.com/containerd/containerd/metadata#
root@361515a017ed:~/containerd_ws/src/github.com/containerd/containerd/metadata# export GOPATH=/root/containerd_ws
root@361515a017ed:~/containerd_ws/src/github.com/containerd/containerd/metadata# go test -buildmode pie -v
=== RUN   TestContainersList
unexpected fault address 0x135889140
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0x135889140 pc=0x135dbd820]

goroutine 19 [running]:
runtime.throw(0x13634f828, 0x5)
	/usr/local/go/src/runtime/panic.go:617 +0x68 fp=0xc0001fb0f0 sp=0xc0001fb0b0 pc=0x135ddff38
runtime.sigpanic()
	/usr/local/go/src/runtime/signal_unix.go:397 +0x464 fp=0xc0001fb130 sp=0xc0001fb0f0 pc=0x135df8d64
runtime.mapiterinit(0x135889100, 0xc00020e360, 0xc0001fb328)
	/usr/local/go/src/runtime/map.go:823 +0xa0 fp=0xc0001fb180 sp=0xc0001fb150 pc=0x135dbd820
github.com/containerd/containerd/vendor/go.etcd.io/bbolt.(*Bucket).spill(0xc0000bb400, 0xc00020e300, 0xc0001fb530)
	/root/containerd_ws/src/github.com/containerd/containerd/vendor/go.etcd.io/bbolt/bucket.go:526 +0x80 fp=0xc0001fb388 sp=0xc0001fb180 pc=0x13600eb40
github.com/containerd/containerd/vendor/go.etcd.io/bbolt.(*Bucket).spill(0xc0000bb380, 0xc00020e200, 0xc0001fb738)
	/root/containerd_ws/src/github.com/containerd/containerd/vendor/go.etcd.io/bbolt/bucket.go:535 +0x308 fp=0xc0001fb590 sp=0xc0001fb388 pc=0x13600edc8
github.com/containerd/containerd/vendor/go.etcd.io/bbolt.(*Bucket).spill(0xc0000bb300, 0xc00020e200, 0xc0001fb940)
	/root/containerd_ws/src/github.com/containerd/containerd/vendor/go.etcd.io/bbolt/bucket.go:535 +0x308 fp=0xc0001fb798 sp=0xc0001fb590 pc=0x13600edc8
github.com/containerd/containerd/vendor/go.etcd.io/bbolt.(*Bucket).spill(0xc0000bb280, 0xc00020e100, 0xc0001fbb48)
	/root/containerd_ws/src/github.com/containerd/containerd/vendor/go.etcd.io/bbolt/bucket.go:535 +0x308 fp=0xc0001fb9a0 sp=0xc0001fb798 pc=0x13600edc8
github.com/containerd/containerd/vendor/go.etcd.io/bbolt.(*Bucket).spill(0xc0001fe1d8, 0x62c845, 0x136b6a680)
	/root/containerd_ws/src/github.com/containerd/containerd/vendor/go.etcd.io/bbolt/bucket.go:535 +0x308 fp=0xc0001fbba8 sp=0xc0001fb9a0 pc=0x13600edc8
github.com/containerd/containerd/vendor/go.etcd.io/bbolt.(*Tx).Commit(0xc0001fe1c0, 0x0, 0x0)
	/root/containerd_ws/src/github.com/containerd/containerd/vendor/go.etcd.io/bbolt/tx.go:160 +0xd0 fp=0xc0001fbce8 sp=0xc0001fbba8 pc=0x13601c1e0
github.com/containerd/containerd/vendor/go.etcd.io/bbolt.(*DB).Update(0xc00019a000, 0xc0001fbed8, 0x0, 0x0)
	/root/containerd_ws/src/github.com/containerd/containerd/vendor/go.etcd.io/bbolt/db.go:677 +0x108 fp=0xc0001fbd48 sp=0xc0001fbce8 pc=0x136013ef8
github.com/containerd/containerd/metadata.TestContainersList(0xc0000f4400)
	/root/containerd_ws/src/github.com/containerd/containerd/metadata/containers_test.go:74 +0x4e0 fp=0xc0001fbf70 sp=0xc0001fbd48 pc=0x13631b2d0
testing.tRunner(0xc0000f4400, 0x1366ca600)
	/usr/local/go/src/testing/testing.go:865 +0xdc fp=0xc0001fbfb0 sp=0xc0001fbf70 pc=0x135eb1dfc
runtime.goexit()
	/usr/local/go/src/runtime/asm_ppc64x.s:856 +0x4 fp=0xc0001fbfb0 sp=0xc0001fbfb0 pc=0x135e135f4
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:916 +0x304

goroutine 1 [chan receive]:
testing.(*T).Run(0xc0000f4400, 0x136359315, 0x12, 0x1366ca600, 0x136b27f00)
	/usr/local/go/src/testing/testing.go:917 +0x320
testing.runTests.func1(0xc0000f4300)
	/usr/local/go/src/testing/testing.go:1157 +0x8c
testing.tRunner(0xc0000f4300, 0xc0000d3dd8)
	/usr/local/go/src/testing/testing.go:865 +0xdc
testing.runTests(0xc000193660, 0x136b29600, 0x10, 0x10, 0x0)
	/usr/local/go/src/testing/testing.go:1155 +0x2a0
testing.(*M).Run(0xc0000f2280, 0x0)
	/usr/local/go/src/testing/testing.go:1072 +0x174
main.main()
	_testmain.go:74 +0x150
exit status 2
FAIL	github.com/containerd/containerd/metadata	0.013s
root@361515a017ed:~/containerd_ws/src/github.com/containerd/containerd/metadata#

root@361515a017ed:~/containerd_ws/src/github.com/containerd/containerd/metadata# uname -a
Linux 361515a017ed 4.4.0-141-generic #167-Ubuntu SMP Wed Dec 5 10:33:00 UTC 2018 ppc64le GNU/Linux
root@361515a017ed:~/containerd_ws/src/github.com/containerd/containerd/metadata# cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
NAME="Debian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
root@361515a017ed:~/containerd_ws/src/github.com/containerd/containerd/metadata# lscpu
Architecture:          ppc64le
Byte Order:            Little Endian
CPU(s):                16
On-line CPU(s) list:   0-15
Thread(s) per core:    8
Core(s) per socket:    1
Socket(s):             2
NUMA node(s):          2
Model:                 2.1 (pvr 004b 0201)
Model name:            POWER8 (architected), altivec supported
L1d cache:             64K
L1i cache:             32K
NUMA node0 CPU(s):     0-15
NUMA node2 CPU(s):
root@361515a017ed:~/containerd_ws/src/github.com/containerd/containerd/metadata#

Test results without -pie option - https://gist.github.com/mkumatag/58736357913eb088323db52cd3bad647 - PASSED

[thaJeztah]

Should we open an issue upstream in the Golang issue tracker?

[mkumatag]

Should we open an issue upstream in the Golang issue tracker?

Reported issue in golang community - golang/go#30283

[laboger]

I have identified the problem and put those details in the golang issue.

…

On Sun, Feb 17, 2019 at 10:57 AM Manjunath A Kumatagi < ***@***.***> wrote: Should we open an issue upstream in the Golang issue tracker? Reported issue in golang community - golang/go#30283 <golang/go#30283> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3005 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AI_wjNNiBeH9uShhOtQUsZWUUGFbqm_hks5vOYnqgaJpZM4a1csY> .

[estesp]

I assume we can close this now that Go 1.12 is released, and the inlining issue is now fixed upstream?

[carlwgeorge]

I haven't verified my Fedora builds work yet as I'm waiting for the Rawhide golang package to get updated to the final 1.12 version, but once that happens I'll report back here and close the issue.

[carlwgeorge]

Fedora rawhide has updated to golang 1.12 final, and now the containerd test suite passes on all architectures. Thanks for getting this nailed down so quickly!