Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: fatal error: unknown caller pc when uprobes are attached #27077

Closed
bobrik opened this issue Aug 19, 2018 · 7 comments

Comments

Projects
None yet
5 participants
@bobrik
Copy link

commented Aug 19, 2018

What version of Go are you using (go version)?

go version go1.10.3 linux/amd64

Does this issue reproduce with the latest release?

This is the latest release.

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/home/ivan/.cache/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/ivan/go"
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build679123346=/tmp/go-build -gno-record-gcc-switches"

What did you do?

package main

import (
	"crypto/x509"
)

var cert = `
-----BEGIN CERTIFICATE-----
MIIIXjCCB0agAwIBAgIISYIp220EO/4wDQYJKoZIhvcNAQELBQAwVDELMAkGA1UE
BhMCVVMxHjAcBgNVBAoTFUdvb2dsZSBUcnVzdCBTZXJ2aWNlczElMCMGA1UEAxMc
R29vZ2xlIEludGVybmV0IEF1dGhvcml0eSBHMzAeFw0xODA4MDcxODQxMDVaFw0x
ODEwMTYxODI5MDBaMGYxCzAJBgNVBAYTAlVTMRMwEQYDVQQIDApDYWxpZm9ybmlh
MRYwFAYDVQQHDA1Nb3VudGFpbiBWaWV3MRMwEQYDVQQKDApHb29nbGUgTExDMRUw
EwYDVQQDDAwqLmdvb2dsZS5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK
AoIBAQC3llkOaVfpyMmE6WLjKVkz/Y2PXnPTb47W8cHAn9MiqcsT/CKi+QvtfIDT
ej1GZQB9HKDuHrvvcwLdLIE+pUvZZl1BYhhAwKyzqL8x2WZCR/0FiY6yvLK+OA70
a6reIr4VZnBJ7slQR3c1rg5C9RFwhJmrMHDmhou20TYbPQvvrUyReLXdm6bpmklv
VWkF44HYmbv+rUOcn9bfFtGcrW/eOojnd2aXhrRu7bKyEdxQ7F8qNCZJXpadOeOs
aVIsSc+dhWOeCIby+hXyU7qN6ViMWKwpzrXxB/bYLzkBcatLl9VJTvpyvzRlB54Y
dj/pVtanr7762sxN9mp42UairbCHAgMBAAGjggUgMIIFHDATBgNVHSUEDDAKBggr
BgEFBQcDATCCA/UGA1UdEQSCA+wwggPoggwqLmdvb2dsZS5jb22CDSouYW5kcm9p
ZC5jb22CFiouYXBwZW5naW5lLmdvb2dsZS5jb22CEiouY2xvdWQuZ29vZ2xlLmNv
bYIUKi5kYjgzMzk1My5nb29nbGUuY26CBiouZy5jb4IOKi5nY3AuZ3Z0Mi5jb22C
FiouZ29vZ2xlLWFuYWx5dGljcy5jb22CCyouZ29vZ2xlLmNhggsqLmdvb2dsZS5j
bIIOKi5nb29nbGUuY28uaW6CDiouZ29vZ2xlLmNvLmpwgg4qLmdvb2dsZS5jby51
a4IPKi5nb29nbGUuY29tLmFygg8qLmdvb2dsZS5jb20uYXWCDyouZ29vZ2xlLmNv
bS5icoIPKi5nb29nbGUuY29tLmNvgg8qLmdvb2dsZS5jb20ubXiCDyouZ29vZ2xl
LmNvbS50coIPKi5nb29nbGUuY29tLnZuggsqLmdvb2dsZS5kZYILKi5nb29nbGUu
ZXOCCyouZ29vZ2xlLmZyggsqLmdvb2dsZS5odYILKi5nb29nbGUuaXSCCyouZ29v
Z2xlLm5sggsqLmdvb2dsZS5wbIILKi5nb29nbGUucHSCEiouZ29vZ2xlYWRhcGlz
LmNvbYIPKi5nb29nbGVhcGlzLmNughQqLmdvb2dsZWNvbW1lcmNlLmNvbYIRKi5n
b29nbGV2aWRlby5jb22CDCouZ3N0YXRpYy5jboINKi5nc3RhdGljLmNvbYISKi5n
c3RhdGljY25hcHBzLmNuggoqLmd2dDEuY29tggoqLmd2dDIuY29tghQqLm1ldHJp
Yy5nc3RhdGljLmNvbYIMKi51cmNoaW4uY29tghAqLnVybC5nb29nbGUuY29tghYq
LnlvdXR1YmUtbm9jb29raWUuY29tgg0qLnlvdXR1YmUuY29tghYqLnlvdXR1YmVl
ZHVjYXRpb24uY29tggcqLnl0LmJlggsqLnl0aW1nLmNvbYIaYW5kcm9pZC5jbGll
bnRzLmdvb2dsZS5jb22CC2FuZHJvaWQuY29tghtkZXZlbG9wZXIuYW5kcm9pZC5n
b29nbGUuY26CHGRldmVsb3BlcnMuYW5kcm9pZC5nb29nbGUuY26CBGcuY2+CBmdv
by5nbIIUZ29vZ2xlLWFuYWx5dGljcy5jb22CCmdvb2dsZS5jb22CEmdvb2dsZWNv
bW1lcmNlLmNvbYIYc291cmNlLmFuZHJvaWQuZ29vZ2xlLmNuggp1cmNoaW4uY29t
ggp3d3cuZ29vLmdsggh5b3V0dS5iZYILeW91dHViZS5jb22CFHlvdXR1YmVlZHVj
YXRpb24uY29tggV5dC5iZTBoBggrBgEFBQcBAQRcMFowLQYIKwYBBQUHMAKGIWh0
dHA6Ly9wa2kuZ29vZy9nc3IyL0dUU0dJQUczLmNydDApBggrBgEFBQcwAYYdaHR0
cDovL29jc3AucGtpLmdvb2cvR1RTR0lBRzMwHQYDVR0OBBYEFD07Ps0NzAXJoaeN
11+6CV+0KHOhMAwGA1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAUd8K4UJpndnaxLcKG
0IOgfqZ+ukswIQYDVR0gBBowGDAMBgorBgEEAdZ5AgUDMAgGBmeBDAECAjAxBgNV
HR8EKjAoMCagJKAihiBodHRwOi8vY3JsLnBraS5nb29nL0dUU0dJQUczLmNybDAN
BgkqhkiG9w0BAQsFAAOCAQEAY6yfAJ2HnClPVqLRQzWPYxm2OnGxcOIdigN39XvH
v3sOipSW1WpRFJ5zg7Ochdp9MUN/2DqlMyDmJCc3YR/6v623oU6NcWDPqFveXI8c
vrnmKwdd60bi9CpTw8STv8ignTGkBaovCzi22jRC7NBx+ZQiU8sNfFlfzMRYxEZr
Ezp405dtkYjqYp4B0xdew+Tvti8UXf3F9e0OuwYP0MV3NHWyDSeBcb99wD5f99Xc
39egtp3LZyDFDXrgF3yLPnhzM5OXWfhyRVtCwkWa59nzfSGHkfovI9Z8Pre6qZ1m
2+nUKuASSco0IikHn7g52EujawBTn9ZZH1t3omt7X4Q4Fg==
-----END CERTIFICATE-----
`

func main() {
	c := x509.Certificate{Raw: []byte(cert)}
	c.Verify(x509.VerifyOptions{})
}

Compiled:

go build -o /tmp/pc pc.go

Then attach uprobe with bcc tools:

$ sudo /usr/share/bcc/tools/funclatency '/tmp/pc:crypto\/x509\.\(*Certificate\)\.Verify'
Tracing 1 functions for "/tmp/pc:crypto\/x509\.\(*Certificate\)\.Verify"... Hit Ctrl-C to end.

Then run the program:

$ /tmp/pc
runtime: unexpected return pc for crypto/x509.(*Certificate).Verify called from 0x7fffffffe000
stack: frame={sp:0xc42012f568, fp:0xc42012f9b8} stack=[0xc42012c000,0xc420130000)
000000c42012f468:  0000000000000000  000000c42012f510
000000c42012f478:  00000000004a90f1 <log.(*Logger).Output+273>  000000c4200f60f0
000000c42012f488:  000000c4200f6120  bed65b96d36e5785
000000c42012f498:  00000000000fbc19  000000000060ca60
000000c42012f4a8:  0000000000000000  0000000000000000
000000c42012f4b8:  0000000000000000  00000000004a0454 <fmt.Sprintf+180>
000000c42012f4c8:  000000c420124000  000000c4201000c0
000000c42012f4d8:  0000000000000017  00000000000fbc19
000000c42012f4e8:  bed65b96d36e5785  0000000000000017
000000c42012f4f8:  0000000000000017  000000c4200f6120
000000c42012f508:  000000000060ca60  000000c42012f558
000000c42012f518:  00000000004a9440 <log.Printf+128>  000000c4200f60f0
000000c42012f528:  0000000000000002  000000c4201000e0
000000c42012f538:  0000000000000017  0000000000000000
000000c42012f548:  0000000000000000  0000000000000017
000000c42012f558:  000000c42012f9a8  00000000004fe79c <crypto/x509.(*Certificate).Verify+492>
000000c42012f568: <0000000000554e7e  0000000000000018
000000c42012f578:  000000c4200ab7c8  0000000000000001
000000c42012f588:  0000000000000001  0000000000000000
000000c42012f598:  0000000000000000  0000000000000000
000000c42012f5a8:  0000000000000000  0000000000000000
000000c42012f5b8:  0000000000000000  0000000000000000
000000c42012f5c8:  0000000000000000  0000000000000000
000000c42012f5d8:  0000000000000000  0000000000000000
000000c42012f5e8:  0000000000000000  0000000000000000
000000c42012f5f8:  0000000000000000  0000000000000000
000000c42012f608:  0000000000000000  0000000000000000
000000c42012f618:  0000000000000000  0000000000000000
000000c42012f628:  0000000000000000  0000000000000000
000000c42012f638:  0000000000000000  0000000000000000
000000c42012f648:  0000000000000000  0000000000000000
000000c42012f658:  0000000000000000  0000000000000000
000000c42012f668:  0000000000000000  0000000000000000
000000c42012f678:  0000000000000000  0000000000000000
000000c42012f688:  0000000000000000  0000000000000000
000000c42012f698:  0000000000000000  0000000000000000
000000c42012f6a8:  0000000000000000  0000000000000000
000000c42012f6b8:  0000000000000000  0000000000000000
000000c42012f6c8:  0000000000000000  0000000000000000
000000c42012f6d8:  0000000000000000  0000000000000000
000000c42012f6e8:  0000000000000000  0000000000000000
000000c42012f6f8:  0000000000000000  0000000000000000
000000c42012f708:  0000000000000000  0000000000000000
000000c42012f718:  bed65b96d36dceef  00000000000f33a8
000000c42012f728:  0000000000000000  0000000000000000
000000c42012f738:  0000000000000000  0000000000000000
000000c42012f748:  0000000000000000  0000000000000000
000000c42012f758:  0000000000000000  0000000000000000
000000c42012f768:  0000000000000000  000000000060ca60
000000c42012f778:  0000000000000000  0000000000000000
000000c42012f788:  0000000000000000  0000000000000000
000000c42012f798:  0000000000000000  0000000000000000
000000c42012f7a8:  0000000000000000  0000000000000000
000000c42012f7b8:  0000000000000000  0000000000000000
000000c42012f7c8:  0000000000521480  000000000062a220
000000c42012f7d8:  0000000000000000  0000000000000000
000000c42012f7e8:  0000000000000000  0000000000000000
000000c42012f7f8:  0000000000000000  0000000000000000
000000c42012f808:  0000000000000000  0000000000000008
000000c42012f818:  0000000000000000  0000000000000000
000000c42012f828:  000000c4200ab880  000000000040f0f6 <runtime.(*mcache).nextFree+198>
000000c42012f838:  00007fb94f2be308  0000000000000000
000000c42012f848:  0000000000000000  0000000000000000
000000c42012f858:  0000000000000051  00007fb94f2be308
000000c42012f868:  0000000000000000  0000000000000000
000000c42012f878:  0000000000000000  0000000000000000
000000c42012f888:  0000000000000000  0000000000000000
000000c42012f898:  0000000000000000  0000000000000000
000000c42012f8a8:  0000000000000000  0000000000000000
000000c42012f8b8:  0000000000000000  0000000000000000
000000c42012f8c8:  0000000000000000  0000000000000000
000000c42012f8d8:  0000000000000000  0000000000000000
000000c42012f8e8:  0000000000000000  0000000000000000
000000c42012f8f8:  0000000000000000  0000000000000000
000000c42012f908:  0000000000000000  0000000000000000
000000c42012f918:  0000000000000000  0000000000000000
000000c42012f928:  0000000000000000  0000000000000000
000000c42012f938:  0000000000000000  0000000000000000
000000c42012f948:  0000000000000000  0000000000000000
000000c42012f958:  0000000000000000  0000000000000000
000000c42012f968:  0000000000000000  0000000000000000
000000c42012f978:  0000000000000000  0000000000000000
000000c42012f988:  0000000000000000  0000000000000000
000000c42012f998:  0000000000000000  0000000000000000
000000c42012f9a8:  000000c4200abf78 !00007fffffffe000
000000c42012f9b8: >000000c420116000  0000000000000000
000000c42012f9c8:  0000000000000000  0000000000000000
000000c42012f9d8:  0000000000000000  0000000000000000
000000c42012f9e8:  0000000000000000  0000000000000000
000000c42012f9f8:  0000000000000000  0000000000000000
000000c42012fa08:  0000000000000000  0000000000000000
000000c42012fa18:  0000000000000000  0000000000000000
000000c42012fa28:  0000000000000000  0000000000000000
000000c42012fa38:  0000000000000000  000000c420116000
000000c42012fa48:  000000c42011a000  0000000000000b94
000000c42012fa58:  0000000000000c00  0000000000000000
000000c42012fa68:  0000000000000000  0000000000000000
000000c42012fa78:  0000000000000000  0000000000000000
000000c42012fa88:  0000000000000000  0000000000000000
000000c42012fa98:  0000000000000000  0000000000000000
000000c42012faa8:  0000000000000000  0000000000000000
fatal error: unknown caller pc

runtime stack:
runtime.throw(0x5538ab, 0x11)
	/usr/local/go/src/runtime/panic.go:616 +0x81
runtime.gentraceback(0xffffffffffffffff, 0xffffffffffffffff, 0x0, 0xc420000180, 0x0, 0x0, 0x7fffffff, 0x55c088, 0x7fff74e5e788, 0x0, ...)
	/usr/local/go/src/runtime/traceback.go:257 +0x1bdb
runtime.copystack(0xc420000180, 0x4000, 0x7fff74e5e901)
	/usr/local/go/src/runtime/stack.go:891 +0x270
runtime.newstack()
	/usr/local/go/src/runtime/stack.go:1063 +0x30f
runtime.morestack()
	/usr/local/go/src/runtime/asm_amd64.s:480 +0x89

goroutine 1 [copystack]:
time.readFile(0xc4200f2250, 0xf, 0x5, 0x551579, 0x1, 0x55225d, 0x9)
	/usr/local/go/src/time/zoneinfo_read.go:412 +0x37f fp=0xc42012f0e8 sp=0xc42012f0e0 pc=0x49706f
time.loadTzinfoFromDirOrZip(0x5518bb, 0x5, 0x55225d, 0x9, 0xc4200f8120, 0x40a6df, 0xc4200ab1f0, 0xe7d1497a, 0xc4200ab1d0)
	/usr/local/go/src/time/zoneinfo_read.go:233 +0x8b fp=0xc42012f140 sp=0xc42012f0e8 pc=0x49587b
time.loadTzinfo(0x55225d, 0x9, 0x5518bb, 0x5, 0xc4200f8100, 0x1, 0xc4200ab1e0, 0xc4200ab210, 0x488258)
	/usr/local/go/src/time/zoneinfo_read.go:383 +0x98 fp=0xc42012f198 sp=0xc42012f140 pc=0x496998
time.loadLocation(0x55225d, 0x9, 0xc42012f260, 0x1, 0x1, 0x426d19, 0xc4200f80f0, 0xc4200ab290)
	/usr/local/go/src/time/zoneinfo_read.go:392 +0xa3 fp=0xc42012f220 sp=0xc42012f198 pc=0x496b23
time.initLocal()
	/usr/local/go/src/time/zoneinfo_unix.go:37 +0x1b1 fp=0xc42012f280 sp=0xc42012f220 pc=0x497231
sync.(*Once).Do(0x6297c0, 0x55c3d8)
	/usr/local/go/src/sync/once.go:44 +0xbe fp=0xc42012f2b8 sp=0xc42012f280 pc=0x45868e
time.(*Location).get(0x60ca60, 0xc4201240ab)
	/usr/local/go/src/time/zoneinfo.go:79 +0x5e fp=0xc42012f2d8 sp=0xc42012f2b8 pc=0x493ebe
time.Time.abs(0xbed65b96d36e5785, 0xfbc19, 0x60ca60, 0x5535f2)
	/usr/local/go/src/time/time.go:441 +0x40 fp=0xc42012f330 sp=0xc42012f2d8 pc=0x492470
time.Time.date(0xbed65b96d36e5785, 0xfbc19, 0x60ca60, 0x1, 0xc4201000c0, 0x48, 0x14, 0x20)
	/usr/local/go/src/time/time.go:936 +0x3f fp=0xc42012f370 sp=0xc42012f330 pc=0x4932ff
time.Time.Date(0xbed65b96d36e5785, 0xfbc19, 0x60ca60, 0x4a7c11, 0xc4201000d5, 0x554e94)
	/usr/local/go/src/time/time.go:481 +0x44 fp=0xc42012f3c0 sp=0xc42012f370 pc=0x4926e4
log.(*Logger).formatHeader(0xc4200f60f0, 0xc4200f6120, 0xbed65b96d36e5785, 0xfbc19, 0x60ca60, 0x0, 0x0, 0x0)
	/usr/local/go/src/log/log.go:103 +0x7ff fp=0xc42012f480 sp=0xc42012f3c0 pc=0x4a8cdf
log.(*Logger).Output(0xc4200f60f0, 0x2, 0xc4201000e0, 0x17, 0x0, 0x0)
	/usr/local/go/src/log/log.go:167 +0x111 fp=0xc42012f520 sp=0xc42012f480 pc=0x4a90f1
log.Printf(0x554e7e, 0x18, 0xc4200ab7c8, 0x1, 0x1)
	/usr/local/go/src/log/log.go:295 +0x80 fp=0xc42012f568 sp=0xc42012f520 pc=0x4a9440
runtime: unexpected return pc for crypto/x509.(*Certificate).Verify called from 0x7fffffffe000
stack: frame={sp:0xc42012f568, fp:0xc42012f9b8} stack=[0xc42012c000,0xc420130000)
000000c42012f468:  0000000000000000  000000c42012f510
000000c42012f478:  00000000004a90f1 <log.(*Logger).Output+273>  000000c4200f60f0
000000c42012f488:  000000c4200f6120  bed65b96d36e5785
000000c42012f498:  00000000000fbc19  000000000060ca60
000000c42012f4a8:  0000000000000000  0000000000000000
000000c42012f4b8:  0000000000000000  00000000004a0454 <fmt.Sprintf+180>
000000c42012f4c8:  000000c420124000  000000c4201000c0
000000c42012f4d8:  0000000000000017  00000000000fbc19
000000c42012f4e8:  bed65b96d36e5785  0000000000000017
000000c42012f4f8:  0000000000000017  000000c4200f6120
000000c42012f508:  000000000060ca60  000000c42012f558
000000c42012f518:  00000000004a9440 <log.Printf+128>  000000c4200f60f0
000000c42012f528:  0000000000000002  000000c4201000e0
000000c42012f538:  0000000000000017  0000000000000000
000000c42012f548:  0000000000000000  0000000000000017
000000c42012f558:  000000c42012f9a8  00000000004fe79c <crypto/x509.(*Certificate).Verify+492>
000000c42012f568: <0000000000554e7e  0000000000000018
000000c42012f578:  000000c4200ab7c8  0000000000000001
000000c42012f588:  0000000000000001  0000000000000000
000000c42012f598:  0000000000000000  0000000000000000
000000c42012f5a8:  0000000000000000  0000000000000000
000000c42012f5b8:  0000000000000000  0000000000000000
000000c42012f5c8:  0000000000000000  0000000000000000
000000c42012f5d8:  0000000000000000  0000000000000000
000000c42012f5e8:  0000000000000000  0000000000000000
000000c42012f5f8:  0000000000000000  0000000000000000
000000c42012f608:  0000000000000000  0000000000000000
000000c42012f618:  0000000000000000  0000000000000000
000000c42012f628:  0000000000000000  0000000000000000
000000c42012f638:  0000000000000000  0000000000000000
000000c42012f648:  0000000000000000  0000000000000000
000000c42012f658:  0000000000000000  0000000000000000
000000c42012f668:  0000000000000000  0000000000000000
000000c42012f678:  0000000000000000  0000000000000000
000000c42012f688:  0000000000000000  0000000000000000
000000c42012f698:  0000000000000000  0000000000000000
000000c42012f6a8:  0000000000000000  0000000000000000
000000c42012f6b8:  0000000000000000  0000000000000000
000000c42012f6c8:  0000000000000000  0000000000000000
000000c42012f6d8:  0000000000000000  0000000000000000
000000c42012f6e8:  0000000000000000  0000000000000000
000000c42012f6f8:  0000000000000000  0000000000000000
000000c42012f708:  0000000000000000  0000000000000000
000000c42012f718:  bed65b96d36dceef  00000000000f33a8
000000c42012f728:  0000000000000000  0000000000000000
000000c42012f738:  0000000000000000  0000000000000000
000000c42012f748:  0000000000000000  0000000000000000
000000c42012f758:  0000000000000000  0000000000000000
000000c42012f768:  0000000000000000  000000000060ca60
000000c42012f778:  0000000000000000  0000000000000000
000000c42012f788:  0000000000000000  0000000000000000
000000c42012f798:  0000000000000000  0000000000000000
000000c42012f7a8:  0000000000000000  0000000000000000
000000c42012f7b8:  0000000000000000  0000000000000000
000000c42012f7c8:  0000000000521480  000000000062a220
000000c42012f7d8:  0000000000000000  0000000000000000
000000c42012f7e8:  0000000000000000  0000000000000000
000000c42012f7f8:  0000000000000000  0000000000000000
000000c42012f808:  0000000000000000  0000000000000008
000000c42012f818:  0000000000000000  0000000000000000
000000c42012f828:  000000c4200ab880  000000000040f0f6 <runtime.(*mcache).nextFree+198>
000000c42012f838:  00007fb94f2be308  0000000000000000
000000c42012f848:  0000000000000000  0000000000000000
000000c42012f858:  0000000000000051  00007fb94f2be308
000000c42012f868:  0000000000000000  0000000000000000
000000c42012f878:  0000000000000000  0000000000000000
000000c42012f888:  0000000000000000  0000000000000000
000000c42012f898:  0000000000000000  0000000000000000
000000c42012f8a8:  0000000000000000  0000000000000000
000000c42012f8b8:  0000000000000000  0000000000000000
000000c42012f8c8:  0000000000000000  0000000000000000
000000c42012f8d8:  0000000000000000  0000000000000000
000000c42012f8e8:  0000000000000000  0000000000000000
000000c42012f8f8:  0000000000000000  0000000000000000
000000c42012f908:  0000000000000000  0000000000000000
000000c42012f918:  0000000000000000  0000000000000000
000000c42012f928:  0000000000000000  0000000000000000
000000c42012f938:  0000000000000000  0000000000000000
000000c42012f948:  0000000000000000  0000000000000000
000000c42012f958:  0000000000000000  0000000000000000
000000c42012f968:  0000000000000000  0000000000000000
000000c42012f978:  0000000000000000  0000000000000000
000000c42012f988:  0000000000000000  0000000000000000
000000c42012f998:  0000000000000000  0000000000000000
000000c42012f9a8:  000000c4200abf78 !00007fffffffe000
000000c42012f9b8: >000000c420116000  0000000000000000
000000c42012f9c8:  0000000000000000  0000000000000000
000000c42012f9d8:  0000000000000000  0000000000000000
000000c42012f9e8:  0000000000000000  0000000000000000
000000c42012f9f8:  0000000000000000  0000000000000000
000000c42012fa08:  0000000000000000  0000000000000000
000000c42012fa18:  0000000000000000  0000000000000000
000000c42012fa28:  0000000000000000  0000000000000000
000000c42012fa38:  0000000000000000  000000c420116000
000000c42012fa48:  000000c42011a000  0000000000000b94
000000c42012fa58:  0000000000000c00  0000000000000000
000000c42012fa68:  0000000000000000  0000000000000000
000000c42012fa78:  0000000000000000  0000000000000000
000000c42012fa88:  0000000000000000  0000000000000000
000000c42012fa98:  0000000000000000  0000000000000000
000000c42012faa8:  0000000000000000  0000000000000000
crypto/x509.(*Certificate).Verify(0xc420116000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/usr/local/go/src/crypto/x509/verify.go:742 +0x1ec fp=0xc42012f9b8 sp=0xc42012f568 pc=0x4fe79c

This doesn't happen with simple calls and I wasn't unable to write a simpler reproduction that doesn't go into guts of certificate validation. With detached uprobes there's no error.

What did you expect to see?

Program successfully terminates, uprobes do their business.

What did you see instead?

Program aborts, uprobes never see anything.

@josharian

This comment has been minimized.

Copy link
Contributor

commented Aug 19, 2018

cc @aclements

Speculation, but one possible reason reading certs might matter is that it makes a cgo call.

@josharian josharian changed the title fatal error: unknown caller pc with attached uprobes runtime: fatal error: unknown caller pc when uprobes are attached Aug 19, 2018

@aclements

This comment has been minimized.

Copy link
Member

commented Aug 20, 2018

This definitely looks like uprobes is installing a return trampoline on the stack to catch function return, which is going to mess up Go stack unwinding no matter what.

I don't see why this would require a cgo call. What happens if you simply have something like

func x() {
    runtime.GC()
}

and set a return uprobe on (or apply funclatency to) x?

@aclements

This comment has been minimized.

Copy link
Member

commented Aug 20, 2018

Yeah, I just poked through the uretprobe implementation and it works by allocating a page in user space, putting an INT $3 instruction on it, and then overwriting the return address on the stack to return to the INT $3. It's not clear to me how the runtime (or anything that unwinds the stack) could account for this.

@bobrik bobrik referenced this issue Aug 21, 2018

Open

Gotrace #1719

@nyarly

This comment has been minimized.

Copy link

commented Aug 22, 2018

@aclements I'm not sure I understand the issue here. uretprobes works for other language platforms - what does Go do differently? Is this related to Plan 9 style stack frames? Is there a reference to the "unwinding the stack" procedure that the runtime does?

@aclements

This comment has been minimized.

Copy link
Member

commented Aug 22, 2018

The only difference is that Go depends on its ability to unwind stacks for GC and stack growth. I assert that uretprobes would break stack unwinding in any language, and regardless of calling convention. I'm actually really curious how it interacts with C++ exception handling; I suspect uretprobes breaks it.

uretprobes clobbers critical unwinding information and, as far as I can tell, doesn't provide a way to get it back. However, I would love to be proved wrong, since I know how powerful uprobes can be.

@sillyousu

This comment has been minimized.

@aclements

This comment has been minimized.

Copy link
Member

commented Aug 23, 2018

Thanks for the references, @sillyousu. Those confirm my suspicions that, unfortunately, there's really nothing we can do to work around uretprobes effectively corrupting the stack.

Since there's nothing we can do, and this isn't really a Go bug, I'm going to close this issue. If uretprobes one day exposes enough information to recover the clobbered return address in user space, we can revisit this issue and might be able to work around it.

@aclements aclements closed this Aug 23, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.