Cross arch pprof build-id mismatch

[nmiculinic]

Please answer these questions before submitting your issue. Thanks!

What version of pprof are you using?

• go tool pprof

go env output:

GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/lpp/Library/Caches/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/lpp/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/Cellar/go/1.12/libexec"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.12/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/zh/1_x_pxls7px7l944c90_xmdm0000gn/T/go-build916206707=/tmp/go-build -gno-record-gcc-switches -fno-common"

What operating system and processor architecture are you using?

 perf_data_converter (master) ✔ uname -a                                          k8s [kubernetes-admin@kubernetes]
Darwin nmiculinics-MacBook-Pro.local 18.2.0 Darwin Kernel Version 18.2.0: Mon Nov 12 20:24:46 PST 2018; root:xnu-4903.231.4~2/RELEASE_X86_64 x86_64

What did you do?

 profiles (master) ✗ PPROF_BINARY_PATH=$(pwd)/bin go tool pprof -http :8080 kubelet-2.pprof
Ignoring local file /Users/lpp/Desktop/profiles/bin/kubelet: build-id mismatch (ccfe84cf2189a82ee5382ab4b869fc207f0a8a46 != )
Local symbolization failed for kubelet: open /usr/local/bin/kubelet: no such file or directory
Some binary filenames not available. Symbolization may be incomplete.
Try setting PPROF_BINARY_PATH to the search path for local binaries.

What did you expect to see?

pprof using arm compiler binary and getting the symbols.

What did you see instead?

pprof screams at me for build-id mismatch despite this binary being c/p from the device the profile was recorded on.

[nolanmar511]

• @kalyanac and possibly + @aalexand for additional insight.

I'm not familiar with this part of pprof's code.

For reference:

The "Ignoring local file" message comes from this line,
The message outputs the build ID from the profile first, and the build ID from the file second, so the file is missing the build ID.

The "Local symbolization failed" message comes from this line.

[kalyanac]

Hello,

Where the arm binary located on your mac? Is it: /Users/lpp/Desktop/profiles/bin/kubelet?

The error indicates that pprof was not able to read the build id from /Users/lpp/Desktop/profiles/bin/kubelet.

You can try passing the path to the correct binary as follows:
pprof /path/to/binary profile.pb.gz

You can manually check the build id using the following command:
readelf -a

Look for the following in the output:
Build ID
Displaying notes found in: .note.gnu.build-id

I havent tested this particular combination of reading arm binaries on a mac, and there are a number of operations in the symbolization pipeline that may have had problems with this combination. I will give it a try.

[aalexand]

Nit: instead of readelf -a you can do readelf -n to just print the notes.

[nmiculinic]

Where the arm binary located on your mac? Is it: /Users/lpp/Desktop/profiles/bin/kubelet?

yes

On the arm device:

debian@bbb-test:~$ readelf -n /usr/local/bin/kubelet

Displaying notes found in: .note.ABI-tag
  Owner                 Data size       Description
  GNU                  0x00000010       NT_GNU_ABI_TAG (ABI version tag)
    OS: Linux, ABI: 3.2.0

Displaying notes found in: .note.go.buildid
  Owner                 Data size       Description
  Go                   0x00000053       Unknown note type: (0x00000004)
   description data: 6b 77 38 59 36 49 69 50 35 6c 2d 44 6d 31 54 52 73 73 6a 39 2f 44 43 70 6b 42 70 65 6c 58 4b 79 4d 64 69 6d 67 2d 33 73 67 2f 79 6c 45 47 5f 5f 61 58 58 49 58 4f 45 35 52 49 4e 58 76 4c 2f 4c 4c 46 77 76 5a 55 44 4e 4f 45 2d 76 62 32 57 4e 64 57 64

Displaying notes found in: .note.gnu.build-id
  Owner                 Data size       Description
  GNU                  0x00000014       NT_GNU_BUILD_ID (unique build ID bitstring)
    Build ID: ccfe84cf2189a82ee5382ab4b869fc207f0a8a46

I cannot run readelf on the mac, this binary is c/p from the device.:

debian@bbb-test:~$ sha256sum /usr/local/bin/kubelet
4051e88174fedc0ea643466081ca461d9d175f714594dbe5208559fed0c4ae49  /usr/local/bin/kubelet

 profiles (master) ✗ sha256sum  /Users/lpp/Desktop/profiles/bin/kubelet                                          k8s [kubernetes-admin@kubernetes]
4051e88174fedc0ea643466081ca461d9d175f714594dbe5208559fed0c4ae49  bin/kubelet
 profiles (master) ✗                                                                k8s [kubernetes-admin@kubernetes]

[aalexand]

Would you be able to share the binary and the profile file?

[nmiculinic]

Sure, binary:

 Desktop (master) ✗ wget https://storage.googleapis.com/kubernetes-release/release/v1.13.3/bin/linux/arm/hyperkube -O kubelet    

 Desktop (master) ✗ sha256sum kubelet                                       
4051e88174fedc0ea643466081ca461d9d175f714594dbe5208559fed0c4ae49  kubelet

pprof:

(( it's not txt, but github only allows me to upload with certain extensions ))

kubelet-containerd.txt

[aalexand]

I seem to be able to open the profile against the binary just fine:

aalexand@mac:~ $ PPROF_BINARY_PATH=. pprof kubelet-containerd.pb.gz
File: kubelet
Build ID: ccfe84cf2189a82ee5382ab4b869fc207f0a8a46
Type: cpu
Time: Feb 26, 2019 at 10:53am (PST)
Duration: 2mins, Total samples = 11.99s ( 9.99%)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top
Showing nodes accounting for 6590ms, 54.96% of 11990ms total
Dropped 615 nodes (cum <= 59.95ms)
Showing top 10 nodes out of 290
      flat  flat%   sum%        cum   cum%
    2570ms 21.43% 21.43%     2790ms 23.27%  syscall.Syscall
    1250ms 10.43% 31.86%     1250ms 10.43%  runtime.futex
     690ms  5.75% 37.61%      720ms  6.01%  syscall.Syscall6
     500ms  4.17% 41.78%      500ms  4.17%  runtime._LostSIGPROFDuringAtomic64
     410ms  3.42% 45.20%      430ms  3.59%  runtime.usleep
     300ms  2.50% 47.71%      300ms  2.50%  runtime.epollctl
     280ms  2.34% 50.04%      280ms  2.34%  runtime.memclrNoHeapPointers
     250ms  2.09% 52.13%      420ms  3.50%  runtime.scanobject
     170ms  1.42% 53.54%      170ms  1.42%  runtime.memmove
     170ms  1.42% 54.96%     1070ms  8.92%  runtime.sysmon
(pprof) exit
aalexand@mac:~ $ ls -l kubelet*
-rw-r--r--@ 1 aalexand  primarygroup  154742504 Feb  1 13:22 kubelet
-rw-r--r--@ 1 aalexand  primarygroup      53071 Feb 27 00:30 kubelet-containerd.pb.gz

[aalexand]

Ah, but I do see the error when using go 1.12 "go tool pprof" rather than the standalone one:

aalexand@mac:~ $ go version
go version go1.12 darwin/amd64
aalexand@mac:~ $ PPROF_BINARY_PATH=. go tool pprof kubelet-containerd.pb.gz
Ignoring local file kubelet: build-id mismatch (ccfe84cf2189a82ee5382ab4b869fc207f0a8a46 != )
Local symbolization failed for kubelet: open /usr/local/bin/kubelet: no such file or directory
Some binary filenames not available. Symbolization may be incomplete.
Try setting PPROF_BINARY_PATH to the search path for local binaries.
File: kubelet
Build ID: ccfe84cf2189a82ee5382ab4b869fc207f0a8a46
Type: cpu
Time: Feb 26, 2019 at 10:53am (PST)
Duration: 2mins, Total samples = 11.99s ( 9.99%)
Entering interactive mode (type "help" for commands, "o" for options)

[kalyanac]

'go tool pprof' works as long as the binary is provided as an argument to pprof directly:

pprof-test: uname -a
Darwin  18.2.0 Darwin Kernel Version 18.2.0: Thu Dec 20 20:46:53 PST 2018; root:xnu-4903.241.1~1/RELEASE_X86_64 x86_64

pprof-test: go version
go version go1.12 darwin/amd64

pprof-test: which go
/usr/local/go/bin/go

pprof-test: file ./kubelet
./kubelet: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 3.2.0, BuildID[sha1]=ccfe84cf2189a82ee5382ab4b869fc207f0a8a46, stripped

pprof-test: file kubelet-containerd.pb.gz
kubelet-containerd.pb.gz: gzip compressed data, max speed, original size 138357

pprof-test: go tool pprof ./kubelet ./kubelet-containerd.pb.gz
File: kubelet
Build ID: ccfe84cf2189a82ee5382ab4b869fc207f0a8a46
Type: cpu
Time: Feb 26, 2019 at 10:53am (PST)
Duration: 2mins, Total samples = 11.99s ( 9.99%)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof)

It fails if PPROF_BINARY_PATH is provided.

(pprof) pprof-test:
pprof-test: PPROF_BINARY_PATH=. go tool pprof ./kubelet-containerd.pb.gz
Ignoring local file kubelet: build-id mismatch (ccfe84cf2189a82ee5382ab4b869fc207f0a8a46 != )
Local symbolization failed for kubelet: open /usr/local/bin/kubelet: no such file or directory
Some binary filenames not available. Symbolization may be incomplete.
Try setting PPROF_BINARY_PATH to the search path for local binaries.
File: kubelet
Build ID: ccfe84cf2189a82ee5382ab4b869fc207f0a8a46
Type: cpu
Time: Feb 26, 2019 at 10:53am (PST)
Duration: 2mins, Total samples = 11.99s ( 9.99%)
Entering interactive mode (type "help" for commands, "o" for options)

It seems to be unable to open the file in this case, as evidenced by the empty buildid on the right of the comparison in the error message.

I tried on linux and noticed another peculiarity - using PPROF_BINARY_PATH works, but providing binary as an argument to pprof doesnt work

$ pprof -symbolize=local ./kubelet ./kubelet-containerd.pb.gz
./kubelet: parsing profile: unrecognized profile format
Fetched 1 source profiles out of 2
Local symbolization failed for kubelet: stat /usr/local/bin/kubelet: no such file or directory
Some binary filenames not available. Symbolization may be incomplete.
Try setting PPROF_BINARY_PATH to the search path for local binaries.
File: kubelet
Build ID: ccfe84cf2189a82ee5382ab4b869fc207f0a8a46
Type: cpu
Time: Feb 26, 2019 at 10:53am (PST)
Duration: 2mins, Total samples = 11.99s ( 9.99%)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) ^D

$ PPROF_BINARY_PATH=. pprof ./kubelet-containerd.pb.gz
File: kubelet
Build ID: ccfe84cf2189a82ee5382ab4b869fc207f0a8a46
Type: cpu
Time: Feb 26, 2019 at 10:53am (PST)
Duration: 2mins, Total samples = 11.99s ( 9.99%)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) ^D

[kalyanac]

@nmiculinic could you try pprof from this repo instead of the one thats bundled with go distribution?

[nmiculinic]

@kalyanac

Tried it, it work:

PPROF_BINARY_PATH=$(pwd)/bin ~/go/bin/pprof -http :8080 kubelet.pprof

Close or?

[kalyanac]

glad to hear it worked. I will close the issue.