Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/pprof: Local symbolization failed for http.test: open /tmp/go-build348974326/b001/http.test: no such file or directory #25743

Closed
bradfitz opened this issue Jun 5, 2018 · 10 comments
Assignees
Milestone

Comments

@bradfitz
Copy link
Contributor

@bradfitz bradfitz commented Jun 5, 2018

I was just profiling something like normal and pprof gave me some angry red text about something not working, but it seems to work. I'm not sure what's broken. @hyangah?

screen shot 2018-06-05 at 1 43 04 pm

bradfitz@gdev:~/go/src$ ./make.bash  
Building Go cmd/dist using /home/bradfitz/go1.4.  
Building Go toolchain1 using /home/bradfitz/go1.4.  
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.  
Building Go toolchain2 using go_bootstrap and Go toolchain1.  
Building Go toolchain3 using go_bootstrap and Go toolchain2.  
Building packages and commands for linux/amd64.  
---  
Installed Go for linux/amd64 in /home/bradfitz/go  
Installed commands in /home/bradfitz/go/bin  


bradfitz@gdev:~/go/src$ cd net/http/  
bradfitz@gdev:~/go/src/net/http$ go test -v -run=NONE -bench=ServeMux -cpuprofile=prof.cpu  
goos: linux  
goarch: amd64  
pkg: net/http  
BenchmarkServeMux-4        10000            113109 ns/op           19907 B/op        540 allocs/op  
PASS  
ok      net/http        1.310s  


bradfitz@gdev:~/go/src/net/http$ go tool pprof prof.cpu  
Local symbolization failed for http.test: open /tmp/go-build348974326/b001/http.test: no such file or directory  
Some binary filenames not available. Symbolization may be incomplete.  
Try setting PPROF_BINARY_PATH to the search path for local binaries.  
File: http.test  
Type: cpu  
Time: Jun 5, 2018 at 8:41pm (UTC)  
Duration: 1.30s, Total samples = 1.17s (89.82%)  
Entering interactive mode (type "help" for commands, "o" for options)  
(pprof) top  
Showing nodes accounting for 630ms, 53.85% of 1170ms total  
Showing top 10 nodes out of 95  
      flat  flat%   sum%        cum   cum%  
     110ms  9.40%  9.40%      270ms 23.08%  runtime.mallocgc  
      80ms  6.84% 16.24%      230ms 19.66%  runtime.concatstrings  
      70ms  5.98% 22.22%     1130ms 96.58%  net/http_test.BenchmarkServeMux  
      70ms  5.98% 28.21%      100ms  8.55%  runtime.mapaccess2_faststr  
      60ms  5.13% 33.33%       60ms  5.13%  runtime.heapBitsSetType  
      60ms  5.13% 38.46%       60ms  5.13%  runtime.memmove  
      60ms  5.13% 43.59%       60ms  5.13%  sync.(*RWMutex).RUnlock  
      40ms  3.42% 47.01%      750ms 64.10%  net/http.(*ServeMux).Handler  
      40ms  3.42% 50.43%       70ms  5.98%  path.Clean  
      40ms  3.42% 53.85%      270ms 23.08%  runtime.concatstring2  
(pprof) ^D

bradfitz@gdev:~/go/src/net/http$ go version  
go version devel +e7ee3b91c4 Tue Jun 5 20:00:37 2018 +0000 linux/amd64  
@bradfitz bradfitz added this to the Go1.11 milestone Jun 5, 2018
@hyangah
Copy link
Contributor

@hyangah hyangah commented Jun 5, 2018

I don't think this is a new error but became more noticeable because of the new terminal support (with color). I happened to ask the same question in google/pprof#386 today.

@aalexand @rauls5382 @rsc

The error was from https://github.com/google/pprof/blob/master/internal/symbolizer/symbolizer.go#L343
while processing the mapping data. Currently, runtime/pprof doesn't set has_functions/filenames/line_numbers fields (https://go.googlesource.com/go/+/master/src/runtime/pprof/proto.go#182) even though we do symbolization (for Go symbols). My understanding is that we didn't set them because we don't know what values to set there - especially when non-Go part is mixed in.

Should we set these values based on the collected samples? (if all pcs from the mapping were successfully symbolized, set those fields)

@aalexand
Copy link
Contributor

@aalexand aalexand commented Jun 6, 2018

Setting those fields may prevent locations in the C++ code from being symbolized, there is discussion on b/36645883 which I believe relevant to this.

@aalexand
Copy link
Contributor

@aalexand aalexand commented Jun 6, 2018

Sorry, I missed that you already mentioned what I said. Setting the flags only if all pcs are symbolized sounds reasonable.

@gopherbot
Copy link

@gopherbot gopherbot commented Jun 12, 2018

Change https://golang.org/cl/118275 mentions this issue: runtime/pprof: set HasFunctions of mapping entries

@hyangah
Copy link
Contributor

@hyangah hyangah commented Jun 12, 2018

I sent cl/118275 for review. I am not sure whether it's for 1.11.

@gopherbot gopherbot closed this in 7b2f55d Jun 13, 2018
dna2github added a commit to dna2fork/go that referenced this issue Jun 14, 2018
The pprof tool utilizes attributes of mapping entries
such as HasFunctions to determine whether the profile
includes necessary symbol information.
If none of the attributes is set, pprof tool tries to
read the corresponding binary to use for local symbolization.
If the binary doesn't exist, it prints out error messages.

Go runtime generated profiles without any of the attributes
set so the pprof tool always printed out the error messages.
The error messages became more obvious with the new
terminal support that uses red color for error messages.

Go runtime can symbolize all Go symbols and generate
self-contained profile for pure Go program. Thus, there
is no reason for the pprof tool to look for the copy of
the binary. So, this CL sets one of the attributes
(HasFunctions) true if all PCs in samples look fully
symbolized.

For non-pure Go program, however, it's possible that
symbolization of non-Go PCs is incomplete. In this case,
we need to leave the attributes all false so pprof can attempt
to symbolize using the local copy of the binary if available.
It's hard to determine whether a mapping includes non-Go
code. Instead, this CL checks PCs from collected samples.
If unsuccessful symbolization is observed, it skips setting
the HasFunctions attribute.

Fixes golang#25743

Change-Id: I5108be45bbc37ab486d145fa03e7ce37d88fad50
Reviewed-on: https://go-review.googlesource.com/118275
Run-TryBot: Hyang-Ah Hana Kim <hyangah@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@gopherbot
Copy link

@gopherbot gopherbot commented Jul 13, 2018

Change https://golang.org/cl/123779 mentions this issue: runtime/pprof: add a fake mapping when /proc/self/maps is unavailable

gopherbot pushed a commit that referenced this issue Jul 16, 2018
Profile's Mapping field is currently populated by reading /proc/self/maps.
On systems where /proc/self/maps is not available, the profile generated
by Go's runtime will not have any Mapping entry. Pprof command then adds
a fake entry and links all Location entries in the profile with the fake
entry to be used during symbolization.
https://github.com/google/pprof/blob/a8644067d5a3c9a6386e7c88fa4a3d9d37877ca3/internal/driver/fetch.go#L437

The fake entry is not enough to suppress the error or warning messages
pprof command produces. We need to tell pprof that Location entries are
symbolized already by Go runtime and pprof does not have to attempt to
perform further symbolization.

In #25743, we made Go runtime mark Mapping entries with HasFunctions=true
when all Location entries from the Mapping entries are successfully
symbolized. This change makes the Go runtime add a fake mapping entry,
otherwise the pprof command tool would add, and set the HasFunctions=true
following the same logic taken when the real mapping information is
available.

Updates #19790.
Fixes #26255. Tested pprof doesn't report the error message any more
for pure Go program.

Change-Id: Ib12b62e15073f5d6c80967e44b3e8709277c11bd
Reviewed-on: https://go-review.googlesource.com/123779
Run-TryBot: Hyang-Ah Hana Kim <hyangah@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@jsha
Copy link

@jsha jsha commented Oct 29, 2018

Note: If anybody is getting this error and wants to suppress it, you can add -symbolize=remote to your pprof invocation, which will use the remote symbols instead of trying to find them locally.

@hyangah
Copy link
Contributor

@hyangah hyangah commented Oct 29, 2018

@jsha That is only when the remote server (the profile origin) or any symbolization service the pprof can utilize is available.
This bug is marked fixed. If you still experience an issue, please file a bug soon.

@jsha
Copy link

@jsha jsha commented Oct 29, 2018

Thanks @hyangah! Maybe what I'm actually looking for for a quick fix is -symbolize=none?

I'm seeing the same error described here, in go version go1.11.1 linux/amd64. My service does include some cgo code, but I don't usually care about symbolizing that code.

Do you think that qualifies enough as "still reproduces" that I should file a separate bug? Or is that working as intended?

@hyangah
Copy link
Contributor

@hyangah hyangah commented Oct 30, 2018

I think that's working as intended.
Go doesn't know how to symbolize symbols outside Go unless it's taught how (see https://golang.org/pkg/runtime/#SetCgoTraceback).

On the other hand, I wonder how -symbolize=none makes pprof symbolize non-Go symbols.
If it just silences the warning message from the pprof tool without actually symbolizing non-Go symbols, I think that's a bug (or insufficient error/warning message) in the pprof tool.

@golang golang locked and limited conversation to collaborators Oct 30, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
5 participants
You can’t perform that action at this time.