cmd/pprof: If the file under the current path contains a colon, go tool pprof will parse it into a url and will not work properly.

[cuishuang]

What version of Go are you using (go version)?

$ go version

go version devel go1.22-6dd7462a04 Fri Oct 13 17:10:31 2023 +0000 darwin/arm64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE='on'
GOARCH='arm64'
GOBIN=''
GOCACHE='/Users/myname/Library/Caches/go-build'
GOENV='/Users/myname/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='arm64'
GOHOSTOS='darwin'
GOINSECURE=''
GOMODCACHE='/Users/myname/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='darwin'
GOPATH='/Users/myname/go'
GOPRIVATE=''
GOPROXY=''
GOROOT='/Users/myname/mypath/go'
GOSUMDB='off'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/Users/myname/mypath/go/pkg/tool/darwin_arm64'
GOVCS=''
GOVERSION='devel go1.22-6dd7462a04 Fri Oct 13 17:10:31 2023 +0000' # I compiled it myself, it can be ignored and has nothing to do with the version.
GCCGO='gccgo'
AR='ar'
CC='clang'
CXX='clang++'
CGO_ENABLED='1'
GOMOD='/Users/myname/mypath/go/src/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/9t/839s3jmj73bcgyp5x_xh3gw00000gn/T/go-build3891298911=/tmp/go-build -gno-record-gcc-switches -fno-common'

What did you do?

I helped a colleague write a sample code to obtain the pprof file when the memory exceeds the threshold.

package main

import (
	"fmt"
	"os"
	"runtime"
	"runtime/pprof"
	"time"
)

func main() {

	ticker := time.NewTicker(10 * time.Second)

	go func() {
		for {
			time.Sleep(1e9)
			var m runtime.MemStats
			runtime.ReadMemStats(&m)

			memUsage := m.Sys / 1024 / 1024 

			if memUsage > 1024 {
				fmt.Printf("memory usage more than 1GB：%dMB\n", memUsage)

				// save pprof file
				filepath := "/xxx/yyy/path/mem" + "." + strconv.Itoa(int(time.Now().Unix()))

				f, err := os.Create(filepath)
				if err != nil {
					fmt.Println("err1:", err)
				}
				defer f.Close()

				if err2 := pprof.WriteHeapProfile(f); err2 != nil {
					fmt.Println("err2:", err2)
				}

				fmt.Println("pprof data has saved in:", filepath)

			} else {
				fmt.Printf(memory usage is：%dMB\n", memUsage)
			}
		}
	}()

	for {
		select {
		case <-ticker.C:
		
			var a = make([]byte, 1073741824)
			_ = a
			fmt.Println("ticker：", time.Now())
		}
	}

	select {}

}

I tested it locally and everything was fine. He made certain modifications and then released it to the production environment. When the memory surge occurred, this program obtained the pprof file as expected, but what was confusing and surprising was that when using go tool pprof thefile, an error was reported:

Fetching profile over HTTP from http://mem_2023-11-01_14:23:16/debug/pprof/profile

profile.pprof.2023-11-03_14:23:16: Get "http://mem_2023-11-01_14:23:16/debug/pprof/profile": dial tcp: lookup mem_2023-11-01_14:23: no such host

I spent a lot of time trying to research, and one of his modifications was to change the timestamp in the file path to time.Now().Format("2006-01-02_15:04:05") in order to be more intuitive.

That is

filepath := "/xxx/yyy/path/mem" + "." + strconv.Itoa(int(time.Now().Unix())) ---> filepath := "/xxx/yyy/path /mem" + "." + time.Now().Format("2006-01-02_15:04:05")

After adding debug information several times, it was found that it was the adjustURL method in internal/driver/fetch.go. Because the file name contained a colon, url.Parse parsed the file path into a url. . .

This is confusing and counter-intuitive. After many experiments, I think we can consider determining whether there is a file with the same name in the directory. If so, treat it as a file instead of a URL.

(We only need to consider the case of executing go tool pprof in the directory where the profile is located. In the case of go tool pprof a/b/mem_pprof_file_2006_01_02_15:04:05, it will not be recognized as a URL now)
​

What did you expect to see?

I want to be able to use pprof normally without renaming the file

What did you see instead?

Fetching profile over HTTP from http://mem_2023-11-01_14:23:16/debug/pprof/profile

profile.pprof.2023-11-03_14:23:16: Get "http://mem_2023-11-01_14:23:16/debug/pprof/profile": dial tcp: lookup mem_2023-11-01_14:23: no such host

[gopherbot]

Change https://go.dev/cl/539595 mentions this issue: cmd/pprof: fix exception when file type contains colon

[cuishuang]

To fix this problem, we also need to update google/pprof. I have already raised an issue google/pprof#816

[rhysh]

I don't see which specific go tool pprof [...] command you ran that didn't work.

Have you tried prefixing the file with ./ to specify that it's a file in the local directory?

Here's how it works (and fails) on my machine. First creating a small demo profile:

$ go1.21 version                                             
go version go1.21.3 darwin/arm64

$ go1.21 test context -o /dev/null -memprofile './localhost:8080'
ok  	context	0.366s

$ ls -al ./localhost:8080
-rw-r--r--  1 rhys  wheel  350 Nov  3 06:24 ./localhost:8080

Then triggering what I think is the behavior you're describing:

$ go1.21 tool pprof -top localhost:8080                        
Fetching profile over HTTP from http://localhost:8080/debug/pprof/profile
localhost:8080: Get "http://localhost:8080/debug/pprof/profile": dial tcp [::1]:8080: connect: connection refused
failed to fetch any source profiles

Then working around that by adding a ./ prefix to the file name:

$ go1.21 tool pprof -top ./localhost:8080                      
Type: alloc_space
Time: Nov 3, 2023 at 6:24am (PDT)
Showing nodes accounting for 512.20kB, 100% of 512.20kB total
      flat  flat%   sum%        cum   cum%
  512.20kB   100%   100%   512.20kB   100%  runtime.malg
         0     0%   100%   512.20kB   100%  runtime.newproc.func1
         0     0%   100%   512.20kB   100%  runtime.newproc1
         0     0%   100%   512.20kB   100%  runtime.systemstack

[cherrymui]

As @rhysh pointed out, it may be hard to for the tool to distinguish a file named something like localhost:8080 and a URL in general. If adding a leading ./ works, I think that is a reasonable workaround to disambiguate. Does this work for you? Thanks.

[cuishuang]

As @rhysh pointed out, it may be hard to for the tool to distinguish a file named something like localhost:8080 and a URL in general. If adding a leading ./ works, I think that is a reasonable workaround to disambiguate. Does this work for you? Thanks.

I think from usage habits, when the target source is a file in the current directory, many developers are likely to omit ./

The current processing method I modified is that if the file contains a colon and will be processed by url.Parse, check whether a file with the same name exists before requesting. If there is no localhost:8080 file locally, it will be treated as a URL normally. But if a file named localhost:8080 does exist, I think the user is more likely to want to handle it as a file.

I don't see which specific go tool pprof [...] command you ran that didn't work.

Have you tried prefixing the file with ./ to specify that it's a file in the local directory?

Here's how it works (and fails) on my machine. First creating a small demo profile:

$ go1.21 version                                             
go version go1.21.3 darwin/arm64

$ go1.21 test context -o /dev/null -memprofile './localhost:8080'
ok  	context	0.366s

$ ls -al ./localhost:8080
-rw-r--r--  1 rhys  wheel  350 Nov  3 06:24 ./localhost:8080

Then triggering what I think is the behavior you're describing:

$ go1.21 tool pprof -top localhost:8080                        
Fetching profile over HTTP from http://localhost:8080/debug/pprof/profile
localhost:8080: Get "http://localhost:8080/debug/pprof/profile": dial tcp [::1]:8080: connect: connection refused
failed to fetch any source profiles

Then working around that by adding a ./ prefix to the file name:

$ go1.21 tool pprof -top ./localhost:8080                      
Type: alloc_space
Time: Nov 3, 2023 at 6:24am (PDT)
Showing nodes accounting for 512.20kB, 100% of 512.20kB total
      flat  flat%   sum%        cum   cum%
  512.20kB   100%   100%   512.20kB   100%  runtime.malg
         0     0%   100%   512.20kB   100%  runtime.newproc.func1
         0     0%   100%   512.20kB   100%  runtime.newproc1
         0     0%   100%   512.20kB   100%  runtime.systemstack

Thank you for taking the time to test and reply. But I think from usage habits, when the target source is a file in the current directory, many developers are likely to omit ./

[cuishuang]

As @rhysh pointed out, it may be hard to for the tool to distinguish a file named something like localhost:8080 and a URL in general. If adding a leading ./ works, I think that is a reasonable workaround to disambiguate. Does this work for you? Thanks.

Thank you very much for your reply. I think from usage habits, when the target source is a file in the current directory, many developers are likely to omit ./

The current processing method I modified is that if the file contains a colon and will be processed by url.Parse, check whether a file with the same name exists before requesting. If there is no localhost:8080 file locally, it will be treated as a URL normally. But if a file named localhost:8080 does exist, I think the user is more likely to want to handle it as a file. (Of course, in most cases the file name is unlikely to be localhost:8080, but like mem_2023-11-01_01:01:00. If this type is processed as a URL, it is best to avoid it)

Since google/pprof also requires almost the same changes, I am also paying attention to the suggestions of the maintainers of this project. In addition, the two fetch methods in go and google/pprof have almost the same logical judgment, which seems to be able to streamline some code.

[cuishuang]

To fix this problem, we also need to update google/pprof. I have already raised an issue google/pprof#816

Merged

[cuishuang]

As @rhysh pointed out, it may be hard to for the tool to distinguish a file named something like localhost:8080 and a URL in general. If adding a leading ./ works, I think that is a reasonable workaround to disambiguate. Does this work for you? Thanks.

Relevant changes have been merged into google/pprof.

According to communication and suggestions with the pprof maintainer, if src is a file that exists in the current directory, it will be treated as a file (even if it contains a colon).

For details please refer to google/pprof#817

I think it's time to move on to the next step.

But I feel that this CL will change the relevant logic of pprof to a certain extent (although it is very small). Should I submit a proposal and request the Go team for review?

Thanks.