Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Breakpoint on absolute path not working #1730

Closed
Meai1 opened this issue Oct 23, 2019 · 9 comments
Closed

Breakpoint on absolute path not working #1730

Meai1 opened this issue Oct 23, 2019 · 9 comments

Comments

@Meai1
Copy link

Meai1 commented Oct 23, 2019

Start listening for debug clients:

dlv --listen=:2345 --headless=true --api-version=2 --accept-multiclient --wd=/home/pc/workarea/projects/GOPATH/src/k8s.io/kubernetes exec /home/pc/.cache/bazel/_bazel_pc/6a1773eb858882fef994ad8c7782523f/execroot/io_k8s_kubernetes/bazel-out/k8-dbg/bin/cmd/kube-apiserver/linux_amd64_pure_debug/kube-apiserver
API server listening at: [::]:2345

Connect:

dlv --api-version 2 connect localhost:2345
(dlv) b cmd/kube-apiserver/apiserver.go:32
Breakpoint 1 set at 0x468b101 for main.main() cmd/kube-apiserver/apiserver.go:32
(dlv) b /home/pc/workarea/projects/GOPATH/src/k8s.io/kubernetes/cmd/kube-apiserver/apiserver.go:34
Command failed: Location "/home/pc/workarea/projects/GOPATH/src/k8s.io/kubernetes/cmd/kube-apiserver/apiserver.go:34" not found
`

So the issue is that /home/pc/workarea/projects/GOPATH/src/k8s.io/kubernetes/cmd/kube-apiserver/apiserver.go definitely exists but it still cant set a breakpoint on it for some reason. The problem is that vscode sets paths via absolute path, so it has to work. I think Goland IDE does that too.

To reproduce, the kubernetes software in question was built by checking out the kubernetes trunk and running bazel build --compilation_mode=dbg //cmd/kube-apiserver:kube-apiserver
The resulting binary is then in a deep nested folder as you've seen above: /home/pc/.cache/bazel/_bazel_pc/6a1773eb858882fef994ad8c7782523f/execroot/io_k8s_kubernetes/bazel-out/k8-dbg/bin/cmd/kube-apiserver/linux_amd64_pure_debug/kube-apiserver

btw my GOPATH folder is actually named GOPATH, it's not a shell expansion problem in case you wondered.

dlv version 1.3.2
go version 1.7.4

My suggestion: Could you maybe add some more detailed logging to the error "Location not found"? Is the file not found? Surely it has to be found, it's right there, the path is absolute. So how can it not be found, there has to be something else going on here.

@derekparker
Copy link
Member

Hm, I agree that the error message could be more descriptive here as to what actually went wrong when looking up the location.

However, is there any actual code at the location you're trying to set a breakpoint at? What if you use the absolute path and line 32 like the breakpoint above which succeeded?

@Meai1
Copy link
Author

Meai1 commented Oct 24, 2019

yeah line 34 has some code on it. I didnt change anything in kubernetes, here: https://github.com/kubernetes/kubernetes/blob/2905275a08f8e3999e7e6bc8b02f407fd6a0ff36/cmd/kube-apiserver/apiserver.go#L34

@dlsniper
Copy link
Contributor

The way your code is built might affect this (paths and all). Let me know if you tried from GoLand 2019.2.3 (since you mentioned the IDE), and if it fails. The IDE itself has some heuristics to figure out where to put the breakpoint and the path matching is automatic.

@Meai1
Copy link
Author

Meai1 commented Oct 24, 2019

I am using Goland IDE EAP 2019.3 it fails the "same" (=I use the Go Remote configuration) as what I'm doing here on the command line, I only tried the command line stuff because Goland wasnt working and then I switched to VsCode and that was showing me the actual error that the breakpoint cant be set, which I didnt know from inside Goland. I put "same" into quotes because I have no idea what Goland is doing internally, I'm not seeing any output in the console window of Goland IDE.

Or did you mean whether the non-bazel build&debug process works in Goland? Yes it does but that isnt useful to me because it takes too long to rebuild constantly even when I changed nothing, whereas with Bazel I can reuse a build.

Also btw regarding Delve it's very hard to get out of the delve command line when it's listening after it got a connection... ctrl+c doesnt work, ctrl+d doesnt, ctrl+z doesnt. I always have to close the whole terminal window.

@aarzilli
Copy link
Member

My suggestion: Could you maybe add some more detailed logging to the error "Location not found"? Is the file not found? Surely it has to be found, it's right there, the path is absolute.

The problem is almost certainly that the file can not be found. I bet bazel just strips the absolute paths from the executable, you can verify it by giving delve the command sources apiserver and looking at what's returned.

If the file had been found the error would be different and would tell you specifically about the line not containing a statement. In fact the error that you get from IDEs should also be different given what API they are using, and I bet it tells you that it couldn't find the file explicitly. The reason the command line client doesn't tell you about files is because break takes a location expression and if it can't find a file it can't decide whether you were trying to set a function breakpoint or a file breakpoint.

Surely it has to be found, it's right there, the path is absolute.

Delve doesn't look on disk, it looks through the debug symbols inside the executable file. I think what's happening is that bazel is stripping the path prefix.

Besides checking the output of the sources command you can also see this from the output of break when it works:

(dlv) b cmd/kube-apiserver/apiserver.go:32
Breakpoint 1 set at 0x468b101 for main.main() cmd/kube-apiserver/apiserver.go:32

if the file had an absolute path it would show you something that either starts with / or with ./.

The solution is that your IDE and your build system need to agree on how they are going to tell the debugger how to locate files.

@Meai1
Copy link
Author

Meai1 commented Oct 24, 2019

Ok but I cant exactly change how these IDEs want to do their debugging and I dont see any option for bazel+go to change the debug symbol paths either, at least I havent found anything with google searches just now.
It would be much easier and flexible if Delve had an option for me to specify a prefix or search path like GDB does. In GDB if it cant find the sources, I usually just add the project directory to the search path and it immediately starts working. I already tried the substitute-path option in ~/.config/dlv/config.yml but I realized that this wont ever work because it just rewrites the stuff that's already inside the binary, it cant help change what is coming in from VsCode or Goland.

EDIT:
oh and just to confirm, yeah it does. These are the paths in the binary:

(dlv) sources apiserver.go
cmd/kube-apiserver/apiserver.go
pkg/registry/admissionregistration/rest/storage_apiserver.go
staging/src/k8s.io/apiextensions-apiserver/pkg/apiserver/apiserver.go
staging/src/k8s.io/apiserver/pkg/server/genericapiserver.go
etc.

@dlsniper
Copy link
Contributor

GoLand looks at whatever the paths are in the binary and based on the configuration of the project, it can infer the paths to match the debugged target when debugging.

By the sounds of it, something went wrong with the heuristics in this case.

However, I don't know how the Bazel part work, but I'll ask my colleagues, @zolotov and @nd to have a look at it as well.

I know there's a Bazel plugin, https://plugins.jetbrains.com/plugin/8609-bazel/, which might be able to help you in this case.

@aarzilli
Copy link
Member

aarzilli commented Oct 24, 2019

It would be much easier and flexible if Delve had an option for me to specify a prefix or search path like GDB does. In GDB if it cant find the sources, I usually just add the project directory to the search path and it immediately starts working

That would be substitute-path, but what you want is the opposite, you want something in delve that removes a prefix from locations requested by IDEs.

Ok but I cant exactly change how these IDEs want to do their debugging and I dont see any option for bazel+go to change the debug symbol paths either

I think you are experiencing a failure in the I part of IDE.

@nd
Copy link
Contributor

nd commented Nov 5, 2019

@Meai1 we improved path mapping in GoLand 2019.2.4 which will be released in a week or so. Please give it a try and file an issue if it doesn't work as expected.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants