Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

VS Code DAP does not map trimmed paths #2754

Closed
SteVwonder opened this issue Oct 21, 2021 · 19 comments
Closed

VS Code DAP does not map trimmed paths #2754

SteVwonder opened this issue Oct 21, 2021 · 19 comments

Comments

@SteVwonder
Copy link

  1. What version of Delve are you using (dlv version)?
❯ dlv version
Delve Debugger
Version: 1.7.2
Build: $Id: 5b6f24e7fbcad3fe3bc574d3763b2f20afa1d6a1 $
  1. What version of Go are you using? (go version)?
go version go1.16.4 linux/amd64
  1. What operating system and processor architecture are you using?

Windows 10 on x86_64 but running inside of WSL2 via VS Code's remote extension

  1. What did you do?

Ran a go binary under delve via VS Code's debugging pane/tooling with the Go extension installed. I stopped the execution of the program, and then clicked on a thread within the call stack frame. That opened up the call stack, and then I clicked on a particular stack frame.

  1. What did you expect to see?

I expected to see the source code of the file containing the function for the stack frame that I clicked on.

  1. What did you see instead?

A new source window in VS Code with the correct filename, but the contents were an error message from VS Code/delve:

Could not load source 'src/common/retryer/retry.go': Unsupported command: cannot process "source" request.

Here is a sanitized version of the trace from VS Code of the DAP communication:

[Trace - 08:39:49.662] client -> {"command":"scopes","arguments":{"frameId":1021},"type":"request","seq":15}

[Trace - 08:39:49.666] client  <- {"seq":0,"type":"response","request_seq":15,"success":true,"command":"scopes","body":{"scopes":[{"name":"Locals (warning: optimized function)","variablesReference":1005,"expensive":false,"source":{}}]}}

[Trace - 08:39:49.722] client -> {"command":"variables","arguments":{"variablesReference":1005},"type":"request","seq":16}

[Trace - 08:39:49.725] client  <- {"seq":0,"type":"response","request_seq":16,"success":true,"command":"variables","body":{"variables" <snipped>

[Trace - 08:45:19.434] client -> {"command":"source","arguments":{"sourceReference":0,"source":{"name":"retry.go","path":"src/common/retryer/retry.go"}},"type":"request","seq":17}

[Trace - 08:45:19.436] client  <- {"seq":0,"type":"response","request_seq":17,"success":false,"command":"source","message":"Unsupported command","body":{"error":{"id":9999,"format":"Unsupported command: cannot process \"source\" request","showUser":false}}}

I'm not familiar with the delve codebase, but I believe the issue is here in the dap server code. It is responding with an unsupported error to all source requests. Curiously, the comment mentions that this command doesn't make sense for Go since strings cannot be eval'd at runtime, but the DAP Source Request is not asking for code/commands to be source'd but instead for the debugger to return a particular piece of source code.

@suzmue
Copy link
Contributor

suzmue commented Oct 21, 2021

Could you include your launch / attach request from the logs as well? If you are using attach or launch exec, could you also include the build command that was used to build the executable you are debugging?

@SteVwonder
Copy link
Author

Could you include your launch / attach request from the logs as well?

Sure! Below

[Verbose - 10:44:42.564] Config: {"name":"Launch //src/services/controller","type":"go","request":"launch","mode":"exec","sourceDirectories":["/home/sherbein/Repositories/nuc/"],"program":"/home/sherbein/Repositories/nuc/bazel-bin/src/services/controller/controller","env":{},"args":["server","--port=8080","--tls.cert-file=\"/home/sherbein/certs/localhost.crt\"","--tls.key-file=\"/home/sherbein/certs/localhost.key\"","--with-agent"],"cwd":"/home/sherbein/Repositories/nuc","showLog":true,"substitutePath":[{"from":"/home/sherbein/Repositories/nuc/bazel-bin","to":"/home/sherbein/.cache/bazel/_bazel_sherbein/2d91ec193e823eeed6c484b1ac0ffa6d/execroot/git_nuc/bazel-out/k8-dbg/bin"}],"trace":"verbose","__configurationTarget":5,"packagePathToGoModPathMap":{"/home/sherbein/Repositories/nuc/src/common/retryer":"/home/sherbein/Repositories/nuc","/home/sherbein/Repositories/nuc/src/common/registry":"/home/sherbein/Repositories/nuc","/home/sherbein/Repositories/nuc/src/common/httputil":"/home/sherbein/Repositories/nuc","/home/sherbein/Repositories/nuc/src/common/mag/util":"/home/sherbein/Repositories/nuc","/home/sherbein/Repositories/nuc/src/common/mag/cli":"/home/sherbein/Repositories/nuc","/home/sherbein/Repositories/nuc/src/common/jwtutil":"/home/sherbein/Repositories/nuc","/home/sherbein/Repositories/nuc/bazel-nuc/external/go_sdk/src/net/http":"/home/sherbein/.cache/bazel/_bazel_sherbein/2d91ec193e823eeed6c484b1ac0ffa6d/external/go_sdk/src"},"debugAdapter":"dlv-dap","apiVersion":2,"dlvLoadConfig":{"followPointers":true,"maxVariableRecurse":1,"maxStringLen":64,"maxArrayValues":64,"maxStructFields":-1},"showGlobalVariables":false,"dlvToolPath":"/home/sherbein/.nuc/bazel-gopath/bin/dlv-dap"}
[Verbose - 10:44:42.567] session add5d9e3-50f6-44ae-a690-2eb0b00dc49f will start with {"name":"Launch //src/services/controller","type":"go","request":"launch","mode":"exec","sourceDirectories":["/home/sherbein/Repositories/nuc/"],"program":"/home/sherbein/Repositories/nuc/bazel-bin/src/services/controller/controller","env":{},"args":["server","--port=8080","--tls.cert-file=\"/home/sherbein/certs/localhost.crt\"","--tls.key-file=\"/home/sherbein/certs/localhost.key\"","--with-agent"],"cwd":"/home/sherbein/Repositories/nuc","showLog":true,"substitutePath":[{"from":"/home/sherbein/Repositories/nuc/bazel-bin","to":"/home/sherbein/.cache/bazel/_bazel_sherbein/2d91ec193e823eeed6c484b1ac0ffa6d/execroot/git_nuc/bazel-out/k8-dbg/bin"}],"trace":"verbose","__configurationTarget":5,"packagePathToGoModPathMap":{"/home/sherbein/Repositories/nuc/src/common/retryer":"/home/sherbein/Repositories/nuc","/home/sherbein/Repositories/nuc/src/common/registry":"/home/sherbein/Repositories/nuc","/home/sherbein/Repositories/nuc/src/common/httputil":"/home/sherbein/Repositories/nuc","/home/sherbein/Repositories/nuc/src/common/maglev/util":"/home/sherbein/Repositories/nuc","/home/sherbein/Repositories/nuc/src/common/maglev/cli":"/home/sherbein/Repositories/nuc","/home/sherbein/Repositories/nuc/src/common/jwtutil":"/home/sherbein/Repositories/nuc","/home/sherbein/Repositories/nuc/bazel-nuc/external/go_sdk/src/net/http":"/home/sherbein/.cache/bazel/_bazel_sherbein/2d91ec193e823eeed6c484b1ac0ffa6d/external/go_sdk/src"},"debugAdapter":"dlv-dap","apiVersion":2,"dlvLoadConfig":{"followPointers":true,"maxVariableRecurse":1,"maxStringLen":64,"maxArrayValues":64,"maxStructFields":-1},"showGlobalVariables":false,"dlvToolPath":"/home/sherbein/.nuc/bazel-gopath/bin/dlv-dap"}
[Trace - 10:44:42.570] client  <- {"seq":0,"type":"event","event":"output","body":{"category":"console","output":"Starting: /home/sherbein/.nuc/bazel-gopath/bin/dlv-dap dap --check-go-version=false --listen=127.0.0.1:44355 --log=true --log-dest=3 from /home/sherbein/Repositories/nuc\n"}}

If you are using attach or launch exec, could you also include the build command that was used to build the executable you are debugging?

It was built with bazel (and I'll admit to be very unfamiliar with the ecosystem). I re-ran bazel with ./bazel build --subcommands -c dbg //src/services/controller and grabbed the last two subcommands (the build of the service static lib and the linking of the service binary). Let me know if you wanted something different (I also lightly redacted the build/link lines).

SUBCOMMAND: # //src/services/controller:controller [action 'GoCompilePkg src/services/controller/controller.a', configuration: a3f7203d89c25af4aef6b71d216c4f38ff51ec21041f5c45e9696e0e435d3774, execution platform: @local_config_platform//:host]
(cd /home/sherbein/.cache/bazel/_bazel_sherbein/2d91ec193e823eeed6c484b1ac0ffa6d/execroot/git_nuc && \
  exec env - \
    CGO_ENABLED=1 \
    GOARCH=amd64 \
    GOOS=linux \
    GOPATH='' \
    GOROOT=external/go_sdk \
    GOROOT_FINAL=GOROOT \
    PATH=/usr/bin:/bin \
  bazel-out/host/bin/external/go_sdk/builder compilepkg -sdk external/go_sdk -installsuffix linux_amd64 -src src/services/controller/main.go -src src/services/controller/server.go -arc 'git.example.com/nuc/napis/mag/udc/v1=git.example.com/nuc/napis/mag/udc/v1=bazel-out/k8-dbg/bin/napis/mag/udc/v1/udc_go_proto.x' -arc 'git.example.com/nuc/napis/mag/volumes/v1=git.example.com/nuc/napis/mag/volumes/v1=bazel-out/k8-dbg/bin/napis/mag/volumes/v1/volumes_go_proto.x' -arc 'git.example.com/nuc/src/common/decider=git.example.com/nuc/src/common/decider=bazel-out/k8-dbg/bin/src/common/decider/decider.x' <snip lots of -arc paths> -importpath git.example.com/nuc/src/services/controller -p main -package_list bazel-out/host/bin/external/go_sdk/packages.txt -o bazel-out/k8-dbg/bin/src/services/controller/controller.a -x bazel-out/k8-dbg/bin/src/services/controller/controller.x -nogo bazel-out/k8-opt-exec-2B5CBBC6/bin/nogo_nuc_/nogo_nuc -gcflags '' -asmflags '')
SUBCOMMAND: # //src/services/controller:controller [action 'GoLink src/services/controller/controller', configuration: a3f7203d89c25af4aef6b71d216c4f38ff51ec21041f5c45e9696e0e435d3774, execution platform: @local_config_platform//:host]
(cd /home/sherbein/.cache/bazel/_bazel_sherbein/2d91ec193e823eeed6c484b1ac0ffa6d/execroot/git_nuc && \
  exec env - \
    CGO_ENABLED=1 \
    GOARCH=amd64 \
    GOOS=linux \
    GOPATH='' \
    GOROOT=external/go_sdk \
    GOROOT_FINAL=GOROOT \
    PATH=/usr/bin:/bin \
  bazel-out/host/bin/external/go_sdk/builder '-param=bazel-out/k8-dbg/bin/src/services/controller/controller-0.params' -- -extld /usr/bin/gcc '-buildid=redacted' -extldflags '-fuse-ld=gold -Wl,-no-as-needed -Wl,-z,relro,-z,now -B/usr/bin -pass-exit-codes -lm')
Target //src/services/controller:controller up-to-date:
  bazel-bin/src/services/controller/controller

@suzmue
Copy link
Contributor

suzmue commented Oct 28, 2021

I would suggest adding {"/home/sherbein/Repositories/nuc/src/": "src/"} to your "substitutePath" configuration, to see if that helps with displaying the correct source. The path that delve is sending back is "src/common/retryer/retryer.go" and assuming that the file you have locally is at "/home/sherbein/Repositories/nuc/src/common/retryer/retryer.go" this should help map it.

@SteVwonder
Copy link
Author

That unfortunately did not fix my issue. I'm working on creating a simpler reproducer with an open-source project. Will report back if I am successful. Thanks for the help so far!

@aarzilli
Copy link
Member

aarzilli commented Dec 8, 2021

Ping

@SteVwonder
Copy link
Author

Sorry for the delay on more info. Still working on a simpler reproducer. I’ll close this issue for now and re-open (or open a new one) once I have a reproducer. Thanks for the help so far.

@polinasok
Copy link
Collaborator

@SteVwonder Could you please get the logs one more time and show the rest of the DAP traffic starting from "event":"stopped"? I would like to see the source info returned for the various stackframes. Is this the only stack frame causing a problem?

I think the issue is a relative path that triggers an on-the-fly request for a source file. If this were an absolute path that was not properly matched and didn't exist, we would have gotten Error: Unable to resolve nonexistent file error. I am not sure why a relative path would be returned by the server (result of substitutePath file mapping?) and why the client doesn't try to resolve it relative to cwd (or maybe it does and it's not there?) What is the full path server and client path of the file in question and what are the launch config values logged with logging on?

"logOutput": "debugger,dap",
"showLog": true,

@aarzilli
Copy link
Member

I am not sure why a relative path would be returned by the server (result of substitutePath file mapping?)

Bazel passes trimpath by default and no substitutePath mapping was specified.

@polinasok
Copy link
Collaborator

Looking at #2754 (comment) more closely, I see

"substitutePath":[{"from":"/home/sherbein/Repositories/nuc/bazel-bin","to":"/home/sherbein/.cache/bazel/_bazel_sherbein/2d91ec193e823eeed6c484b1ac0ffa6d/execroot/git_nuc/bazel-out/k8-dbg/bin"}]

and then @suzmue suggested {"/home/sherbein/Repositories/nuc/src/": "src/"} , which didn't help. I think it might have needed to be /home/sherbein/Repositories/nuc/bazel-bin/src/. In any case, it is important to locate the actual source files to figure out what the proper prefix for the mapping should be to translate the relative paths inside of the binary to the absolute paths that VS Code expects.

@aarzilli, thanks for the -trimpath clarification. I found bazelbuild/rules_go#2258 pointing to this being a known issue with VSCode debugging. Shouldn't there be a problem with breakpoints as well then? If the binary has only relative paths, but the editor sets breakpoints with absolute paths, this shouldn't work without the right substitutePath either? @SteVwonder, are breakpoints a problem for you as well?

@SteVwonder
Copy link
Author

Breakpoints are also a problem for me. I should have some time today to grab a new trace and try different substitution paths. Thanks for the help all!

@SteVwonder
Copy link
Author

I think it might have needed to be /home/sherbein/Repositories/nuc/bazel-bin/src/. In any case, it is important to locate the actual source files to figure out what the proper prefix for the mapping should be to translate the relative paths inside of the binary to the absolute paths that VS Code expects.

I believe the substitution path is now correct. Here is my vs code launch config:

{
    "version": "0.2.0",
    "configurations": [
        {
                "name": "Launch //src/services/controller",
                "type": "go",
                "request": "launch",
                "mode": "exec",
                "sourceDirectories": [
                    "${workspaceRoot}/"
                ],
                "program": "${workspaceRoot}/bazel-bin/src/services/controller/controller",
                "env": {},
                "args": ["server",
                    "--port=8080",
                    "--tls.cert-file=\"/home/sherbein/certs/localhost.crt\"",
                    "--tls.key-file=\"/home/sherbein/certs/localhost.key\"",
                    "--with-agent"],
                //"preLaunchTask": "build-controller",
                "cwd": "${workspaceRoot}",
                "logOutput": "debugger,dap",
                "showLog": true,
                "substitutePath": [
                    {
                        "from": "src/",
                        //"to": "${workspaceRoot}/bazel-nuc/src",
                        "to": "/home/sherbein/Repositories/nuc/bazel-nuc/src/",
                    },
                ],
                "trace": "verbose",
        }
    ]
}

and a little sanity scheck of the substitutionpath:

❯ stat $(echo "src/common/future/pool.go" | sed 's$src/$/home/sherbein/Repositories/nuc/bazel-nuc/src/$')
  File: /home/sherbein/Repositories/nuc/bazel-nuc/src/common/future/pool.go
  Size: 1492            Blocks: 8          IO Block: 4096   regular file
Device: 830h/2096d      Inode: 99857       Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1000/sherbein)   Gid: ( 1000/sherbein)
Access: 2022-01-09 18:54:33.767070300 -0800
Modify: 2021-10-14 15:16:12.416227800 -0700
Change: 2021-10-14 15:16:12.416227800 -0700
 Birth: -

Could you please get the logs one more time and show the rest of the DAP traffic starting from "event":"stopped"? I would like to see the source info returned for the various stackframes. Is this the only stack frame causing a problem?

It is not the only stack frame causing issues. I have this problem for any attempt to access any of the source files in the as well as getting any of the breakpoints to work.

Starting: /home/sherbein/.nuc/bazel-gopath/bin/dlv-dap dap --check-go-version=false --listen=127.0.0.1:43525 --log=true --log-output=debugger,dap --log-dest=3 from /home/sherbein/Repositories/nuc
DAP server listening at: 127.0.0.1:43525
2022-01-09T20:19:50-08:00 debug layer=dap DAP server pid = 26477
2022-01-09T20:19:50-08:00 debug layer=dap DAP connection 1 started
2022-01-09T20:19:50-08:00 debug layer=dap [<- from client]{"seq":1,"type":"request","command":"initialize","arguments":{"clientID":"vscode","clientName":"Visual Studio Code","adapterID":"go","locale":"en-us","linesStartAt1":true,"columnsStartAt1":true,"pathFormat":"path","supportsVariableType":true,"supportsVariablePaging":true,"supportsRunInTerminalRequest":true,"supportsMemoryReferences":true,"supportsProgressReporting":true,"supportsInvalidatedEvent":true}}
2022-01-09T20:19:50-08:00 debug layer=dap [-> to client]{"seq":0,"type":"response","request_seq":1,"success":true,"command":"initialize","body":{"supportsConfigurationDoneRequest":true,"supportsFunctionBreakpoints":true,"supportsConditionalBreakpoints":true,"supportsEvaluateForHovers":true,"supportsSetVariable":true,"supportsExceptionInfoRequest":true,"supportTerminateDebuggee":true,"supportsDelayedStackTraceLoading":true,"supportsLogPoints":true,"supportsDisassembleRequest":true,"supportsClipboardContext":true,"supportsSteppingGranularity":true,"supportsInstructionBreakpoints":true}}
2022-01-09T20:19:50-08:00 debug layer=dap [<- from client]{"seq":2,"type":"request","command":"launch","arguments":{"name":"Launch //src/services/controller","type":"go","request":"launch","mode":"exec","sourceDirectories":["/home/sherbein/Repositories/nuc/"],"program":"/home/sherbein/Repositories/nuc/bazel-bin/src/services/controller/controller","env":{},"args":["server","--port=8080","--tls.cert-file=\"/home/sherbein/certs/localhost.crt\"","--tls.key-file=\"/home/sherbein/certs/localhost.key\"","--with-agent"],"cwd":"/home/sherbein/Repositories/nuc","logOutput":"debugger,dap","showLog":true,"substitutePath":[{"from":"src/","to":"/home/sherbein/Repositories/nuc/bazel-nuc/src/"}],"trace":"verbose","__configurationTarget":5,"packagePathToGoModPathMap":{"/home/sherbein/Repositories/nuc/src/pipelines/janitor/mark":"","/home/sherbein/Repositories/nuc/src/services/controller":""},"debugAdapter":"dlv-dap","dlvToolPath":"/home/sherbein/.nuc/bazel-gopath/bin/dlv-dap","__sessionId":"86eb7588-f9bf-4411-889e-590a9b2886f4"}}
2022-01-09T20:19:50-08:00 debug layer=dap parsed launch config: {
        "mode": "exec",
        "program": "/home/sherbein/Repositories/nuc/bazel-bin/src/services/controller/controller",
        "args": [
                "server",
                "--port=8080",
                "--tls.cert-file=\"/home/sherbein/certs/localhost.crt\"",
                "--tls.key-file=\"/home/sherbein/certs/localhost.key\"",
                "--with-agent"
        ],
        "cwd": "/home/sherbein/Repositories/nuc",
        "backend": "default",
        "stackTraceDepth": 50,
        "substitutePath": [
                {
                        "from": "src/",
                        "to": "/home/sherbein/Repositories/nuc/bazel-nuc/src/"
                }
        ]
}
2022-01-09T20:19:50-08:00 debug layer=dap launching binary '/home/sherbein/Repositories/nuc/bazel-bin/src/services/controller/controller' with config: {
        "mode": "exec",
        "program": "/home/sherbein/Repositories/nuc/bazel-bin/src/services/controller/controller",
        "args": [
                "server",
                "--port=8080",
                "--tls.cert-file=\"/home/sherbein/certs/localhost.crt\"",
                "--tls.key-file=\"/home/sherbein/certs/localhost.key\"",
                "--with-agent"
        ],
        "cwd": "/home/sherbein/Repositories/nuc",
        "backend": "default",
        "stackTraceDepth": 50,
        "substitutePath": [
                {
                        "from": "src/",
                        "to": "/home/sherbein/Repositories/nuc/bazel-nuc/src/"
                }
        ]
}
2022-01-09T20:19:50-08:00 info layer=debugger launching process with args: [/home/sherbein/Repositories/nuc/bazel-bin/src/services/controller/controller server --port=8080 --tls.cert-file="/home/sherbein/certs/localhost.crt" --tls.key-file="/home/sherbein/certs/localhost.key" --with-agent]
2022-01-09T20:19:51-08:00 warning layer=debugger reading debug_info: concrete subprogram without address range at 0x110dc8d
<snip>
2022-01-09T20:19:51-08:00 warning layer=debugger reading debug_info: concrete subprogram without address range at 0x1113afa
2022-01-09T20:19:51-08:00 debug layer=dap [-> to client]{"seq":0,"type":"event","event":"initialized"}
2022-01-09T20:19:51-08:00 debug layer=dap [-> to client]{"seq":0,"type":"response","request_seq":2,"success":true,"command":"launch"}
2022-01-09T20:19:51-08:00 debug layer=dap [<- from client]{"seq":3,"type":"request","command":"setBreakpoints","arguments":{"source":{"name":"main.go","path":"/home/sherbein/Repositories/nuc/src/services/controller/main.go"},"breakpoints":[{"line":16}],"lines":[16]}}
2022-01-09T20:19:51-08:00 debug layer=dap [-> to client]{"seq":0,"type":"response","request_seq":3,"success":true,"command":"setBreakpoints","body":{"breakpoints":[{"verified":false,"message":"could not find file /home/sherbein/Repositories/nuc/src/services/controller/main.go","source":{}}]}}
2022-01-09T20:19:51-08:00 debug layer=dap [<- from client]{"seq":4,"type":"request","command":"setFunctionBreakpoints","arguments":{"breakpoints":[]}}
2022-01-09T20:19:51-08:00 debug layer=dap [-> to client]{"seq":0,"type":"response","request_seq":4,"success":true,"command":"setFunctionBreakpoints","body":{"breakpoints":[]}}
2022-01-09T20:19:51-08:00 debug layer=dap [<- from client]{"seq":5,"type":"request","command":"setInstructionBreakpoints","arguments":{"breakpoints":[]}}
2022-01-09T20:19:51-08:00 debug layer=dap [-> to client]{"seq":0,"type":"response","request_seq":5,"success":true,"command":"setInstructionBreakpoints","body":{"breakpoints":[]}}
2022-01-09T20:19:51-08:00 debug layer=dap [<- from client]{"seq":6,"type":"request","command":"configurationDone","arguments":{}}
2022-01-09T20:19:51-08:00 debug layer=dap [-> to client]{"seq":0,"type":"event","event":"output","body":{"category":"console","output":"Type 'dlv help' for list of commands.\n","source":{}}}
2022-01-09T20:19:51-08:00 debug layer=dap [-> to client]{"seq":0,"type":"response","request_seq":6,"success":true,"command":"configurationDone"}
2022-01-09T20:19:51-08:00 debug layer=debugger continuing
Type 'dlv help' for list of commands.
2022-01-09T20:19:51-08:00 debug layer=dap [<- from client]{"seq":7,"type":"request","command":"threads"}
2022-01-09T20:19:51-08:00 debug layer=dap [-> to client]{"seq":0,"type":"response","request_seq":7,"success":true,"command":"threads","body":{"threads":[{"id":-1,"name":"Current"}]}}
2022/01/09 20:19:51 INFO - profiler manager will not be started, restart required to enable continuous profiling
2022/01/09 20:19:51 Setup workflow notifications
2022/01/09 20:19:51 Setup resource usage store
2022/01/09 20:19:51 [main] Resource usage store disabled
2022/01/09 20:19:51 Setup terminations queue
2022/01/09 20:19:51 Setup allow container registry: map[]
2022/01/09 20:19:51 [PRIME PIPELINES] starting state restore
2022/01/09 20:19:51 [PRIME PIPELINES] completed state restore
2022/01/09 20:19:51 [PRIME INACTIVE PIPELINES] starting state restore
2022/01/09 20:19:51 [PRIME INACTIVE PIPELINES] completed state restore
2022/01/09 20:19:51 Continuing with agent docker
2022/01/09 20:19:51 starting debug agent...
2022/01/09 20:19:51 Serving controller on localhost:8080
2022-01-09T20:19:54-08:00 debug layer=dap [<- from client]{"seq":8,"type":"request","command":"pause","arguments":{"threadId":-1}}
2022-01-09T20:19:54-08:00 debug layer=dap halting
2022-01-09T20:19:54-08:00 debug layer=debugger halting
2022-01-09T20:19:54-08:00 error layer=debugger error loading binary "/lib/x86_64-linux-gnu/libpthread.so.0": could not parse .eh_frame section: unknown CIE_id 0x2 at 0x0
2022-01-09T20:19:54-08:00 warning layer=debugger reading debug_info: concrete subprogram without address range at 0x52ef
2022-01-09T20:19:54-08:00 warning layer=debugger reading debug_info: concrete subprogram without address range at 0x532e
2022-01-09T20:19:54-08:00 warning layer=debugger reading debug_info: concrete subprogram without address range at 0x53f7
<snip>
2022-01-09T20:19:54-08:00 warning layer=debugger reading debug_info: concrete subprogram without address range at 0x19b653
2022-01-09T20:19:55-08:00 debug layer=dap "continue" command stopped - reason "manual", location GOROOT/src/runtime/sys_linux_amd64.s:580
2022-01-09T20:19:55-08:00 debug layer=dap manual halt requested, stop reason "pause" converted to "pause"
2022-01-09T20:19:55-08:00 debug layer=dap [-> to client]{"seq":0,"type":"event","event":"stopped","body":{"reason":"pause","allThreadsStopped":true}}
2022-01-09T20:19:55-08:00 debug layer=dap [-> to client]{"seq":0,"type":"response","request_seq":8,"success":true,"command":"pause"}
2022-01-09T20:19:55-08:00 debug layer=dap [<- from client]{"seq":9,"type":"request","command":"threads"}
2022-01-09T20:19:55-08:00 debug layer=dap [-> to client]{"seq":0,"type":"response","request_seq":9,"success":true,"command":"threads","body":{"threads":[{"id":1,"name":"[Go 1] internal/poll.runtime_pollWait"},{"id":2,"name":"[Go 2] runtime.gopark"},{"id":3,"name":"[Go 3] runtime.gopark"},{"id":4,"name":"[Go 4] runtime.gopark"},{"id":5,"name":"[Go 5] runtime.gopark"},{"id":6,"name":"[Go 6] k8s.io/klog.(*loggingT).flushDaemon"},{"id":7,"name":"[Go 7] runtime.gopark"},{"id":18,"name":"[Go 18] runtime.gopark"},{"id":34,"name":"[Go 34] runtime.gopark"},{"id":35,"name":"[Go 35] runtime.gopark"},{"id":8,"name":"[Go 8] runtime.gopark"},{"id":19,"name":"[Go 19] runtime.gopark"},{"id":36,"name":"[Go 36] runtime.gopark"},{"id":50,"name":"[Go 50] runtime.gopark"},{"id":51,"name":"[Go 51] runtime.gopark"},{"id":20,"name":"[Go 20] runtime.gopark"},{"id":37,"name":"[Go 37] runtime.gopark"},{"id":52,"name":"[Go 52] runtime.gopark"},{"id":83,"name":"[Go 83] github.com/NVIDIA/vdisc/pkg/httputil.AddDNSCache.func1.1"},{"id":153,"name":"[Go 153] github.com/klauspost/compress/zstd.(*blockDec).startDecoder"},{"id":154,"name":"[Go 154] github.com/klauspost/compress/zstd.(*blockDec).startDecoder"},{"id":155,"name":"[Go 155] github.com/klauspost/compress/zstd.(*blockDec).startDecoder"},{"id":156,"name":"[Go 156] github.com/klauspost/compress/zstd.(*blockDec).startDecoder"},{"id":157,"name":"[Go 157] github.com/klauspost/compress/zstd.(*blockDec).startDecoder"},{"id":158,"name":"[Go 158] github.com/klauspost/compress/zstd.(*blockDec).startDecoder"},{"id":159,"name":"[Go 159] github.com/klauspost/compress/zstd.(*blockDec).startDecoder"},{"id":160,"name":"[Go 160] github.com/klauspost/compress/zstd.(*blockDec).startDecoder"},{"id":161,"name":"[Go 161] github.com/klauspost/compress/zstd.(*blockDec).startDecoder"},{"id":162,"name":"[Go 162] github.com/klauspost/compress/zstd.(*blockDec).startDecoder"},{"id":163,"name":"[Go 163] github.com/klauspost/compress/zstd.(*blockDec).startDecoder"},{"id":164,"name":"[Go 164] github.com/klauspost/compress/zstd.(*blockDec).startDecoder"},{"id":196,"name":"[Go 196] github.com/desertbit/timer.timerRoutine"},{"id":249,"name":"[Go 249] git.example.com/nuc/src/common/etl/controller/jobdb.(*boltDBManager).coordinate"},{"id":250,"name":"[Go 250] git.example.com/nuc/src/common/future.futureWorker"},{"id":251,"name":"[Go 251] git.example.com/nuc/src/common/future.futureWorker"},{"id":252,"name":"[Go 252] git.example.com/nuc/src/common/future.futureWorker"},{"id":253,"name":"[Go 253] git.example.com/nuc/src/common/future.futureWorker"},{"id":254,"name":"[Go 254] git.example.com/nuc/src/common/future.futureWorker"},{"id":255,"name":"[Go 255] git.example.com/nuc/src/common/future.futureWorker"},{"id":256,"name":"[Go 256] git.example.com/nuc/src/common/future.futureWorker"},{"id":257,"name":"[Go 257] git.example.com/nuc/src/common/future.futureWorker"},{"id":258,"name":"[Go 258] git.example.com/nuc/src/common/future.futureWorker"},{"id":259,"name":"[Go 259] git.example.com/nuc/src/common/future.futureWorker"},{"id":260,"name":"[Go 260] git.example.com/nuc/src/common/future.futureWorker"},{"id":261,"name":"[Go 261] git.example.com/nuc/src/common/future.futureWorker"},{"id":262,"name":"[Go 262] git.example.com/nuc/src/common/future.futureWorker"},{"id":263,"name":"[Go 263] git.example.com/nuc/src/common/future.futureWorker"},{"id":264,"name":"[Go 264] git.example.com/nuc/src/common/future.futureWorker"},{"id":265,"name":"[Go 265] git.example.com/nuc/src/common/future.futureWorker"},{"id":266,"name":"[Go 266] git.example.com/nuc/src/common/future.futureWorker"},{"id":267,"name":"[Go 267] git.example.com/nuc/src/common/future.futureWorker"},{"id":268,"name":"[Go 268] git.example.com/nuc/src/common/future.futureWorker"},{"id":269,"name":"[Go 269] git.example.com/nuc/src/common/future.futureWorker"},{"id":270,"name":"[Go 270] git.example.com/nuc/src/common/future.futureWorker"},{"id":271,"name":"[Go 271] git.example.com/nuc/src/common/future.futureWorker"},{"id":272,"name":"[Go 272] git.example.com/nuc/src/common/future.futureWorker"},{"id":273,"name":"[Go 273] git.example.com/nuc/src/common/future.futureWorker"},{"id":274,"name":"[Go 274] git.example.com/nuc/src/common/future.futureWorker"},{"id":275,"name":"[Go 275] git.example.com/nuc/src/common/future.futureWorker"},{"id":276,"name":"[Go 276] git.example.com/nuc/src/common/future.futureWorker"},{"id":277,"name":"[Go 277] git.example.com/nuc/src/common/future.futureWorker"},{"id":278,"name":"[Go 278] git.example.com/nuc/src/common/future.futureWorker"},{"id":279,"name":"[Go 279] git.example.com/nuc/src/common/future.futureWorker"},{"id":280,"name":"[Go 280] git.example.com/nuc/src/common/future.futureWorker"},{"id":281,"name":"[Go 281] git.example.com/nuc/src/common/future.futureWorker"},{"id":282,"name":"[Go 282] git.example.com/nuc/src/common/future.futureWorker"},{"id":283,"name":"[Go 283] git.example.com/nuc/src/common/future.futureWorker"},{"id":284,"name":"[Go 284] git.example.com/nuc/src/common/future.futureWorker"},{"id":285,"name":"[Go 285] git.example.com/nuc/src/common/future.futureWorker"},{"id":286,"name":"[Go 286] git.example.com/nuc/src/common/future.futureWorker"},{"id":287,"name":"[Go 287] git.example.com/nuc/src/common/future.futureWorker"},{"id":288,"name":"[Go 288] git.example.com/nuc/src/common/future.futureWorker"},{"id":289,"name":"[Go 289] git.example.com/nuc/src/common/future.futureWorker"},{"id":290,"name":"[Go 290] git.example.com/nuc/src/common/future.futureWorker"},{"id":291,"name":"[Go 291] git.example.com/nuc/src/common/future.futureWorker"},{"id":292,"name":"[Go 292] git.example.com/nuc/src/common/future.futureWorker"},{"id":293,"name":"[Go 293] git.example.com/nuc/src/common/future.futureWorker"},{"id":294,"name":"[Go 294] git.example.com/nuc/src/common/future.futureWorker"},{"id":295,"name":"[Go 295] git.example.com/nuc/src/common/future.futureWorker"},{"id":296,"name":"[Go 296] git.example.com/nuc/src/common/future.futureWorker"},{"id":297,"name":"[Go 297] git.example.com/nuc/src/common/future.futureWorker"},{"id":298,"name":"[Go 298] git.example.com/nuc/src/common/future.futureWorker"},{"id":299,"name":"[Go 299] git.example.com/nuc/src/common/future.futureWorker"},{"id":300,"name":"[Go 300] git.example.com/nuc/src/common/future.futureWorker"},{"id":301,"name":"[Go 301] git.example.com/nuc/src/common/future.futureWorker"},{"id":302,"name":"[Go 302] git.example.com/nuc/src/common/future.futureWorker"},{"id":303,"name":"[Go 303] git.example.com/nuc/src/common/future.futureWorker"},{"id":304,"name":"[Go 304] git.example.com/nuc/src/common/future.futureWorker"},{"id":305,"name":"[Go 305] git.example.com/nuc/src/common/future.futureWorker"},{"id":306,"name":"[Go 306] git.example.com/nuc/src/common/future.futureWorker"},{"id":307,"name":"[Go 307] git.example.com/nuc/src/common/future.futureWorker"},{"id":308,"name":"[Go 308] git.example.com/nuc/src/common/future.futureWorker"},{"id":309,"name":"[Go 309] git.example.com/nuc/src/common/future.futureWorker"},{"id":310,"name":"[Go 310] git.example.com/nuc/src/common/future.futureWorker"},{"id":311,"name":"[Go 311] git.example.com/nuc/src/common/future.futureWorker"},{"id":312,"name":"[Go 312] git.example.com/nuc/src/common/future.futureWorker"},{"id":313,"name":"[Go 313] git.example.com/nuc/src/common/future.futureWorker"},{"id":314,"name":"[Go 314] git.example.com/nuc/src/common/future.futureWorker"},{"id":315,"name":"[Go 315] git.example.com/nuc/src/common/future.futureWorker"},{"id":316,"name":"[Go 316] git.example.com/nuc/src/common/future.futureWorker"},{"id":317,"name":"[Go 317] git.example.com/nuc/src/common/future.futureWorker"},{"id":318,"name":"[Go 318] git.example.com/nuc/src/common/future.futureWorker"},{"id":319,"name":"[Go 319] git.example.com/nuc/src/common/future.futureWorker"},{"id":320,"name":"[Go 320] git.example.com/nuc/src/common/future.futureWorker"},{"id":321,"name":"[Go 321] git.example.com/nuc/src/common/future.futureWorker"},{"id":322,"name":"[Go 322] git.example.com/nuc/src/common/future.futureWorker"},{"id":323,"name":"[Go 323] git.example.com/nuc/src/common/future.futureWorker"},{"id":324,"name":"[Go 324] git.example.com/nuc/src/common/future.futureWorker"},{"id":325,"name":"[Go 325] git.example.com/nuc/src/common/future.futureWorker"},{"id":326,"name":"[Go 326] git.example.com/nuc/src/common/future.futureWorker"},{"id":327,"name":"[Go 327] git.example.com/nuc/src/common/future.futureWorker"},{"id":328,"name":"[Go 328] git.example.com/nuc/src/common/future.futureWorker"},{"id":329,"name":"[Go 329] git.example.com/nuc/src/common/future.futureWorker"},{"id":330,"name":"[Go 330] git.example.com/nuc/src/common/future.futureWorker"},{"id":331,"name":"[Go 331] git.example.com/nuc/src/common/future.futureWorker"},{"id":332,"name":"[Go 332] git.example.com/nuc/src/common/future.futureWorker"},{"id":333,"name":"[Go 333] git.example.com/nuc/src/common/future.futureWorker"},{"id":334,"name":"[Go 334] git.example.com/nuc/src/common/future.futureWorker"},{"id":335,"name":"[Go 335] git.example.com/nuc/src/common/future.futureWorker"},{"id":336,"name":"[Go 336] git.example.com/nuc/src/common/future.futureWorker"},{"id":337,"name":"[Go 337] git.example.com/nuc/src/common/future.futureWorker"},{"id":338,"name":"[Go 338] git.example.com/nuc/src/common/future.futureWorker"},{"id":339,"name":"[Go 339] git.example.com/nuc/src/common/future.futureWorker"},{"id":340,"name":"[Go 340] git.example.com/nuc/src/common/future.futureWorker"},{"id":341,"name":"[Go 341] git.example.com/nuc/src/common/future.futureWorker"},{"id":342,"name":"[Go 342] git.example.com/nuc/src/common/future.futureWorker"},{"id":343,"name":"[Go 343] git.example.com/nuc/src/common/future.futureWorker"},{"id":344,"name":"[Go 344] git.example.com/nuc/src/common/future.futureWorker"},{"id":345,"name":"[Go 345] git.example.com/nuc/src/common/future.futureWorker"},{"id":346,"name":"[Go 346] git.example.com/nuc/src/common/future.futureWorker"},{"id":347,"name":"[Go 347] git.example.com/nuc/src/common/future.futureWorker"},{"id":348,"name":"[Go 348] git.example.com/nuc/src/common/future.futureWorker"},{"id":349,"name":"[Go 349] git.example.com/nuc/src/common/future.futureWorker"},{"id":350,"name":"[Go 350] git.example.com/nuc/src/services/controller/server.(*SummaryCache).Run"},{"id":217,"name":"[Go 217] google.golang.org/grpc.(*ccBalancerWrapper).watcher"},{"id":218,"name":"[Go 218] google.golang.org/grpc.(*addrConn).resetTransport"},{"id":61,"name":"[Go 61] git.example.com/nuc/src/common/etl/agent/invokers/docker.(*DockerInvoker).coordinate"},{"id":486,"name":"[Go 486] git.example.com/nuc/src/common/pubsub.(*debugSubscription).dispatch"},{"id":690,"name":"[Go 690] internal/poll.runtime_pollWait"},{"id":661,"name":"[Go 661] net/http.(*http2serverConn).serve"},{"id":676,"name":"[Go 676] google.golang.org/grpc/internal/transport.(*controlBuffer).get"},{"id":627,"name":"[Go 627] git.example.com/nuc/src/common/etl/agent.(*jobSchedulerImpl).AcquireJob"},{"id":674,"name":"[Go 674] net/http.(*http2serverConn).serve"},{"id":485,"name":"[Go 485] os/signal.signal_recv (Thread 26508)"},{"id":487,"name":"[Go 487] git.example.com/nuc/src/common/pubsub.(*debugSubscription).dispatch"},{"id":483,"name":"[Go 483] main.(*serverCommand).Execute.func1"},{"id":484,"name":"[Go 484] runtime.gopark"},{"id":653,"name":"[Go 653] google.golang.org/grpc.(*ccBalancerWrapper).watcher"},{"id":480,"name":"[Go 480] git.example.com/nuc/src/common/etl/agent.(*jobSchedulerImpl).schedule"},{"id":626,"name":"[Go 626] main.(*serverCommand).Execute.func2"},{"id":648,"name":"[Go 648] git.example.com/nuc/src/services/controller/api.RegisterETLControllerHandlerFromEndpoint.func1.1"},{"id":646,"name":"[Go 646] google.golang.org/grpc.(*ccBalancerWrapper).watcher"},{"id":647,"name":"[Go 647] google.golang.org/grpc.(*addrConn).resetTransport"},{"id":642,"name":"[Go 642] git.example.com/nuc/src/common/etl/command.(*CommandCenter).handleCommands"},{"id":691,"name":"[Go 691] google.golang.org/grpc/internal/transport.(*controlBuffer).get"},{"id":675,"name":"[Go 675] internal/poll.runtime_pollWait"},{"id":706,"name":"[Go 706] internal/poll.runtime_pollWait"},{"id":654,"name":"[Go 654] google.golang.org/grpc.(*addrConn).resetTransport"},{"id":724,"name":"[Go 724] internal/poll.runtime_pollWait"}]}}
2022-01-09T20:20:01-08:00 debug layer=dap [<- from client]{"seq":10,"type":"request","command":"stackTrace","arguments":{"threadId":276,"levels":20,"format":{}}}
2022-01-09T20:20:01-08:00 debug layer=dap [-> to client]{"seq":0,"type":"response","request_seq":10,"success":true,"command":"stackTrace","body":{"stackFrames":[{"id":1000,"name":"runtime.gopark","source":{"name":"proc.go","path":"GOROOT/src/runtime/proc.go"},"line":337,"column":0,"instructionPointerReference":"0x43e305","presentationHint":"subtle"},{"id":1001,"name":"runtime.chanrecv","source":{"name":"chan.go","path":"GOROOT/src/runtime/chan.go"},"line":576,"column":0,"instructionPointerReference":"0x409b6f","presentationHint":"subtle"},{"id":1002,"name":"runtime.chanrecv2","source":{"name":"chan.go","path":"GOROOT/src/runtime/chan.go"},"line":444,"column":0,"instructionPointerReference":"0x4097eb","presentationHint":"subtle"},{"id":1003,"name":"git.example.com/nuc/src/common/future.futureWorker","source":{"name":"pool.go","path":"src/common/future/pool.go"},"line":69,"column":0,"instructionPointerReference":"0x194d94e"},{"id":1004,"name":"runtime.goexit","source":{"name":"asm_amd64.s","path":"GOROOT/src/runtime/asm_amd64.s"},"line":1371,"column":0,"instructionPointerReference":"0x475621","presentationHint":"subtle"}],"totalFrames":5}}
2022-01-09T20:20:03-08:00 debug layer=dap [<- from client]{"seq":11,"type":"request","command":"source","arguments":{"source":{"name":"pool.go","path":"src/common/future/pool.go"},"sourceReference":0}}
2022-01-09T20:20:03-08:00 debug layer=dap Unsupported command: cannot process "source" request
2022-01-09T20:20:03-08:00 debug layer=dap [-> to client]{"seq":0,"type":"response","request_seq":11,"success":false,"command":"source","message":"Unsupported command","body":{"error":{"id":9999,"format":"Unsupported command: cannot process \"source\" request","showUser":false}}}
2022-01-09T20:20:03-08:00 debug layer=dap [<- from client]{"seq":12,"type":"request","command":"scopes","arguments":{"frameId":1003}}
2022-01-09T20:20:03-08:00 debug layer=dap [-> to client]{"seq":0,"type":"response","request_seq":12,"success":true,"command":"scopes","body":{"scopes":[{"name":"Locals","variablesReference":1000,"expensive":false,"source":{}}]}}
2022-01-09T20:20:03-08:00 debug layer=dap [<- from client]{"seq":13,"type":"request","command":"variables","arguments":{"variablesReference":1000,"format":{}}}
2022-01-09T20:20:03-08:00 debug layer=dap [-> to client]{"seq":0,"type":"response","request_seq":13,"success":true,"command":"variables","body":{"variables":[{"name":"workChannel","value":"\u003c-chan *git.example.com/nuc/src/common/future.workItem 0/1024","type":"\u003c-chan *git.example.com/nuc/src/common/future.workItem","presentationHint":{},"evaluateName":"workChannel","variablesReference":1001},{"name":"wg","value":"*sync.WaitGroup {noCopy: sync.noCopy {}, state1: [3]uint32 [0,100,0]}","type":"*sync.WaitGroup","presentationHint":{},"evaluateName":"wg","variablesReference":1002}]}}

@aarzilli
Copy link
Member

You put /home/sherbein/Repositories/nuc/bazel-nuc/src/ in your configuration but the editor is requesting a breakpoint in /home/sherbein/Repositories/nuc/src/services/controller/main.go, different directory.

@polinasok polinasok changed the title VS Code DAP source request not supported VS Code DAP does not map trimmed paths Jan 14, 2022
@susarlanikhilesh
Copy link

I would suggest adding {"/home/sherbein/Repositories/nuc/src/": "src/"} to your "substitutePath" configuration, to see if that helps with displaying the correct source. The path that delve is sending back is "src/common/retryer/retryer.go" and assuming that the file you have locally is at "/home/sherbein/Repositories/nuc/src/common/retryer/retryer.go" this should help map it.

Can you help me for windows, I'm unable to figure this part. As when I attach local debugger to process using dlv and add a breakpoint, it says could not find file

@hyangah
Copy link
Contributor

hyangah commented Feb 3, 2022

@susarlanikhilesh here is a tip:
(1) start a debug session from your terminal: dlv.exe exec <your binary>
(2) run sources to see what file paths embedded in the compiled binary are like.
(3) formulate your substitutePath in your launch.json to have
* from -> prefix of paths your editor sees
* to -> prefix of paths appeared in the sources output from above.

VSCode Go also offers to do (1) and (2) from DEBUG CONSOLE (dlv sources command) once a debug session is activated, but, first, you will need to make the debug session stay active. One way to do it is to use "stopOnEntry": true property in launch.json and inspect from DEBUG CONSOLE. I found running dlv.exe from terminal is easier.

--
We are working on automating part of this process. @suzmue

@polinasok
Copy link
Collaborator

You can also set substitutePath on the fly via DEBUG CONSOLE (see dlv help config)

dlv config substitutePath <from> <to>
dlv config substitutePath <from>
	
Adds or removes a path substitution rule.

@hyangah
Copy link
Contributor

hyangah commented Feb 3, 2022

@polinasok only if the user managed to stop the debug session without breaking other parts of UI. (stopOnEntry is one way to do it, but not sure how vscode behaves when pausing and stack trace can't be processed)

@aarzilli
Copy link
Member

aarzilli commented May 4, 2022

I'm going to close this since it seems that we're not going to do any sort of automatic mapping for now and the original report was likely a configuration error.

@aarzilli aarzilli closed this as completed May 4, 2022
@garymm
Copy link

garymm commented Oct 13, 2022

Is there a way to set a substitutePath that applies to all relative paths?
In VSCode config at least "to" must be set to something non-empty, so I have to add an substitutePath entry for every subdirectory of my workspace, e.g.:

"substitutePath": [
    // I can't figure out how to set a single substitution for all paths,
    // so I have to do it for each subdirectory of the workspace.
    {
        "from": "${workspaceFolder}/external/",
        "to": "external/",
    },
    {
        "from": "${workspaceFolder}/infra/",
        "to": "infra/",
    },
    {
        "from": "${workspaceFolder}/models/",
        "to": "models/",
    }
]

@firelizzard18
Copy link

@garymm It may not be an option for you, but if you switch to modules (instead of GOPATH) all of your source files are prefixed with the module name so "to" would be "example.com/your/module/".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

8 participants