Can't attach to the container after exiting the current debug session

[baodongli]

I am able to debug the microservice following the instructions. I got a few issues though:

1. I issued 'exit' to the current debug session, but didn't kill the process. After 'dlv connect' to the same container, the process is no longer debuggable

2. If I delete the current attachment, and attach to the container again, the attachment would show as in error.

squash --url=http://localhost:8001/api/v1/namespaces/squash/services/squash-server:http-squash-api/proxy/api/v2 debug-container docker.io/baodongli/microservice:0.1 example-microservice-rc-j68c7 example-microservice dlv
Debug config id: ONRhfKsUOH

squash --url=http://localhost:8001/api/v1/namespaces/squash/services/squash-server:http-squash-api/proxy/api/v2 list attachments 
State |ID         |Debugger |Image                                |Debugger Address
error |ONRhfKsUOH |dlv      |docker.io/baodongli/microservice:0.1 |

3. After deleting/adding the microservice pod, sometimes, the 'wait' command returns timeout, and the 'list' command shows the attachment as 'none' state.

[baodongli]

more info:

squash --url=http://localhost:8001/api/v1/namespaces/squash/services/squash-server:http-squash-api/proxy/api/v2 debug-container docker.io/baodongli/squash-demo:0.1 squash-demo-rc-db5rh squash-demo dlv
Debug config id: oF8iVdiJs5
devuser@c2-master:~/go/src/squash$ squash --url=http://localhost:8001/api/v1/namespaces/squash/services/squash-server:http-squash-api/proxy/api/v2 list attachments 
State |ID         |Debugger |Image                               |Debugger Address
none  |oF8iVdiJs5 |dlv      |docker.io/baodongli/squash-demo:0.1 |
devuser@c2-master:~/go/src/squash$ squash --url=http://localhost:8001/api/v1/namespaces/squash/services/squash-server:http-squash-api/proxy/api/v2 list attachments 
State |ID         |Debugger |Image                               |Debugger Address
none  |oF8iVdiJs5 |dlv      |docker.io/baodongli/squash-demo:0.1 |

[yuval-k]

Thanks for the feedback! will need to do some digging to understand the causes, and will update the ticket.

[yuval-k]

I'm writing automated e2e test for your scenario so should have updates in the next few days.

[baodongli]

thanks. Just a little more info about the first issue:
exit and connect back, and get the following result:
devuser@c2-master:~/go/src/squash$ dlv connect localhost:38408
Handling connection for 38408
Type 'help' for list of commands.
(dlv) break /home/devuser/go/src/istio.io/istio/pilot/proxy/envoy/discovery.go:477
Command failed: no such process

My workaround is to kill the pods, delete the attach, restart the pods, attach again. It would be great if this can be fixed

[yuval-k]

it seems like that from some reason dlv in server mode doesn't work very well with this use case of a client re-connecting (regardless of squash - i tested this with a bunch of terminals).

I'm going to implement a fix so that you will only need to delete the attachment and recreate it in order for it to work (i.e. no need to delete the pods).

[yuval-k]

I pushed the fix (as described in my previous comment) to master, so it will be out in the next release.

[pavius]

@yuval-k unsure if the same root cause, but I had a somewhat similar scenario that doesn't seem delve related.

1. Successfully attached to a microservice
2. Noticed I forgot to indicate dlv, so it attached gdb
3. Tried to attach dlv - failed
4. Deleted all attachments (list attachments returned an empty list)
5. Tried to attach with dlv - failed
6. Looked @ client logs, saw Already debugging pid. ignoring (

   squash/pkg/debuggers/controller.go

   Line 196 in 1592ac0

   log.WithField("pid", pid).Warn("Already debugging pid. ignoring")

   )
7. Deleted attachment
8. Recreated pod
9. Re-ran on new pod - attached OK

[yuval-k]

Sounds like the same problem - should be resolved in our next release where deleting the attachment will automatically detach the debugger