Unable to do remote debugging of mongodb-datastore due to liveness-probe

[warber]

Since the introduction of the livenessProbe for the mongodb-datstore and the distributor it is not possible to remotely debug the mongodb-datastore. The problem at hand is, that while sitting at a breakpoint inside the mongodb-datastore, the liveness-probe inside the cluster fails which causes the pod to be restarted by kubernetes. Hence, the debugging connection breaks.

Affected Component

• Datastore

Note, that other components might suffer from this issue too.

To Reproduce

1. Set a breakpoint in mongodb-datastore.
2. Deploy the mongodb-datastore with remote debugging.
3. While the execution stops at the breakpoint, observe that in kubernetes a "Liveness probe failed" event gets triggered and the debugging connection breaks.

Expected behavior
Pausing the execution of the process does not affect the lifecycle of the pod in the cluster.

Current behavior
The pod is restarted because liveness checks fail

[johannes-b]

• It needs to be figured out whether the liveness and/or the readiness probe is the blocker.
• Are there other services affected?