[DevicePlugin] Should we retry when ListAndWatch gRPC call failed?

[ScorpioCPH]

Is this a BUG REPORT or FEATURE REQUEST?:

/area hw-accelerators
/sig node

What happened:

We return immediately when endpoint ListAndWatch gRPC call failed and endpoint will stop:

go func() {
	e.run()
	e.stop()
}

Should we give another chance (e.g. wait 3 seconds) to wait for device plugin client to come up?

Another thing: maybe this is why we get so many gRPC error message in unit test randomly.

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

I will send a PR for this proposal.

@vikaschoudhary16 @jiayingz @RenaudWasTaken WDYT?

Environment:

• Kubernetes version (use kubectl version):
• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Others:

[ScorpioCPH]

/assign

[vikaschoudhary16]

Since the client that endpoint is using to wait on ListAndWatch has already ensured that connection was READY with server, in that case if still it fails, that could happen only if device plugin has died.
So your question becomes, should we wait for device plugin restart?

I think device plugin restart is already handled and we create a new endpoint on reregistration. Thinking all this I am unable to see benefit of waiting there.

[ScorpioCPH]

Hi,

that could happen only if device plugin has died.

SGTM if we can make sure about this.

By the way, Is there any case that device plugin is still running but ClientStream is down?
Just FYI here:

// RecvMsg blocks until it receives a message or the stream is
// done. On client side, it returns io.EOF when the stream is done. On
// any other error, it aborts the stream and returns an RPC status.

[jiayingz]

I think the stream should be kept around during the life cycle of device plugin, so agree with @vikaschoudhary16 that the current behavior seems WAI.

[RenaudWasTaken]

Should we give another chance (e.g. wait 3 seconds) to wait for device plugin client to come up?

This is already backed by gRPC mechanism of retry.

Another thing: maybe this is why we get so many gRPC error message in unit test randomly.

These messages are not random. They happen because the client disconnects (i.e: device plugin closes the connexion).
The order is pretty random though probably because it doesn't use the same logging mechanism.

[ScorpioCPH]

Thanks for the detailed explanation, seems we are ok about this so close it.