WithWaitForReady: Set retry timeout?

[jmariller]

Hello,

I am using withWaitForReady to ensure calls will be done only once the channel is ready. What I noticed is, when starting my client some time before starting the server (for example around 30 seconds), it takes a while for the client to connect (e.g. around 10 seconds, sometimes a bit less/more) - and this seems to be exponential to the waiting time on the client side.

What I found in the gRPC documentation is this: https://github.com/grpc/grpc/blob/master/doc/connection-backoff.md
But I am not sure how to tell gRPC that I want the retry timeout to be more constant/shorter - can you help? Thanks!

[ejona86]

gRPC has two separate concepts that are independent: retries and reconnects. Retries (who's subsystem isn't yet enabled be default) are per-RPC. Reconnects are connection-level. There is no feedback between the two systems. The connection-backoff.md doc talks about reconnects.

Can you describe why it takes 10 seconds to connect to the server? The backoff only applies after a failure and starts at 1s, so the only way to see 10s delay is 1) the connect to actually take a long time or 2) many previous connect attempts failed.

What backoff settings are you hoping for that are "constant/shorter"? Like, are you wanting to backoff up to 1s, or 1ms?

[jmariller]

Thank you @ejona86 for the quick answer and the explanation on the two concept - I guess I am more looking into the reconnect then!

On your questions, actually I am not sure why it takes 10 seconds to connect to the server - maybe the latter, where previous connect attempts failed, but I am also not sure how to verify this? (Keep in mind I am completely new to gRPC).

I guess some more details on the use case could help understand what I am looking for: so we have this project where different types of client devices (smartphones, smart glasses, etc.) connect to a remote server that is streaming various data (coordinates, images, etc.).
Now it is very likely that some of those clients will launch (way) before the server is up and running - therefore what we want to ensure is that as soon as the server is ready, the client should connect to it (with as little latency/waiting time as possible).

What I had initially tried in order to verify that connection is established was to build a loop around getState() until status was ready with a manual timeout. Maybe you could give me a hint on how to best implement such a check?

[ejona86]

If you are expecting the server to be running and seeing the 10s, then I'd suggest trying with withoutWaitForReady (the default). If you get an error, it would probably explain what is going on.

Why will the server not be running? Depending on what's happening, this could be a quite reasonable case where some configuration of the backoff would be appropriate. Right now there is no configuration option for this sort of thing. (Related: #7456) Reducing the backoff can still lead to issues, especially in poor networking situations, so you may still need to make things semi-conservative. But maybe adjusting MULTIPLIER would work well for you.

For the short-term, you could call managedChannel.resetConnectBackoff() if getState() == TRANSIENT_FAILURE (and poll like once every couple seconds). But that's for the case you know the server isn't running yet.

[jmariller]

So I checked with withoutWaitForReady and I do not get an error when the server is not running yet: the channel is basically in IDLE state.

The server may not be running because it is "connected" to a main device which is the root source of data (in our case it is a medical device to which the server is attached, but it could be something else).

Still not completely sure on how to best implement that polling - I tried something the following in my connect method, but it does not work as expected (e.g. it takes some for the client to actually connect to the server):

while (managedChannel.getState(false) != ConnectivityState.READY) {
   managedChannel.getState(true);
   TimeUnit.SECONDS.sleep(100);
}

[jmariller]

Hi @ejona86

Whenever you get the chance, may I ask you for your support on the polling implementation? See my previous message for details.

Many thanks for your patience and support!

[ejona86]

So I checked with withoutWaitForReady and I do not get an error when the server is not running yet: the channel is basically in IDLE state.

The channel cannot be in IDLE state while an RPC is trying to be sent. At the very least it'd be CONNECTING. If the server is not running, then the RPC clearly can't complete successfully. When you say you do not get an error, you are saying that the RPC hangs?

If a non-wait-for-ready RPC is "hanging", that means the connect() itself is taking a long time (e.g., 20 seconds). connect() does exponential backoff itself, but that's all implemented within the kernel's TCP stack.

Since non-wait-for-ready doesn't improve your situation, I don't think you are hitting the grpc backoff (at least in that test). But if you were, I had been suggesting something more along the lines of:

while (managedChannel.getState(true) != ConnectivityState.READY) {
   TimeUnit.SECONDS.sleep(10);
   managedChannel.resetConnectBackoff();
}

[jmariller]

Thanks a lot for getting back to me so quickly!

The channel cannot be in IDLE state while an RPC is trying to be sent. At the very least it'd be CONNECTING. If the server is not running, then the RPC clearly can't complete successfully. When you say you do not get an error, you are saying that the RPC hangs?

Now I believe I start to understand my problem/confusion - based on what you wrote I understand that when I run my connect method in the main program, that method should already try to execute an RPC...? As things stand on my side the connect method is only setting up the channel and a blocking stub that is later used by the various RPCs methods.

Can you confirm the connect method should execute an RPC, or clarify in case I misunderstood?

[ejona86]

based on what you wrote I understand that when I run my connect method in the main program, that method should already try to execute an RPC...?

Hard for me to say. Most of the time call sites should just send an RPC if they want to send an RPC. If the channel can't connect it will fail, otherwise it can succeed. For you, reading between the lines, it seems you have a connect() method that you run first and it waits for the channel to become healthy. Then after connect() returns you would send RPCs. That approach can work, but you won't get any useful debugging information from gRPC without sending RPCs.

What's a bit more common is for you to have code that will send RPCs and then separately (like in a different thread), you can have code that keeps connections warm and monitor's their health. That's semi-common for servers, but your case sounds like it is a UI where the user is actively waiting. For your case, wait-for-connect-and-then-send-RPCs is an okay approach, except for not getting any real error messages to tell the user what's wrong.

[ejona86]

Seems like this is likely resolved. If not, comment, and it can be reopened.

[jmariller]

Yes sorry for not getting back to you - I tried the bit of code you shared above, and that actually worked! So thanks again for your help!