Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[usage] Delay gRPC self connection #11747

Merged
merged 1 commit into from
Aug 1, 2022
Merged

Conversation

andrew-farries
Copy link
Contributor

Description

The billing client tries to connect to the usage component gRPC server before the gRPC server is ready. This is a non-fatal error as gRPC dialing has built-in backoff/retry logic but it does produce ugly warnings in the usage component logs.

Here we use a simple solution to delay the self-connection attempt for a short period to give the gRPC server time to start.

Related Issue(s)

Fixes #11743

How to test

Warning messages at usage component startup (as described in #11743) are no longer present.

Release Notes

NONE

Documentation

Werft options:

  • /werft with-preview

The billing client tries to connect to the usage component gRPC server
before the gRPC server is ready. This is a non-fatal error as gRPC
dialing has built-in backoff/retry logic but it does produce ugly
warnings in the usage component logs.

Here we use a simple solution to delay the self-connection attempt for a
short period to give the gRPC server time to start.
@andrew-farries andrew-farries changed the title [usage] Delay self connection [usage] Delay gRPC self connection Jul 29, 2022
@andrew-farries andrew-farries marked this pull request as ready for review July 29, 2022 15:04
@andrew-farries andrew-farries requested a review from a team July 29, 2022 15:04
@github-actions github-actions bot added the team: webapp Issue belongs to the WebApp team label Jul 29, 2022
@easyCZ
Copy link
Member

easyCZ commented Aug 1, 2022

How does one know what delay is sufficient? I think possibly instead of this passing in a string address of the target service and the service establishes the connection the first time a request arrives may be easier to to deal with

@andrew-farries
Copy link
Contributor Author

How does one know what delay is sufficient?

It doesn't. There is no synchronization, just a 'long enough' one second delay to give the gRPC server some time to start before the client makes the first connection attempt.

I think possibly instead of this passing in a string address of the target service and the service establishes the connection the first time a request arrives may be easier to to deal with.

Not sure I follow. You mean don't have the client connect at startup but somehow do the connection on demand?

@roboquat roboquat merged commit 0805d02 into main Aug 1, 2022
@roboquat roboquat deleted the af/usage-delay-self-connection branch August 1, 2022 14:34
@roboquat roboquat added deployed: webapp Meta team change is running in production deployed Change is completely running in production labels Aug 2, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
deployed: webapp Meta team change is running in production deployed Change is completely running in production release-note-none size/M team: webapp Issue belongs to the WebApp team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Race condition at usage component startup
3 participants