Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Daemon client fails while closing connection to the daemon #25905

Closed
ghale opened this issue Jul 25, 2023 · 0 comments · Fixed by #25925
Closed

Daemon client fails while closing connection to the daemon #25905

ghale opened this issue Jul 25, 2023 · 0 comments · Fixed by #25925
Assignees
Milestone

Comments

@ghale
Copy link
Member

ghale commented Jul 25, 2023

When a build completes in the daemon, it sends the result back to the daemon client, along with a "build complete" message and then waits for a "finished" message from the client. Once the client receives the complete message, it then sends the "finished" message. The daemon waits for up to 60s to receive the finished message before giving up and closing the connection. There are a few scenarios where this could cause a problem:

  • The client is delayed in sending the finished message (possibly because it is flooded with log messages, or some other spurious delay). The daemon can timeout and close the connection, causing a failure when the client tries to send the finished message.
  • The daemon sends the complete message and then suffers a failure while waiting for the finished message. This could be because the daemon experiences an OOM or a daemon expiration event, but the end result is that the connection is closed before the finished message is sent. When the client tries to send the finished message, it gets a failure.

Expected Behavior

Since the build itself succeeds, we expect the build invocation to succeed, even if there are some communication failures while closing the connection to the daemon.

Current Behavior

When the client tries to send the "finished" message it gets a socket error, causing the build invocation to fail, even if the build itself succeeded.

@ghale ghale added this to the 8.4 RC1 milestone Jul 25, 2023
@ghale ghale self-assigned this Jul 25, 2023
bot-gradle added a commit that referenced this issue Jul 27, 2023
…connection

Once the daemon client has received the build result from the daemon, it tries to send a "finished" message to the daemon to initiate shutdown of the connection.  If the daemon has already shut the connection down from its side, the client should not care since it's preparing to shutdown its side of the connection anyways.  If the daemon is actually waiting, but does not receive the message for some reason, it will just timeout and close the connection.

This changes the client to ignore any errors while sending the "finished" message so that this scenario does not cause a failure in the client on a potentially successful build.

Fixes #25905

Co-authored-by: Gary Hale <gary@gradle.com>
bot-gradle added a commit that referenced this issue Jul 27, 2023
…connection

Once the daemon client has received the build result from the daemon, it tries to send a "finished" message to the daemon to initiate shutdown of the connection.  If the daemon has already shut the connection down from its side, the client should not care since it's preparing to shutdown its side of the connection anyways.  If the daemon is actually waiting, but does not receive the message for some reason, it will just timeout and close the connection.

This changes the client to ignore any errors while sending the "finished" message so that this scenario does not cause a failure in the client on a potentially successful build.

Fixes #25905

Co-authored-by: Gary Hale <gary@gradle.com>
bot-gradle added a commit that referenced this issue Jul 28, 2023
…connection

Once the daemon client has received the build result from the daemon, it tries to send a "finished" message to the daemon to initiate shutdown of the connection.  If the daemon has already shut the connection down from its side, the client should not care since it's preparing to shutdown its side of the connection anyways.  If the daemon is actually waiting, but does not receive the message for some reason, it will just timeout and close the connection.

This changes the client to ignore any errors while sending the "finished" message so that this scenario does not cause a failure in the client on a potentially successful build.

Fixes #25905

Co-authored-by: Gary Hale <gary@gradle.com>
bot-gradle added a commit that referenced this issue Jul 29, 2023
…connection

Once the daemon client has received the build result from the daemon, it tries to send a "finished" message to the daemon to initiate shutdown of the connection.  If the daemon has already shut the connection down from its side, the client should not care since it's preparing to shutdown its side of the connection anyways.  If the daemon is actually waiting, but does not receive the message for some reason, it will just timeout and close the connection.

This changes the client to ignore any errors while sending the "finished" message so that this scenario does not cause a failure in the client on a potentially successful build.

Fixes #25905

Co-authored-by: Gary Hale <gary@gradle.com>
bot-gradle added a commit that referenced this issue Jul 30, 2023
…connection

Once the daemon client has received the build result from the daemon, it tries to send a "finished" message to the daemon to initiate shutdown of the connection.  If the daemon has already shut the connection down from its side, the client should not care since it's preparing to shutdown its side of the connection anyways.  If the daemon is actually waiting, but does not receive the message for some reason, it will just timeout and close the connection.

This changes the client to ignore any errors while sending the "finished" message so that this scenario does not cause a failure in the client on a potentially successful build.

Fixes #25905

Co-authored-by: Gary Hale <gary@gradle.com>
bot-gradle added a commit that referenced this issue Jul 30, 2023
…connection

Once the daemon client has received the build result from the daemon, it tries to send a "finished" message to the daemon to initiate shutdown of the connection.  If the daemon has already shut the connection down from its side, the client should not care since it's preparing to shutdown its side of the connection anyways.  If the daemon is actually waiting, but does not receive the message for some reason, it will just timeout and close the connection.

This changes the client to ignore any errors while sending the "finished" message so that this scenario does not cause a failure in the client on a potentially successful build.

Fixes #25905

Co-authored-by: Gary Hale <gary@gradle.com>
bot-gradle added a commit that referenced this issue Jul 30, 2023
…connection

Once the daemon client has received the build result from the daemon, it tries to send a "finished" message to the daemon to initiate shutdown of the connection.  If the daemon has already shut the connection down from its side, the client should not care since it's preparing to shutdown its side of the connection anyways.  If the daemon is actually waiting, but does not receive the message for some reason, it will just timeout and close the connection.

This changes the client to ignore any errors while sending the "finished" message so that this scenario does not cause a failure in the client on a potentially successful build.

Fixes #25905

Co-authored-by: Gary Hale <gary@gradle.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant