Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: buildlet client needs to continually ping in the background #11109

bradfitz opened this issue Jun 7, 2015 · 1 comment
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge


Copy link

bradfitz commented Jun 7, 2015

I've noticed buildlet exec commands hang.

This can be for many reasons: VM crash, preemptible batch resource death (VM pulled away from us), us accidentally deleting the VM, us deleting the VM on purpose, etc.

To catch all those cases, currently we use TCP keep-alives to make sure the TCP connection is alive at least. But that appears to take several minutes, despite us using the http DefaultTransport (with TCP KeepAlive of 30 seconds?).

We should notice much quicker.

We should either reduce the TCP keep-alive time to a few seconds, or just have a background goroutine always pinging the buildlet in the background and interrupt any operation from a buildlet client (be it an exec, PutTar, etc... all commands should wait for both the real operation to complete, or for the buildlet health check goroutine to mark itself dead).

/cc @adg

@ianlancetaylor ianlancetaylor added this to the Unreleased milestone Jun 7, 2015
@rsc rsc added the Builders x/build issues (builders, bots, dashboards) label Jun 11, 2015
Copy link
Contributor Author

bradfitz commented Oct 7, 2016

This happened.

@bradfitz bradfitz closed this as completed Oct 7, 2016
@golang golang locked and limited conversation to collaborators Oct 7, 2017
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge
None yet

No branches or pull requests

4 participants