Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: buildlet client needs to continually ping in the background #11109

Closed
bradfitz opened this issue Jun 7, 2015 · 1 comment

Comments

Projects
None yet
4 participants
@bradfitz
Copy link
Member

commented Jun 7, 2015

I've noticed buildlet exec commands hang.

This can be for many reasons: VM crash, preemptible batch resource death (VM pulled away from us), us accidentally deleting the VM, us deleting the VM on purpose, etc.

To catch all those cases, currently we use TCP keep-alives to make sure the TCP connection is alive at least. But that appears to take several minutes, despite us using the http DefaultTransport (with TCP KeepAlive of 30 seconds?).

We should notice much quicker.

We should either reduce the TCP keep-alive time to a few seconds, or just have a background goroutine always pinging the buildlet in the background and interrupt any operation from a buildlet client (be it an exec, PutTar, etc... all commands should wait for both the real operation to complete, or for the buildlet health check goroutine to mark itself dead).

/cc @adg

@ianlancetaylor ianlancetaylor added this to the Unreleased milestone Jun 7, 2015

@rsc rsc added the Builders label Jun 11, 2015

@bradfitz

This comment has been minimized.

Copy link
Member Author

commented Oct 7, 2016

This happened.

@bradfitz bradfitz closed this Oct 7, 2016

@golang golang locked and limited conversation to collaborators Oct 7, 2017

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.