Adding Connection Timeout to fix flaky connection issues

[joshuabremerdexcom]

What existing functionality needs improvement?

For our usage, the buildkitd would fail with the following error once out of every 100 builds or so.

build new buildkitd client: maybe start buildkitd: maybe restart: could not connect to buildkitd to shut down container

This would even happen immediately after a successful build. So our hunch is that everything is working right, but we we need to give a little more time to connect.

I made a Proof of Concept PR here:
joshuabremerdexcom#1

This adds a buildkit-conn-timeout flag to the CLI to let it take longer to connect.

If it's something you'd be willing to add in, I can finish this up and make sure it meets the contributing standards.

Expected Behavior

When you run:

earthly --buildkit-conn-timeout=30 +build

It will take 30 seconds to timeout if it can't connect to the buildkitd container instead of 5 seconds.

[kmannislands]

I think the earthly config.yml would be the better place to specify this sort of configuration.

Not sure if there is overlap with your issue but there's buildkit_restart_timeout_s which can be varied to control initial connection timeout to buildkitd and is specified with:

global:
    buildkit_restart_timeout_s: 120

[joshuabremerdexcom]

Interesting, let me investigate that. I don't think I've looked much at the timing of some of the failures. I'll investigate that before I come back to this issue. For now I'll close this and reopen it if I have more questions or a different proposal. Thank you!