Retry controller join for transient failures #962
Conversation
| err = retry.Do(func() error { | ||
| caData, err = joinClient.GetCA() | ||
| if err != nil { | ||
| return fmt.Errorf("failed to sync CA: %w", err) | ||
| } | ||
| return nil | ||
| }) |
There was a problem hiding this comment.
FYI, total retry time is ~52 seconds, pause between two last attempts is 25 seconds.
Maybe could be improved by giving up early in case of some unrecoverable failures:
err = retry.Do(
func() error {
....
},
retry.RetryIf(
func(err error) bool {
return !strings.Contains(err.Error(), "something")
},
),
)There was a problem hiding this comment.
Maybe could be improved by giving up early in case of some unrecoverable failures
I actually did think of this, but it's just super hard to make difference what is un-recoverable and what is not. E.g. with k0sctl (as it's super fast) the initial trys might give plain IO timeout (the join API is not yet up on the first controller), some RBAC related 40x (initial controller not yet pushed RBAC stuff for join tokens) or something else.
FYI, total retry time is ~52 seconds, pause between two last attempts is 25 seconds.
I think this is OK for this case.
Signed-off-by: Jussi Nummelin <jnummelin@mirantis.com>
jnummelin
force-pushed
the
retry-controller-join
branch
from
4298e92
to
85a7104
Compare
Signed-off-by: Jussi Nummelin jnummelin@mirantis.com
Issue
Fixes #866
What this PR Includes
This PR makes controller join process to retry the CA sync. With automation tools like k0sctl the controllers are joined so fast that on the initial controller the sync API might not yet be up-and-running. That'll make the underlying init system (e.g. systemd) to restart the service. The job will get done for sure, but it'll take a bit long. Testing with this fix and with k0sctl shows this shaves ~1min off from the 3+3 setup.