Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kubeadm: wait for the etcd cluster to be available when growing it #72984

Merged
merged 1 commit into from
Jan 20, 2019
Merged

kubeadm: wait for the etcd cluster to be available when growing it #72984

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
21 changes: 15 additions & 6 deletions cmd/kubeadm/app/phases/etcd/local.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ import (
"os"
"path/filepath"
"strings"
"time"

"github.com/pkg/errors"
"k8s.io/klog"
Expand All @@ -36,8 +37,10 @@ import (
)

const (
etcdVolumeName = "etcd-data"
certsVolumeName = "etcd-certs"
etcdVolumeName = "etcd-data"
certsVolumeName = "etcd-certs"
etcdHealthyCheckInterval = 5 * time.Second
etcdHealthyCheckRetries = 20
)

// CreateLocalEtcdStaticPodManifestFile will write local etcd static pod manifest file.
Expand All @@ -61,13 +64,13 @@ func CreateLocalEtcdStaticPodManifestFile(manifestDir string, cfg *kubeadmapi.In
return err
}

klog.V(1).Infof("[etcd] wrote Static Pod manifest for a local etcd instance to %q\n", kubeadmconstants.GetStaticPodFilepath(kubeadmconstants.Etcd, manifestDir))
klog.V(1).Infof("[etcd] wrote Static Pod manifest for a local etcd member to %q\n", kubeadmconstants.GetStaticPodFilepath(kubeadmconstants.Etcd, manifestDir))
return nil
}

// CheckLocalEtcdClusterStatus verifies health state of local/stacked etcd cluster before installing a new etcd member
func CheckLocalEtcdClusterStatus(client clientset.Interface, cfg *kubeadmapi.InitConfiguration) error {
fmt.Println("[etcd] Checking Etcd cluster health")
fmt.Println("[etcd] Checking etcd cluster health")

// creates an etcd client that connects to all the local/stacked etcd members
klog.V(1).Info("creating etcd client that connects to etcd pods")
Expand Down Expand Up @@ -120,7 +123,13 @@ func CreateStackedEtcdStaticPodManifestFile(client clientset.Interface, manifest
return err
}

fmt.Printf("[etcd] Wrote Static Pod manifest for a local etcd instance to %q\n", kubeadmconstants.GetStaticPodFilepath(kubeadmconstants.Etcd, manifestDir))
fmt.Printf("[etcd] Wrote Static Pod manifest for a local etcd member to %q\n", kubeadmconstants.GetStaticPodFilepath(kubeadmconstants.Etcd, manifestDir))

fmt.Printf("[etcd] Waiting for the new etcd member to join the cluster. This can take up to %v\n", etcdHealthyCheckInterval*etcdHealthyCheckRetries)
if _, err := etcdClient.WaitForClusterAvailable(etcdHealthyCheckRetries, etcdHealthyCheckInterval); err != nil {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't like this function printing

    [util/etcd] Attempt timed out
    [util/etcd] Waiting 5s until next retry
    [util/etcd] Attempt timed out
    [util/etcd] Waiting 5s until next retry

IMO those output should be removed (or converted into log messages) in order to be consistent with all the other waiters in kubeadm.

However, considering that this requires to add "This can take up to ..." in every place where WaitForClusterAvailable is used, this goes out of the scope of this PR, so please open an issue to track this as a todo/good first issue

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I proposed to use klog here too but @rosti didn't want to address this change on this PR, only changing the one potentially long with the endpoints. I agree with your point of view though @fabriziopandini.

@rosti, wdyt? Should I change this now that @fabriziopandini also raised this issue?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Marked as resolved as per discussion with @fabriziopandini, leaving it as it was as @rosti proposed.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do think, that we need some sort of indication about the reason we wait another 5 seconds. This is tightly coupled with the UX of end users, that run kubeadm directly on command line. For that matter I am not a fan of klogging this. In my opinion it should go out via print.
On the other hand, we can certainly reduce the output here to say a single, more descriptive message per retry.
However, as @fabriziopandini mentioned, this will require changes in a few more places, thus it may better be done in another PR. We can file a backlog issue for now.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i wanted to get more feedback on the 5 seconds and the interval of 20 tries.
if we can get a check faster than 5 on the average, possibly we can reduce the value?
also 20 tries is a lot. in reality, we might get the failed state much sooner.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the last run it took 4 retries (of 5 seconds interval), this one was way off-charts and with a clean environment :(

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i've mentioned this on slack:

we may want to keep the overall time under 40seconds to match the kubelet timeout.
how about 2 seconds rate with 20 retries.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@fabriziopandini @rosti please give you stamp of approval for the above comment.

Copy link
Contributor

@rosti rosti Jan 18, 2019 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the 40 seconds idea, but let's keep the steps at 5 sec. Bear in mind, that we have just written out the static pod spec, so the kubelet needs to detect it, spin it up and for etcd to become responsive. On some systems it's easy for this to come above 2 seconds.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, @rosti is voting for 5 sec / 8 retries.
@fabriziopandini ?

return err
}

return nil
}

Expand Down Expand Up @@ -172,7 +181,7 @@ func getEtcdCommand(cfg *kubeadmapi.InitConfiguration, initialCluster []etcdutil
if len(initialCluster) == 0 {
defaultArguments["initial-cluster"] = fmt.Sprintf("%s=%s", cfg.GetNodeName(), etcdutil.GetPeerURL(cfg))
} else {
// NB. the joining etcd instance should be part of the initialCluster list
// NB. the joining etcd member should be part of the initialCluster list
endpoints := []string{}
for _, member := range initialCluster {
endpoints = append(endpoints, fmt.Sprintf("%s=%s", member.Name, member.PeerURL))
Expand Down
18 changes: 15 additions & 3 deletions cmd/kubeadm/app/util/etcd/etcd.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ import (
"crypto/tls"
"fmt"
"net"
"net/url"
"path/filepath"
"strconv"
"strings"
Expand Down Expand Up @@ -73,7 +74,7 @@ func New(endpoints []string, ca, cert, key string) (*Client, error) {
return &client, nil
}

// NewFromCluster creates an etcd client for the the etcd endpoints defined in the ClusterStatus value stored in
// NewFromCluster creates an etcd client for the etcd endpoints defined in the ClusterStatus value stored in
// the kubeadm-config ConfigMap in kube-system namespace.
// Once created, the client synchronizes client's endpoints with the known endpoints from the etcd membership API (reality check).
func NewFromCluster(client clientset.Interface, certificatesDir string) (*Client, error) {
Expand Down Expand Up @@ -146,7 +147,15 @@ type Member struct {
}

// AddMember notifies an existing etcd cluster that a new member is joining
func (c Client) AddMember(name string, peerAddrs string) ([]Member, error) {
func (c *Client) AddMember(name string, peerAddrs string) ([]Member, error) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as a note/TODO: in a separate PR we need to make all the client methods to use pointers.
golang encourages a matching pattern.

// Parse the peer address, required to add the client URL later to the list
// of endpoints for this client. Parsing as a first operation to make sure that
// if this fails no member addition is performed on the etcd cluster.
parsedPeerAddrs, err := url.Parse(peerAddrs)
if err != nil {
return nil, errors.Wrapf(err, "error parsing peer address %s", peerAddrs)
}

cli, err := clientv3.New(clientv3.Config{
Endpoints: c.Endpoints,
DialTimeout: 20 * time.Second,
Expand Down Expand Up @@ -176,6 +185,9 @@ func (c Client) AddMember(name string, peerAddrs string) ([]Member, error) {
}
}

// Add the new member client address to the list of endpoints
c.Endpoints = append(c.Endpoints, GetClientURLByIP(parsedPeerAddrs.Hostname()))
ereslibre marked this conversation as resolved.
Show resolved Hide resolved

return ret, nil
}

Expand Down Expand Up @@ -255,7 +267,7 @@ func (c Client) WaitForClusterAvailable(retries int, retryInterval time.Duration
fmt.Printf("[util/etcd] Waiting %v until next retry\n", retryInterval)
time.Sleep(retryInterval)
}
fmt.Printf("[util/etcd] Attempting to see if all cluster endpoints are available %d/%d\n", i+1, retries)
klog.V(2).Infof("attempting to see if all cluster endpoints (%s) are available %d/%d", c.Endpoints, i+1, retries)
resp, err := c.ClusterAvailable()
if err != nil {
switch err {
Expand Down