Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

implements StatefulSet update #46669

Merged
merged 3 commits into from Jun 7, 2017

Conversation

@kow3ns
Copy link
Member

commented May 31, 2017

What this PR does / why we need it:

  1. Implements rolling update for StatefulSets
  2. Implements controller history for StatefulSets.
  3. Makes StatefulSet status reporting consistent with DaemonSet and ReplicaSet.

kubernetes/enhancements#188

Special notes for your reviewer:

Release note:

Implements rolling update for StatefulSets. Updates can be performed using the RollingUpdate, Paritioned, or OnDelete strategies. OnDelete implements the manual behavior from 1.6. status now tracks 
replicas, readyReplicas, currentReplicas, and updatedReplicas. The semantics of replicas is now consistent with DaemonSet and ReplicaSet, and readyReplicas has the semantics that replicas did prior to this release.
"updatedReplicas": {
"type": "integer",
"format": "int32",
"description": "updatedReplicas is the number of replicas"

This comment has been minimized.

Copy link
@krmayankk

krmayankk May 31, 2017

Contributor

nit: at updatedRevision

},
"updateRevision": {
"type": "string",
"description": "updateRevision is the revision the StatefulSet is attempting to roll out"

This comment has been minimized.

Copy link
@krmayankk

krmayankk May 31, 2017

Contributor

nit: comma after revision as in updateRevision is the revision, the StatefulSet is ...

@@ -6452,6 +6535,47 @@ <h3 id="_v1_configmapenvsource">v1.ConfigMapEnvSource</h3>

</div>
<div class="sect2">
<h3 id="_v1beta1_statefulsetupdatestrategy">v1beta1.StatefulSetUpdateStrategy</h3>
<div class="paragraph">
<p>StatefulSetUpdateStrategy indicates the strategy that the StatefulSet controller will use to perform updates. It includes any additional parameters necessary to preform the update for the indicated strategy.</p>

This comment has been minimized.

Copy link
@krmayankk

krmayankk May 31, 2017

Contributor

nit: perform

// PartitionStatefulSetStrategyType indicates that updates will only be
// applied to a partition of the StatefulSet. This is useful for canaries
// and phased roll outs. When a scale operation is performed with this
// strategy, new Pods will be created from the updated specification.

This comment has been minimized.

Copy link
@krmayankk

krmayankk May 31, 2017

Contributor

updated specification means updateRevision here, right ?

// tracking and ordered rolling restarts are disabled. Pods are recreated
// from the StatefulSetSpec when they are manually deleted. When a scale
// operation is performed with this strategy, new Pods will be created
// from the current specification.

This comment has been minimized.

Copy link
@krmayankk

krmayankk May 31, 2017

Contributor

current specification means currentRevision here , right ?

// from the StatefulSetSpec when they are manually deleted. When a scale
// operation is performed with this strategy, new Pods will be created
// from the current specification.
OnDeleteStatefulSetStrategyType = "OnDelete"

This comment has been minimized.

Copy link
@krmayankk

krmayankk May 31, 2017

Contributor

is this the default strategy ?

This comment has been minimized.

Copy link
@kow3ns

kow3ns May 31, 2017

Author Member

no. you should look at defaults.go.

type PartitionStatefulSetStrategy struct {
// Ordinal indicates the ordinal at which the StatefulSet should be
// partitioned.
Ordinal int32

This comment has been minimized.

Copy link
@krmayankk

krmayankk May 31, 2017

Contributor

Can this be omitted in the Partitioned case to assume a default of one ? In which case,this should be a pointer to distringuish between the default case .

This comment has been minimized.

Copy link
@krmayankk

krmayankk May 31, 2017

Contributor

i think a default one one assumption will be helpful

This comment has been minimized.

Copy link
@kow3ns

kow3ns May 31, 2017

Author Member

RollingUpdate is default

This comment has been minimized.

Copy link
@kow3ns

kow3ns May 31, 2017

Author Member

I disagree with defaulting the partition. The user has to declare that they want to partition at a particular point.

This comment has been minimized.

Copy link
@krmayankk

krmayankk Jun 1, 2017

Contributor

the default partition size of 1 is the canary case. Most people would have this use case. Without it, it means specifying one extra param for partitioning in the most common case. It seems like a reasonable default with no surprises .

This comment has been minimized.

Copy link
@kow3ns

kow3ns Jun 1, 2017

Author Member

I disagree that canaries are the most common case. I don't think specifying the fraction of a staged roll out is an undo burden.

This comment has been minimized.

Copy link
@smarterclayton

smarterclayton Jun 1, 2017

Contributor

I think we should require partition to be specified. The default doesn't add anything for the machine operated use case which is going to be much more common


// updateStrategy indicates the StatefulSetUpdateStrategy that will be
// employed to update Pods in the StatefulSet when a revision is made to
// Template or VolumeClaimsTemplate.

This comment has been minimized.

Copy link
@krmayankk

krmayankk May 31, 2017

Contributor

Can we remove the VolumeClaimsTemplate from here, since we dont support the update to it.

// currentReplicas is the number of replicas at the CurrentRevision
CurrentReplicas int32

// updatedReplicas is the number of replicas

This comment has been minimized.

Copy link
@krmayankk

krmayankk May 31, 2017

Contributor

nit: at the updateRevision

}
if in.RevisionHistoryLimit != nil {
out.RevisionHistoryLimit = new(int32)
*out.RevisionHistoryLimit = *in.RevisionHistoryLimit

This comment has been minimized.

Copy link
@krmayankk

krmayankk May 31, 2017

Contributor

and if its nil, set it to 2 ?

This comment has been minimized.

Copy link
@kow3ns

kow3ns May 31, 2017

Author Member

defaults.go not conversions

@@ -140,8 +147,37 @@ func Convert_apps_StatefulSetSpec_To_v1beta1_StatefulSetSpec(in *apps.StatefulSe
} else {
out.VolumeClaimTemplates = nil
}
if in.RevisionHistoryLimit != nil {
out.RevisionHistoryLimit = new(int32)
*out.RevisionHistoryLimit = *in.RevisionHistoryLimit

This comment has been minimized.

Copy link
@krmayankk

krmayankk May 31, 2017

Contributor

in nil case, set it to 2

This comment has been minimized.

Copy link
@kow3ns

kow3ns May 31, 2017

Author Member

no that is done in defaults not conversions

UpdateStrategy: apps.StatefulSetUpdateStrategy{Type: "foo"},
},
},
"empty parition": {

This comment has been minimized.

Copy link
@krmayankk

krmayankk May 31, 2017

Contributor

nit: partition

t.Errorf("Expected 0 events for successful status update %d", eventCount)
}
}

This comment has been minimized.

Copy link
@krmayankk

krmayankk May 31, 2017

Contributor

has all this content moved elsewhere ?

This comment has been minimized.

Copy link
@kow3ns

kow3ns May 31, 2017

Author Member

The status tests are in stateful_set_status_updater.go

@@ -305,6 +308,32 @@ func (ssc *StatefulSetController) getPodsForStatefulSet(set *apps.StatefulSet, s
return cm.ClaimPods(pods, filter)
}

// adoptOrphanRevisions adopts any orphaned ControllerRevisions matched by set's Selector.
func (ssc *StatefulSetController) adoptOrphanRevisions(set *apps.StatefulSet) error {
revisions, err := ssc.control.ListRevisions(set)

This comment has been minimized.

Copy link
@krmayankk

krmayankk May 31, 2017

Contributor

is this listing of revisions based on the label on those revisions ?

This comment has been minimized.

Copy link
@kow3ns

kow3ns May 31, 2017

Author Member

Yes the revisions are constructed with the StatefulSet's selector

}
}
if hasOrphans {
fresh, err := ssc.kubeClient.AppsV1beta1().StatefulSets(set.Namespace).Get(set.Name, metav1.GetOptions{})

This comment has been minimized.

Copy link
@krmayankk

krmayankk May 31, 2017

Contributor

curious why cant we use the set.UID instead of querying again ?

This comment has been minimized.

Copy link
@kow3ns

kow3ns May 31, 2017

Author Member

We have to refresh the Object to ensure that it hasn't been deleted ensure that we don't patch a ControllerRevision to be owned by a cached deleted/(deleted and recreated) StatefulSet.

This comment has been minimized.

Copy link
@krmayankk

krmayankk Jun 1, 2017

Contributor

hmm how can you be sure of that ?What if its gets deleted after your query ?

This comment has been minimized.

Copy link
@kow3ns

kow3ns Jun 1, 2017

Author Member

I suppose in theory it could. In practice we have seen issues where a stale cache causes invalid adoption. This pattern is used in all controllers for orphan Pods.

if pod.Labels == nil {
pod.Labels = make(map[string]string)
}
pod.Labels[apps.StatefulSetRevisionLabel] = revision

This comment has been minimized.

Copy link
@krmayankk

krmayankk May 31, 2017

Contributor

check if revision is empty ?

This comment has been minimized.

Copy link
@kow3ns

kow3ns May 31, 2017

Author Member

It can't be empty based on the call path. I don't see a need for a check.

return patch, err
}

// newRevision creates a new ControllerRevision containing a patch that can convert a StatefulSet to an equivalent state to set.

This comment has been minimized.

Copy link
@krmayankk

krmayankk May 31, 2017

Contributor

nit: the last part of the sentence is not clear convert a StatefulSet to an equivalent state to set.

return history.NewControllerRevision(set,
controllerKind,
selector,
runtime.RawExtension{Raw: patch},

This comment has been minimized.

Copy link
@krmayankk

krmayankk May 31, 2017

Contributor

this is confusing and at the same time interesting to me as a concept :-) I have not seen this pattern before, but it seems like what you are doing is keeping a strategic merge patch in each revision history , so that any time we want to restore the StatefulSet to that revision, you just use the ready to apply patch from the Raw field to restore it. I think this is worth some more comment

This comment has been minimized.

Copy link
@kow3ns

kow3ns Jun 1, 2017

Author Member

I added commentary to that effect above the getPatch and newRevision methods.

return nil, err
}
clone := obj.(*apps.StatefulSet)
patched, err := strategicpatch.StrategicMergePatch([]byte(runtime.EncodeOrDie(patchCodec, clone)), revision.Data.Raw, clone)

This comment has been minimized.

Copy link
@krmayankk

krmayankk May 31, 2017

Contributor

check for err here ?

This comment has been minimized.

Copy link
@kow3ns

kow3ns May 31, 2017

Author Member

I added a check under strategicpatch
If you mean check that the clone casts correctly, we generally do not do that when deep copying from using Scheme.

if err != nil {
return nil, err
}
return clone, err

This comment has been minimized.

Copy link
@krmayankk

krmayankk May 31, 2017

Contributor

return clone, nil

func nextRevision(revisions []*apps.ControllerRevision) int64 {
if count := len(revisions); count <= 0 {
return 1
} else {

This comment has been minimized.

Copy link
@krmayankk

krmayankk May 31, 2017

Contributor

remove the else

t.Fatal(err)
}
set.Spec.Template.Spec.Containers[0].Name = "foo"
restored, err := applyRevision(set, revision)

This comment has been minimized.

Copy link
@krmayankk

krmayankk May 31, 2017

Contributor

nit: restoredSet ?

if err != nil {
t.Fatal(err)
}
if !history.EqualRevision(revision, restoredRevision) {

This comment has been minimized.

Copy link
@krmayankk

krmayankk May 31, 2017

Contributor

do you ignore the revision number when comparing the revisions ?

This comment has been minimized.

Copy link
@kow3ns

kow3ns May 31, 2017

Author Member

Yes the revisions are only compared by hash and by contents. Two revisions with different ".Revisions" can be equal using this comparison.

if err != nil {
return "", fmt.Errorf("failed to retrieve statefulset %s", err)
}
revisions, err := h.c.Apps().ControllerRevisions(namespace).List(metav1.ListOptions{LabelSelector: sts.Spec.Selector.String()})

This comment has been minimized.

Copy link
@krmayankk

krmayankk May 31, 2017

Contributor

will this sometimes also report orphaned revisions ? To avoid that, we should further validate if the ownerReferences are for this set

),
recorder),
NewRealStatefulSetStatusUpdater(kubeClient, setInformer.Lister()),
history.NewHistory(kubeClient, revInformer.Lister()),
),
pvcListerSynced: pvcInformer.Informer().HasSynced,

This comment has been minimized.

Copy link
@kargakis

kargakis Jun 6, 2017

Member

Since you are using the controllerRevision cache to list CRs, you should wait for it to be synced before starting the sts controller. Means you need a crListerSynced field which you then pass in controller.WaitForCacheSync (L150).

@wojtek-t

This comment has been minimized.

Copy link
Member

commented Jun 6, 2017

/approve no-issue (there is issue from features repo, but bot still doesn't understand it).

@wojtek-t

This comment has been minimized.

Copy link
Member

commented Jun 6, 2017

/approve no-issue

@kargakis

This comment has been minimized.

Copy link
Member

commented Jun 6, 2017

there is issue from features repo, but bot still doesn't understand it

FWIW kubernetes/enhancements#188

@kow3ns kow3ns force-pushed the kow3ns:statefulset-update branch from 1af3cf1 to 6b41b34 Jun 6, 2017

@kow3ns kow3ns force-pushed the kow3ns:statefulset-update branch from 6b41b34 to bec6c55 Jun 6, 2017

Kenneth Owens added some commits Jun 4, 2017

Kenneth Owens
Implements StatefulSet update
Implements history utilities for ControllerRevision in the controller/history package
StatefulSetStatus now has additional fields for consistency with DaemonSet and Deployment
StatefulSetStatus.Replicas now represents the current number of createdPods and StatefulSetStatus.ReadyReplicas is the current number of ready Pods

@kow3ns kow3ns force-pushed the kow3ns:statefulset-update branch from bec6c55 to fabbcd6 Jun 6, 2017

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

commented Jun 6, 2017

@kow3ns: The following test failed, say /retest to rerun them all:

Test name Commit Details Rerun command
pull-kubernetes-federation-e2e-gce c560deb link @k8s-bot pull-kubernetes-federation-e2e-gce test this

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@kow3ns kow3ns force-pushed the kow3ns:statefulset-update branch from fabbcd6 to 1a784ef Jun 6, 2017

return nil, err
}
var raw map[string]interface{}
json.Unmarshal([]byte(str), &raw)

This comment has been minimized.

Copy link
@janetkuo

janetkuo Jun 6, 2017

Member

Would you want to check the returned error?

// PodSpecTemplate. We can modify this later to encompass more state (or less) and remain compatible with previously
// recorded patches.
func getPatch(set *apps.StatefulSet) ([]byte, error) {
str, err := runtime.Encode(patchCodec, set)

This comment has been minimized.

Copy link
@janetkuo

janetkuo Jun 6, 2017

Member

Remove patchCodec and use json.Marshal here instead, csi team wants to avoid using api.Codecs as much as possible, see #47075 (comment)

@janetkuo

This comment has been minimized.

Copy link
Member

commented Jun 6, 2017

LGTM, comments can be fixed in follow-up PRs

@janetkuo

This comment has been minimized.

Copy link
Member

commented Jun 6, 2017

/lgtm

@k8s-github-robot

This comment has been minimized.

Copy link
Contributor

commented Jun 6, 2017

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: janetkuo, kargakis, kow3ns, smarterclayton, wojtek-t

Associated issue: 188

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

@k8s-github-robot

This comment has been minimized.

Copy link
Contributor

commented Jun 7, 2017

Automatic merge from submit-queue (batch tested with PRs 46235, 44786, 46833, 46756, 46669)

@k8s-github-robot k8s-github-robot merged commit 0613ae5 into kubernetes:master Jun 7, 2017

9 checks passed

Submit Queue Queued to run github e2e tests a second time.
Details
cla/linuxfoundation kow3ns authorized
Details
pull-kubernetes-bazel Job succeeded.
Details
pull-kubernetes-e2e-gce-etcd3 Jenkins job succeeded.
Details
pull-kubernetes-e2e-kops-aws Jenkins job succeeded.
Details
pull-kubernetes-kubemark-e2e-gce Jenkins job succeeded.
Details
pull-kubernetes-node-e2e Jenkins job succeeded.
Details
pull-kubernetes-unit Jenkins job succeeded.
Details
pull-kubernetes-verify Jenkins job succeeded.
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.