StatefulSet should allow optional burst mode (don't wait for readiness)

[brendandburns]

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/.): No

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.):

Petset
DNS
StatefulSet

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

Kubernetes version (use kubectl version): 1.5.1

Environment:

• Cloud provider or hardware configuration: minikube 0.14.0
• OS (e.g. from /etc/os-release): Debian

What happened:
DNS appears to not register host names for petset members (e.g. mongo-0.mongo...) until that Pod passes a readiness check.

This is problematic because you may want to configure everything to use that hostname (e.g. registering a replication participant with mong) prior to declaring that the cluster is ready.

Given that there is nothing about readiness in the DNS address (unlike a traditional Service) we
should register DNS before the Pod is "ready".

I suspect this is because we are re-using the standard endpoints controller here, we probably need
a special case for stateful set.

How to reproduce it (as minimally and precisely as possible):

Create a StatefulSet with a readiness delay of 20 seconds.
Exec into the pod and try to resolve the DNS address of the Pod
observe it fails.
Wait until the Pod is ready
observe it works.

Anything else do we need to know:
Nope

@bprashanth @smarterclayton

[hongchaodeng]

Same as #39207. We encountered this problem for a while...

IMHO, this doesn't mean that we need to change readiness guarantee. But we need a special phase that isn't coupled to networking and we can use it to check service has been bootstrapped.

[smarterclayton]

If you set the annotation for tolerate unready you should see DNS entries almost instantly. Can you double check that?

[hongchaodeng]

You mean this field?

TolerateUnreadyEndpointsAnnotation = "service.alpha.kubernetes.io/tolerate-unready-endpoints"

Didn't realize it. Will give a try!

[resouer]

@smarterclayton What do you think about tolerate-unready-endpoint as default?

[bprashanth]

See the other petset examples: https://github.com/kubernetes/charts/blob/master/incubator/zookeeper/templates/zookeeper-petset.yaml#L12 and #25283, #37093

[smarterclayton]

That would break existing applications. Readiness is a good thing, it's unusual for software to tolerate unreadiness. Tolerate-unready-endpoint also breaks deployments (you'd have things taking traffic that can't serve it)

…

On Wed, Jan 4, 2017 at 6:42 AM, Prashanth B ***@***.***> wrote: See the other petset examples: https://github.com/kubernetes/ charts/blob/master/incubator/zookeeper/templates/zookeeper-petset.yaml#L12 and #25283 <#25283>, #37093 <#37093> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#39363 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABG_pzKPEcMGw574bQHO1K5JMMZgHEfbks5rO4WvgaJpZM4LZVwL> .

[kow3ns]

@hongchaodeng Can we close this issue as resolved?

[smarterclayton]

There's been some discussion that for a subset of apps the SS controller should be able to fill out gaps more quickly - to not wait for readiness to continue. We'd need to clearly explain what the risks are for that:

1. If you must form an initial quorum, and someone rescaled you quickly, your app would need to either only join with pod-0, or ignore additional joiners until initial quorum is filled (otherwise, high members could form a different quorum).
2. The app must not assume that pods join in order, which is unlikely to be normal.

The benefit of such a change would be that for apps that can form stable quorums safely they can do so much more quickly. For instance, and etcd set of 3 with an env var EXPECT_INITIAL=3 could only allow 0-2 to join before 3+ joins. And could disallow 5-6 from joining until 3-4 check in. This would also allow readiness to be more effective (pod is ready once it joins quorum, instead of before).

This is something that would have to be opt in and may have implications for future update scenarios (we'd want to talk through that before adding anything beyond an alpha feature).

@kubernetes/sig-apps-feature-requests for additional opinions and comments. Was raised to be in the context of ES data nodes (which are non-voting) and AP stores which don't assume quorum is even possible (like infinispan). I'd be willing to endorse an alpha annotation that allows SS to proceed to create new pods in this fashion.

[smarterclayton]

(Im repurposing this since in spirit both outcomes described are desirable)

[erictune]

@kow3ns

[smarterclayton]

Relevant to #38418 - same problem as I described above in a different form.

[kow3ns]

@smarterclayton I think #38418 is somewhat separate, but I concur that we should allow an annotation that optimizes for burst roll outs.
I was thinking something like [statefulset.alpha.kubernetes.io/burst]= to allow up to sequential replicas to be concurrently unready simultaneously with [statefulset.alpha.kubernetes.io/burst]="all" being equivalent to =set.Spec.Replicas.
Where you thinking along these lines or did you have something else in mind?

[smarterclayton]

Yeah, that works for me. We could consider "burst: majority" or a percentage to allow someone who isn't dependent on ordering to ensure that the controller will keep trying to fill the slots to achieve a majority even if the rate of evictions > rate of renewal. But burst=all would be the first knob to try probably.

[ethernetdan]

Moving to 1.7 as late to happen in 1.6. Feel free to switch back if this is incorrect.

[tslater]

@smarterclayton @hongchaodeng Just to clarify, service.alpha.kubernetes.io/tolerate-unready-endpoints is only for services, right? I'm using that on my headless service, but I'm not able to use it with my StatefulSet, right? There's no way to do that ahead of this bursting proposal, with the current api?

[smarterclayton]

Correct. I was going to open a PR for the bursting if I get a chance in the next few days.

[tslater]

That's awesome. After getting started with StatefulSets, I can definitely see why this feature is important. I really appreciate all this great work from the community. Everytime I need something, it is is either already in K8s, or being worked on. It's amazing!

…

On Mon, Apr 24, 2017 at 12:09 PM, Clayton Coleman ***@***.***> wrote: Correct. I was going to open a PR for the bursting if I get a chance in the next few days. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#39363 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAbbLi6o_g7AJvHCMbXm6Q1wHiBOJ6Fmks5rzOVQgaJpZM4LZVwL> .

[enisoc]

The original purpose of this issue seems to be addressed by tolerate-unready-endpoints. I'm renaming it to reflect the shift in purpose since I don't see that discussion captured elsewhere.

[smarterclayton]

Bursting was implemented in 1.7 in alpha and will be beta in 1.8.