[Federation] Federated statefulsets design proposal #437

irfanurrehman · 2017-03-07T18:20:08Z

Design proposal for federated statefulsets.

@kubernetes/sig-federation-misc @kubernetes/sig-federation-proposals @kubernetes/sig-federation-pr-reviews

cc @quinton-hoole @deepak-vij @shashidharatd @dhilipkumars

Please feel free to cc any probable reviewers, not part of the targeted groups.

liggitt · 2017-03-07T19:46:09Z

cc @kubernetes/sig-federation-misc @smarterclayton

smarterclayton · 2017-03-08T01:14:46Z

contributors/design-proposals/federated-statefulsets.md

+
+1 – An unique, consistent and discoverable identity for each replica/instance across the federated clusters.
+
+2 – Predictability on the number of instances and sequentialization of pod creation (for example instance 1 creation starts only when instance 0 is up and running).


We've discussed relaxing this.

Yes, I have specified (later in the doc) that we might not be able to guarantee this requirement for the federated statefulsets.

I have now dropped this requirement altogether from this section.

smarterclayton · 2017-03-08T01:16:36Z

Please break your lines at some fixed interval, the proposal is unreviewable asnis

irfanurrehman · 2017-03-08T07:08:47Z

Please break your lines at some fixed interval, the proposal is unreviewable asnis

Thanks @smarterclayton for having a look. I have added line breaks on all the lines. PTAL.

dhilipkumars · 2017-03-08T18:59:24Z

CC : @erictune @Kargakis @kow3ns

madhusudancs · 2017-03-22T22:16:45Z

Some high level questions:

What happens when a new cluster joins or a cluster leaves federation? Do you rebalance?
Say I have 2 clusters A and B, and replicas: 5. Cluster A gets 3 and B gets 2 replicas. Cluster A dies. According to your limitation Port over SIG table/docs from kubernetes.wiki #3, there will be only 2 replicas and no quorum now. This means the stateful set cannot function even though there is enough capacity in the other cluster. Is that correct?

smarterclayton · 2017-03-23T02:31:49Z

contributors/design-proposals/federated-statefulsets.md

+
+If we consider the use cases listed above, the main design requirements can roughly be listed as:
+
+1 – An unique, consistent and discoverable identity for each replica/instance across the federated clusters.


I'd like a new design requirement.

It must be possible for the federated set to SAFELY form an initial quorum that adds the rest of the set.

Any design which doesn't allow (for instance) cluster 1 to form an initial quorum and then safely add members in cluster 2 (or any other order, really), is a non-starter because then stateful set at federation has a different set of guarantees.

I think that's implicit in some of your comments, but it needs to be explicit, obvious, and impossible to break.

When you mention quorum what quorum are you referring to?

@smartclayton I have mentioned the requirement explicitly as you suggested. I have also dropped the sequentialization requirement, because we are not honoring that as of now.
@chrislovecnm I think @smarterclayton is referring to the quorum application pod instances form after discovering each other.

chrislovecnm

Questions. Looks awesome!!!

chrislovecnm · 2017-03-25T01:02:45Z

contributors/design-proposals/federated-statefulsets.md

+1 – A stateful app, for the reasons of high availability, wants the stateful pods distributed in different clusters, such that the set can withstand cluster failures. This represents an app with one single global quorum.
+
+2 – A stateful app, wants replicas distributed in multiple clusters, such that it can form multiple smaller sized local clusters using only local replicas.
+


What about two stateful sets that need to comminicate with each other. Many applications such as Cassandra, Kafka and elasticsearch include the capability to span physical data centers?

The scenario you mention is infact solved by design alternative 2 specified below, where a particular statefulset instance gets a local and global identity both. Applications can choose to use either identities or both at the same time. The use cases I mentioned are for reference alone and not exhaustive. Do you think its important to mention another use case specifying the scenario u mention?

chrislovecnm · 2017-03-25T01:12:32Z

contributors/design-proposals/federated-statefulsets.md

+
+## Storage volumes
+
+There are two ways a federated statefulset can be assigned a persistent storage.


How are we handling storage classes? Am I my missing that section?

I think we need not mention anything related to the storage classes. The chosen design proposal intends to reuse the cluster local statefulsets implementation, when the statefulsets are deployed in individual clusters.
One catch might be that the some clusters among the federated clusters might not have the given storage class available. I think for now, we leave it to the user to ensure that the specified storage class is available across all federated clusters that he/she mentions in the federated statefulset spec. The behavior if this does not happen is also deterministic for now. The volume provisioning for the stateful pods in a particular cluster which does not have that class will fail.

Should we say that we are going to test against them?

I am sorry; I did not get you here.. ? You mean test against storage classes.. ?

chrislovecnm · 2017-03-25T01:13:36Z

contributors/design-proposals/federated-statefulsets.md

+
+elaborated by @quinton-hoole
+
+Strictly speaking, migration between clusters in the same zone is quite feasible.


Please add more clarity to this statement.

More content as well.

Sure, Will update with something more explanatory.

More detail is still required here. If storage can be replicated between clusters (at least those hosted by the same infrastructure e.g. gce), then including the cluster name in the identity doesn't seem like such a good idea.

chrislovecnm · 2017-03-25T01:15:07Z

contributors/design-proposals/federated-statefulsets.md

+
+If we consider the use cases listed above, the main design requirements can roughly be listed as:
+
+1 – An unique, consistent and discoverable identity for each replica/instance across the federated clusters.


When you mention quorum what quorum are you referring to?

chrislovecnm · 2017-03-25T01:16:00Z

contributors/design-proposals/federated-statefulsets.md

+
+2 – Predictability on the number of instances and sequentialization of pod creation (for example instance 1 creation starts only when instance 0 is up and running).
+
+3 – The ability to scale, across clusters in some deterministic fashion.


Can this be optional? Some technologies do not allow more than one pod at a time.

The scaling feature will be on demand, but I think we need to have the ability in the federated statefulsets.

chrislovecnm · 2017-03-25T01:18:33Z

contributors/design-proposals/federated-statefulsets.md

+
+_3 – What happens if a cluster dies_
+
+Nothing, statefulset would need to run with lesser replicas


One section that I am not seeing is networking. Often ha distributed applications use patterns such as gossip. That can require every pod to talk to every other pod. Thoughts around how we are going to handle those patterns?

We handle this using the dns names similar to current k8s statefulset implementation. The "Instance identity and discovery" sections specify the details.
Apps need to discover and use the dns names to communicate across instances and not the IPs.

But how will the routing work? Does CNI provide the capability to route between a headless service in the uk and a headless service in the US? Kinda confused a bit.

This routing is not handled by CNI.
This routing will be handled very similar to the way it is handled for federated services.
(check one of my other reply to your comments about need of ELB).

irfanurrehman · 2017-03-27T18:28:19Z

Some high level questions:

What happens when a new cluster joins or a cluster leaves federation? Do you rebalance?

In the current proposal, when a new cluster joins and the federation already has a running statefulset, nothing will happen (meaning no rebalance). However if the statefulset is scaled after the cluster joins it might get replica(s).

Say I have 2 clusters A and B, and replicas: 5. Cluster A gets 3 and B gets 2 replicas. Cluster A dies. According to your limitation #3, there will be only 2 replicas and no quorum now. This means the stateful set cannot function even though there is enough capacity in the other cluster. Is that correct?

What you mention is a valid scenario, but this is probably something we need to live with as of now. I believe the federated statefulsets will still be pretty useful even with this constraint.
Also this constraint (no migration) is mainly because there is no direct or automated way of migrating the persistent storage as is across k8s clusters. Any suggestions relating this are certainly welcome.

irfanurrehman · 2017-03-27T18:30:19Z

Thanks @smarterclayton @chrislovecnm @madhusudancs for the comments!
Some updates, PTAL!

chrislovecnm

Couple more questions :)

chrislovecnm · 2017-03-30T22:09:19Z

contributors/design-proposals/federated-statefulsets.md

+
+In the case of in-cluster statefulset, pods discover each other using the in-cluster dns names.
+A headless service with selectors enables creation of dns records against the pod names.
+This cannot work across clusters, as local pod IPs cannot be reached across the clusters.


Is this true with all CNI providers or are we just saying that this is the expectation?

As of now (and in my limited knowledge) the CNI overlays are limited within the cluster. However there probably isn't much of a technical bottleneck which stops the overlay to work across clusters. Perhaps that might be another proposal for cluster federation in the pipeline if the need arises. Right now this is true with all CNI providers (and again in my limited knowledge :) ).

There's nothing preventing a CNI overlay from working cross-cluster. Tigera has discussed use cases involving multi-cluster overlay in the past.

@caseydavenport Thoughts?

If that is true and if there is a solution, which can easily enable federated objects to use or access networks across k8s clusters; I would want to pursue it.
In the absence of such available or usable solution, we go ahead with the current proposal, and improve the same when a cross cluster overlay is easy to achieve or inherently built in as cross cluster communication (probably another proposal within the scope of federation)

@irfanurrehman because this is internal to the cluster using a networking solution that can mesh or expose pod ips, is a possible solution. The networking is complex, so for smaller deployments loadbalancers may be another solution as well.

This is a challenging problem to say the least ;) But I have done a POC with weave and have gotten the network setup that I would need.

chrislovecnm · 2017-03-30T22:15:10Z

contributors/design-proposals/federated-statefulsets.md

+A headless service with selectors enables creation of dns records against the pod names.
+This cannot work across clusters, as local pod IPs cannot be reached across the clusters.
+
+The proposal to make it work across the clusters is to assign a service type &#39;LoadBalancer&#39; for each pod instance that is created in the clusters.


Is there any way to do this differently? Think about this at scale. Cost, quotas and provisioning would get seriously fun :) How are we doing guaranteed network identity for each ELB?

Also one of the amazing advantages is to be able to have a baremetal stateful set talking to a cloud stateful set. How are we going to allow for that? Think about expanding your footprint during a busy shopping season.

I concur, this is a horrible idea. Imagine trying to manage a Cassandra cluster of hundreds, of nodes per DC. This isn't feasible. Do one headless service per k8s cluster, and then create a VPN link between virtual networks. Setup DNS resolution across clusters.

@chrislovecnm you are completely right about the cost involved in provisioning ELB's. Right now the options (using k8s available features) to communicate across clusters are:

ELB's provisioned from the cloud provider

Ingress (autoprovisioning is tricky, and is quite cloud specific)

Nodeport

As of today, the federated services have out of the box support for ELB only and a proper working for ingress and nodeport for federated services is still evolving (somebody please correct me if I am wrong). This proposal in this design in fact is to adhere to the same facility/evolution of federated services for communication across clusters.
The point I am trying to make is that, there already are use cases which demand cross cluster/federated statefulsets and would benefit from, even with the current proposal in place.
Better solutions can evolve over time.
@mstump thanks for your suggestion. Can u please elaborate a little bit more with the point of this functionality getting into some existing k8s/federation feature, or it being a new feature in itself. The point is the overlay u mention ideally need to be possible as an auto provisioned method of communication across clusters. If it makes sense in near term and is doable, I would not mind pursuing the same.

chrislovecnm · 2017-03-30T22:18:23Z

contributors/design-proposals/federated-statefulsets.md

+In this approach, the federated statefulset controller will behave quite similar to federated replicaset or the federated deployment controller.
+The federated controller would create and monitor individual statefulsets (rather then pods directly) partitioning and distributing the total stateful replicas across the federated clusters.
+
+As a proposal in this design we suggest the possibility of the pods having multiple identities.


Doesn't this break the contract that statefuls sets already have? I probably need more details to fully understand this design case.

No it doe not. In a normal in-cluster statefulset a stateful pod gets 2 identities:

the dns name accessible within the cluster

the hostname visible to the stateful app

The suggestion in this design retains both and then adds another dns name accessible across the cluster. I specify the additional dns name as another identity, thus multiple identities. I dont know of any such use case as of now, but if needed the same design can be extended to have even more dns names visible locally or globally reflecting more identities.

chrislovecnm · 2017-03-30T22:26:34Z

contributors/design-proposals/federated-statefulsets.md

+
+We propose using alternative 1 listed here, as it fits broader scheme of things and is more consistent with the user expectation of being able to query all needed resources from the federation control plane and is less confusing to use at the same time.
+
+# Conclusion


With some applications, you can only create one pod at a time. How does this design proposal maintain the ordinal order as defined by the stateful set contract?

Does this proposal define the capability of migrating from a regular stateful set to a federated stateful set?

If u see line 99 (and as suggested by @smarterclayton ) the proposal is to not have a hard requirement to preserve the order of stateful pod creation.
This document has not handled the migration of a regular in-cluster statefulset to a federated one. Do u think its needed to address it in this design.. ?

mstump · 2017-03-31T03:31:22Z

contributors/design-proposals/federated-statefulsets.md

+
+It is known, that as of now, the same persistent volume, even if can be used in a different cluster, k8s directly does not provide an API, yet, which can aid the same. In the absence of no direct way of quick migration of the storage data from one zone to another, or from one cloud provider to another if the case be, the proposal is to disallow migration of pods across clusters.
+
+## Scale up/Scale down


For the database and search systems that I can think of this would be undesirable. k8s isn't workload aware, what it means to scale up, re-shard data and all the implications of those actions. Additionally, workloads across DCs are not always homogenous or even distributed, you want to be able to scale them independently.

I did not get the question completely.
Are you saying that scale up/down of a statefulset is undesirable, or an overall scale up/down of a federated statefulset whose pods are distributed across clusters undesirable?
In either case I think its wise to have a scale function atleast available to the users rather then not having this function at all.

mstump · 2017-03-31T03:39:30Z

contributors/design-proposals/federated-statefulsets.md

+
+(1) only allow the use of node-local persistent (bad),
+
+(2) disallow migration of replicas between clusters if they use non-node-local volumes (simple, but perhaps overly restrictive)


This is the preferred method. Don't attempt to move storage across clusters. Most distributed databases have their own notion of identity, consistency and replication. This is an app specific concern, don't overcomplicate k8s by trying to manage replication for them.

agree.. ! That is what this doc also proposes.

mstump · 2017-03-31T03:48:10Z

contributors/design-proposals/federated-statefulsets.md

+
+### Replica distribution (across federated clusters)
+
+The proposed default behaviour of the federation controller, when a statefulset creation request is sent to federation API server, would be to partition the statefulset replicas and create a statefulset into each of the clusters with the reduced replica number (after partitioning), quite similar to the behaviour of the replicaset or daemonset controllers of the k8s federation.


I don't understand why this is desirable or needed. Let each statefulset be managed independently, but create network links between the federated sets.

The point is that the federation user needs to see a consolidated view of the statefulset. Also, the current k8s federation design philosophy is that the federation ideally appears as another k8s cluster, where some users might not even know that they are talking to federation and not a normal k8s cluster.

irfanurrehman · 2017-04-13T07:20:24Z

cc @kubernetes/sig-apps-misc @kubernetes/sig-apps-feature-requests

More line breaks and minor nits.

ghost · 2017-04-17T23:30:43Z

@irfanurrehman I think that the design looks good overall. It would be useful to add a few non-trivial, concrete examples to illustrate both useful applications, and some of the explicit limitations. Off the top of my head, these examples might include:

A federated etcd statefulset of 3 replicas across >2 clusters (using maxReplicas=1, PVC's) - this creates a global quorum able to withstand the temporary failure of any one cluster, as well as the permanent failure of any node in any cluster. Replicas would move around in each cluster, but not be moved between clusters. You already allude to this, but you could explain in some detail how the replicas would discover each other, and recover from failure.
A federated cassandra statefulset of 100 replicas across, say 5 clusters. If they use local disks, replicas could move between clusters.
A federated etcd statefulset of 30 replicas across 10 clusters (using minreplicas=3, maxreplicas=3, PVC's) - this creates a local quorum in each of the 10 clusters. The potential global quorum is ignored in this example. If a cluster goes down, the quorum in that cluster is dead, but the quora in all the other clusters remain up. You could place a federated service over the top to illustrate that that service remains available, even though one of the clusters is dead.

soltysh · 2017-04-19T15:34:57Z

contributors/design-proposals/federated-statefulsets.md

+in each of the clusters.
+It will further partition the total number of replicas and create statefulsets with partitioned 
+replica numbers into at least 1 or more clusters.
+The noteworthy point is the proposal that federated stateful controller would additionally modify 


With this you might hit name limit problem. When any of the names is close to max and you concatenate them you might exceed allowed limit. It is an important problem that should be described in this proposal, imho.

Thanks for the specific suggestion, and apologies for the delayed response.
I have added as solution for this implementable as an admission controller, but given that its a difficult to hit problem, it might not be a necessary implementation in the first phase solution. Hope that is ok?

soltysh · 2017-04-19T15:42:34Z

#503 seems also very relevant for this proposal.

irfanurrehman · 2017-06-05T15:38:53Z

#503 seems also very relevant for this proposal.

yes, It is indeed relevant. My suggestion however for now is to treat the federated statefulset update design separate (or a later extension) from the federated statefulset feature. The same way its happening for this feature in local k8s.

…8-themes touches up SIG Node 1.8 release themes

k8s-github-robot · 2017-10-17T20:36:48Z

This PR hasn't been active in 109 days. Closing this PR. Please reopen if you would like to work towards merging this change, if/when the PR is ready for the next round of review.

cc @irfanurrehman @quinton-hoole

You can add 'keep-open' label to prevent this from happening again, or add a comment to keep it open another 90 days

jwaldrip · 2017-11-14T23:00:41Z

Why was this closed?

kow3ns · 2017-11-16T22:05:57Z

It auto closed because it was not merged and has not received attention. I don't think this proposal is complete enough to implement.

irfanurrehman · 2017-11-17T07:31:52Z

@jwaldrip, as @kow3ns pointed out correctly, it was autoclosed. Meanwhile sig multicluster relooked at its priority list a little while back, and we refocused our efforts to moving the federation code out of core first and then moving the existing features to GA as the top priorities, rather then implementing advanced features like this now. This does not mean, we are not going to implement this; however it will take some time until we get back to this (probably a quarter). Reopening this, as this will be implemented anyhow.

irfanurrehman · 2017-11-17T07:32:13Z

@kubernetes/sig-multicluster-feature-requests

fejta-bot · 2018-02-15T08:31:02Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

fejta-bot · 2018-03-17T09:17:50Z

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

dionysius · 2018-04-05T14:32:28Z

Don't know if I'm allowed to respond here. I am waiting for this feature, so I don't want this issue to be closed (because of lifecycle/rotten), otherwise it gets completely forgotten...

irfanurrehman · 2018-04-05T17:23:08Z

/remove-lifecycle rotten

ghost · 2018-04-06T20:25:22Z

@dionysius This is not dead. It's being pursued in https:://github.com/kubernetes/federation . Don't worry :-)

fejta-bot · 2018-07-05T21:08:12Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

fejta-bot · 2018-08-04T21:56:28Z

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

fejta-bot · 2018-09-03T22:42:45Z

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

k8s-ci-robot · 2018-09-03T22:42:51Z

@fejta-bot: Closing this PR.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

* Update VOTERS.md * Update steering/elections/2020/VOTERS.md Co-authored-by: craigbox <craigbox@google.com> Co-authored-by: craigbox <craigbox@google.com>

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Mar 7, 2017

smarterclayton reviewed Mar 8, 2017

View reviewed changes

ghost self-assigned this Mar 9, 2017

smarterclayton reviewed Mar 23, 2017

View reviewed changes

chrislovecnm reviewed Mar 25, 2017

View reviewed changes

[Federation] Federated statefulsets design proposal

c5b7c9e

irfanurrehman force-pushed the federated-statefulsets-design branch from f3a9511 to c5b7c9e Compare March 27, 2017 18:18

chrislovecnm reviewed Mar 30, 2017

View reviewed changes

mstump reviewed Mar 31, 2017

View reviewed changes

[Federation] Federated statefulsets design proposal updates

685aa56

More line breaks and minor nits.

soltysh suggested changes Apr 19, 2017

View reviewed changes

dhilipkumars mentioned this pull request Apr 22, 2017

Federated StatefulSets feature kubernetes/enhancements#259

Closed

irfanurrehman force-pushed the federated-statefulsets-design branch from e2143d3 to 20168e3 Compare June 5, 2017 15:30

[Federation][Statefulset] Handle review comments

3a39431

irfanurrehman force-pushed the federated-statefulsets-design branch from 20168e3 to 3a39431 Compare June 6, 2017 14:41

irfanurrehman force-pushed the federated-statefulsets-design branch 2 times, most recently from fc3e785 to 215def2 Compare June 15, 2017 12:19

shyamjvs pushed a commit to shyamjvs/community that referenced this pull request Sep 22, 2017

Merge pull request kubernetes#437 from calebamiles/update-sig-node-1.…

f0ce01c

…8-themes touches up SIG Node 1.8 release themes

k8s-github-robot closed this Oct 17, 2017

tusharnt mentioned this pull request Oct 27, 2017

VCP needs to support federated statefulsets feature vmware-archive/kubernetes-archived#357

Open

irfanurrehman reopened this Nov 17, 2017

k8s-ci-robot added sig/multicluster Categorizes an issue or PR as relevant to SIG Multicluster. kind/feature Categorizes issue or PR as related to a new feature. labels Nov 17, 2017

k8s-github-robot added the kind/design Categorizes issue or PR as related to design. label Feb 6, 2018

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Feb 15, 2018

k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Mar 17, 2018

k8s-ci-robot removed the lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. label Apr 5, 2018

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jul 5, 2018

k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Aug 4, 2018

k8s-ci-robot closed this Sep 3, 2018


		1 – An unique, consistent and discoverable identity for each replica/instance across the federated clusters.

		2 – Predictability on the number of instances and sequentialization of pod creation (for example instance 1 creation starts only when instance 0 is up and running).


		If we consider the use cases listed above, the main design requirements can roughly be listed as:

		1 – An unique, consistent and discoverable identity for each replica/instance across the federated clusters.

		1 – A stateful app, for the reasons of high availability, wants the stateful pods distributed in different clusters, such that the set can withstand cluster failures. This represents an app with one single global quorum.

		2 – A stateful app, wants replicas distributed in multiple clusters, such that it can form multiple smaller sized local clusters using only local replicas.


		## Storage volumes

		There are two ways a federated statefulset can be assigned a persistent storage.


		elaborated by @quinton-hoole

		Strictly speaking, migration between clusters in the same zone is quite feasible.


		2 – Predictability on the number of instances and sequentialization of pod creation (for example instance 1 creation starts only when instance 0 is up and running).

		3 – The ability to scale, across clusters in some deterministic fashion.


		_3 – What happens if a cluster dies_

		Nothing, statefulset would need to run with lesser replicas


		We propose using alternative 1 listed here, as it fits broader scheme of things and is more consistent with the user expectation of being able to query all needed resources from the federation control plane and is less confusing to use at the same time.

		# Conclusion


		It is known, that as of now, the same persistent volume, even if can be used in a different cluster, k8s directly does not provide an API, yet, which can aid the same. In the absence of no direct way of quick migration of the storage data from one zone to another, or from one cloud provider to another if the case be, the proposal is to disallow migration of pods across clusters.

		## Scale up/Scale down


		(1) only allow the use of node-local persistent (bad),

		(2) disallow migration of replicas between clusters if they use non-node-local volumes (simple, but perhaps overly restrictive)


		### Replica distribution (across federated clusters)

		The proposed default behaviour of the federation controller, when a statefulset creation request is sent to federation API server, would be to partition the statefulset replicas and create a statefulset into each of the clusters with the reduced replica number (after partitioning), quite similar to the behaviour of the replicaset or daemonset controllers of the k8s federation.

Uh oh!

[Federation] Federated statefulsets design proposal #437

[Federation] Federated statefulsets design proposal #437

Uh oh!

Conversation

irfanurrehman commented Mar 7, 2017 • edited by k8s-github-robot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

liggitt commented Mar 7, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

smarterclayton commented Mar 8, 2017

Uh oh!

irfanurrehman commented Mar 8, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dhilipkumars commented Mar 8, 2017

Uh oh!

madhusudancs commented Mar 22, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

smarterclayton Mar 23, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chrislovecnm left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

irfanurrehman Mar 27, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

irfanurrehman commented Mar 27, 2017

Uh oh!

irfanurrehman commented Mar 27, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chrislovecnm left a comment

Choose a reason for hiding this comment

irfanurrehman commented Mar 7, 2017 •

edited by k8s-github-robot

Loading

irfanurrehman commented Mar 8, 2017 •

edited

Loading

madhusudancs commented Mar 22, 2017 •

edited

Loading

smarterclayton Mar 23, 2017 •

edited

Loading

irfanurrehman Mar 27, 2017 •

edited

Loading

irfanurrehman commented Mar 27, 2017 •

edited

Loading