Cluster Federation: task list

[nikhiljindal]

Documenting the list of tasks for ubernetes for 1.3 based on various discussions:

APIServer:

• Add code for federated-apiserver ( duplicate kube-apiserver to federated-apiserver #23509)
• Add federation/v1alpha1 group version and cluster resource ( Federation apiobject cluster #23847)
• Add service resource to federated-apiserver (add service object without proxy #25232)
• Add an e2e test for federation apiserver (@colhom e2e tests for federated-apiserver #25125)

Client:

• Generate auto generated clients (to be used by controllers) (Federation client for cluster #24117)
• kubectl: Add support for cluster objects (kubectl get clusters) (Federation kubectl for clusters #24016)
• kubectl: Add federated-apiserver auth details to kubeconfig so that kubectl can talk to federated-apiserver.
• kubectl: Verify if any change is required for kubectl describe service (not required)
• hack/test-cmd.sh tests

Controller:

• Bring up ubernetes controller manager ( Implement the cluster-controller of kubernetes federation #23430)
• Write cluster controller: syncs state of registered kubernetes clusters.(Implement the cluster-controller of kubernetes federation #23430)
• Service controller: delegate service creation to underlying clusters (Ubernetes Service Controller #23848, WIP: ubernetes federated service controller. #24009, Implement the cluster-controller of kubernetes federation #23430 )
• Service controller: Create load balancing apparatus on top of resulting cluster services.
• e2e tests

Tying together:

• Add a script to bring up the control plane (brings up federation-apiserver, controller managers and atleast one underlying kubernetes cluster) (e2e tests for federated-apiserver #25125)
• Docs and example showing how it works

Future tasks (post 1.3):

• Add replicaset resource to federated-apiserver ( Add extensions/replicaset to federation-apiserver #24764)
• Write RS and subRS controllers/scheduler (Discussion for Ubernetes scheduler flow #23269)
• HA: Etcd quorum
• HA: Multiple ubernetes api servers and controller managers (one in each cluster)

@kubernetes/sig-cluster-federation

[ghost]

LGTM @nikhiljindal
Some minor comments:

1. Service controller comprises two major parts (see docs/design/federated-service.md for details):
   1. Delegating service creation to underlying Kubernetes clusters.
   2. Creating load balancing apparatus across the top of the resulting Kubernetes services.
2. I think that we can leave the HA items until all the other stuff is working.

[huangyuqi]

It is a great list @nikhiljindal .
Excepting HA items,

1. Bring up an ubernetes api server with its own etcd
2. Add ubernetes API objects to it (cluster)

In PR #21190, pushed by @jianhuiz

1.Bring up ubernetes controller manager
1.Write RS and subRS controllers (#23269)
1.Write Service controller

WIP PR is #23430 pushed by me.

At present, I am working on the controller manager, cluster controller, subrs controller , rs controller.
But, I have a doubt that shall we need to split cluster, subrs, rs into different controllers or not. Now, i have put these into a single controller called cluster controller.

BTW, we have missed a federation scheduler task implemented by @mfanjie . #23269 . Please add in the task list.

[ghost]

@huangyuqi FYI I have started working on the Service Controller in parallel in #23848

[huangyuqi]

@alfred-huangjian is also working on service controller, would you like to exchange with @quinton-hoole ?

[madhusudancs]

@huangyuqi @mfanjie @alfred-huangjian @jianhuiz It looks like you are all doing a lot of work and I am not entirely sure if that is getting enough code-review attention. Also, given that Ubernetes is a high priority for 1.3 and 1.3's aggressive timeline, I think we will have to collaborate and better distribute the work among all of us to get this over the finish line.

So I have a request for all of you. Could you all please make a list of things you are working on, sort of sub-tasks, along with PRs attached to it in the order they should be reviewed? Even WIP PRs are fine and we can do that here in this issue or in a Google Doc and attach that doc here. Correct me if I am wrong, but IMHO it is better to claim smaller tasks and have that finished (reviewed and merged) before moving on to other things so that people can claim pending tasks as they finish their tasks.

@huangyuqi you did some of what I am proposing in #23653 (comment), but it is still not clear to me in what order they should be reviewed.

cc @nikhiljindal @quinton-hoole

[huangyuqi]

@madhusudancs
About cluster controller what i am dealing with, I have split it into some sub-task:

• duplicate kube-controller-manager to uber-controller-manager
• Add rs-controller
• Add subrs-controller
• Add cluster-controller
• Reuse codes of kubectl to create sub-resource in kubernetes cluster

I will create a issue to track.

[madhusudancs]

@huangyuqi Thanks! Also, is there a way to split your current PR into these tasks so that the completed PRs can be independently reviewed and merged as you work in parallel on WIP PRs?

[huangyuqi]

Thanks for your suggestion @madhusudancs
For the first sub-task duplicate kube-controller-manager to uber-controller-manager , i will take a independent PR today.
And other sub-tasks will be put in one PR with different commits.

[mfanjie]

#23269 was discussing federation scheduling and then there are some discussions on replicaset and sub replicaset. I am taking scheduling part and rs/subrs are covered by @jianhuiz. It's better if the task can be split to two.
and the wip of federation scheduler is #24038 (edited)

[ghost]

@alfred-huangjian Have you started any work on the service controller design or implementation yet? If so, please submit work in progress PR's so that we don't duplicate work. See comment from @huangyuqi above. I'm working on it too, in (#23848, #24009)

[mfanjie]

when testing federation scheduler, I got a problem of listing and watching clusters and replicasets, I opened a issue #24656, and so far I suppose there is defect on decoder.
Please kindly advise.

[mfanjie]

A quick update, #24656 is closed as after I merge the latest codec PR from master, it works properly. No blocker anymore at this moment.

[nikhiljindal]

Updated the task list based on recent discussions:

APIServer:

• Add code for federated-apiserver
• Add federation/v1alpha1 group version and cluster resource
• Add replicaset resource to federated-apiserver
• Add an e2e test for federation apiserver

Client:

• Generate auto generated clients (to be used by controllers)
• kubectl: Add support for cluster objects
• kubectl: Add federated-apiserver auth details to kubeconfig so that kubectl can talk to federated-apiserver.
• hack/test-cmd.sh tests

Controller:

• Bring up ubernetes controller manager
• Write cluster controller: syncs state of registered kubernetes clusters.
• Write RS and subRS controllers (Discussion for Ubernetes scheduler flow #23269)
• Service controller: delegate service creation to underlying clusters (Ubernetes Service Controller #23848, WIP: ubernetes federated service controller. #24009, Implement the cluster-controller of kubernetes federation #23430 )
• Service controller: Create load balancing apparatus on top of resulting cluster services.
• e2e tests

Tying together:

• Add a script to bring up the control plane (brings up federation-apiserver, controller managers and atleast one underlying kubernetes cluster)
• Docs and example showing how it works

Future tasks (post 1.3):

• HA: Etcd quorum
• HA: Multiple ubernetes api servers and controller managers (one in each cluster)

[nikhiljindal]

Remaining 1.3 tasks:

• Create secrets for all clusters (@nikhiljindal Updating federation up script to create secrets with federation-apiserver and k8s apiservers kubeconfigs #26914)
• Add e2e tests for cluster controller (@nikhiljindal, Updating federation up scripts to work in non e2e setup #27260)
• Add e2e tests for service controller (@madhusudancs, Implement first set of federated service e2e tests. #26636)
• Add federation e2e to tests that merge bot runs (include federation e2e in merge bot testing #26723 @quinton-hoole)
• Get e2e tests to pass on GCE ([k8s.io] [Feature:Federation] Federated Services DNS non-local federated service should be able to discover a non-local federated service {Kubernetes e2e suite} #27739, [k8s.io] Federation apiserver [Feature:Federation] Cluster objects should be created and deleted successfully {Kubernetes e2e suite} #27738)
• Verify e2e tests on GKE (@colhom)
• Verify e2e tests on AWS (@colhom)
• Add a user-guide for admins explaining how to setup federation (@nikhiljindal, Adding a guide for admins to set up federation website#662)
• Add a guide for users explaining how they can use federation (@quinton-hoole, @kelseyhightower )
• Update the federation-{up, down} scripts so that admins can also run them to create the federation control plane (right now they are e2e specific, for ex: the namespace they create is federation-e2e) (@nikhiljindal, Updating federation up scripts to work in non e2e setup #27260)
• Make sure federation control plane images are shipped as part of release.sh (Pushing public images of federation control plane on every release #27382)

Everyone please feel free to add anything that I missed and also feel free to pick up tasks from here.
@kubernetes/sig-cluster-federation @colhom @mfanjie @huangyuqi @jianhuiz

[davidopp]

cc/ @mml (see #23653 (comment) )

[colhom]

As per offline discussion w/ @nikhiljindal I will be taking care of the following items:

* Verify e2e tests on GKE
* Verify e2e tests on AWS

[nikhiljindal]

Remaining 1.3 tasks (things we couldnt get to in time for 1.3 cut, but should fix now):

• Get our current service e2es to pass on jenkins
• Add federation e2e to tests that merge bot runs (include federation e2e in merge bot testing #26723 @quinton-hoole)
• Add more test cases to our service e2e (@mml)
• Add documentation for setting up federation control plane (@nikhiljindal, Adding a guide for admins to set up federation website#662)
• Add documentation for users to use federation (@quinton-hoole, @kelseyhightower )
• Add API ref docs (http://kubernetes.io/docs/reference/) for federation group version (Generate API reference docs for federation API #30541, @nikhiljindal)
• Set up federation e2es on GKE (@colhom)
• Setup federation e2e on AWS (@colhom)

[nikhiljindal]

@colhom Were you able to make any progress on running federation e2es on GKE and AWS?

[colhom]

@nikhiljindal last I checked, AWS was functional. GKE should work, though when I tested there were some what I believe to be unrelated issues with the gcloud account I was using.

[ghost]

@colhom To be clear, as discussed, you're going to send in some PR's to configure our standard Jenkins to run federation e2es on GKE and AWS, right?

[colhom]

@quinton-hoole I'm now officially back from vacation- and yes and yes!

[goltermann]

@quinton-hoole @colhom what remains to finish this issue off?

[nikhiljindal]

There are 2 remaining tasks, both of which have individual issues filed.
Closing this one.