Improve node interface

[ddysher]

Kubernetes’s minion interface is obscure and brittle.

Externally, we don’t have a formal document on how minion rest api works. For example, in #1315, user is confused about kubecfg create minion. In #1995, user is unware of the assumption that to create a minion, kubelet should be started first. This is probably a bad assumption, but the worst part is the reported error, which gives user no clue of what’s happening.

Internally, I don’t know if there is a clear direction of how we want to manage minions, especially:

1. kubernetes admin interface
2. minion health check and monitoring

For 1, I think we need to give kubernetes admin a consistent view of how minion management is handled in kubernetes. For example, we now have a minion controller that sychronizes machines from cloudprovider, but we still expose the API for admin to create a minion. Mixing the two options are confusing and error prone. Also, to create a minion is only to create a representation of a minion in kubernetes, we are unable to really “create” (provision) the minion for user.

For 2, we currently haven’t defined any minion state, nor do we define a strategy on how kubernetes reacts on each state. A related issus #1923. A better monitoring and reporting mechanism is needed to make kubernetes even better.

Instead of proposing any abstract idea, I’m thinking of some concrete stags that we can step in.

• Create a minion-controller with two functionalities: synchronization and health check. To do this, we need to:
  • Move the health check functionality from api-server to minion-controller.
  • Define minion state. To keep it simple, we can just use RUNNING, VANISHED for now. In the long term, we can provide more states and health check policies.
• Make the synchronization part configurable. If user prefers managing worker nodes themselves, Kubernetes cluster can be created without the synchronization process. However, if synchronization is enabled, creating via user interface (command line, UI or their own plugins) is disabled, or report properly (via event if it’s ready and approprite for such use case).
• Allows user to easily see the status of his/her machines, something like

$ kubectl get minions
Minion Name                         Minion State             Utilization         Etc
kubernetes-1                          RUNNING                  30%
kubernetes-2                          RUNNING                  80%
kubernetes-3                          VANISHED                 0%

• Enable master server to provision a minion. I don’t know how this works yet, but a rough procedure like installing salt & k8s config, deploying binaries, registering through API server, etc, might work.
• Provide better documentation.

[jbeda]

Agree that good APIs and interfaces here are needed. Except we should s/minion/node/ as per #1111 :)

Anyone want to sign up to drive this? @ddysher are you offering to make this happen?

[ddysher]

@jbeda Yeah, I've done some work with node controller, and find some problems along the way. I'm hoping to fix at least some of the problems.

[ddysher]

I may also take the pains to do the renaming. Rename the titile to show my resolution. :)

[smarterclayton]

If you rename please do it in the context of #1519 - there's quite a bit of preparation work in the client, apiserver, etcd, etc to handle it cleanly.

[ddysher]

@smarterclayton Acknowledged

[eparis]

You have so many good thoughts in there!

I like where this is going, especially in light of all the discussion in #1315. Especially here

---

"For example, we now have a minion controller that sychronizes machines from cloudprovider, but we still expose the API for admin to create a minion. Mixing the two options are confusing and error prone."

I agree that a first step would be to just disable the kubectl add minions (proposed) interface if there is a cloudprovider/node controller plugin which is doing the sync. Maybe someday such operations could be worked into the 'regex' mechanism GCE uses when syncing the minion list. But for today, just banning it is a good start.

---

"Also, to create a minion is only to create a representation of a minion in kubernetes, we are unable to really “create” (provision) the minion for user."

Agreed, this is a big thing. It's the whole concept I don't think we have a good understanding of that I talked about in #2003. You point out all the same problems. How should we represent resources like minions which we can't (today) create and solve ourselves? I think we have the same problems with stroage and with load balancers and external ips... In my mind, just making the admin create the representation is fine. Making the sync loop create the representation is fine.

You say: "Enable master server to provision a minion. I don’t know how this works yet, but a rough procedure like installing salt & k8s config, deploying binaries, registering through API server, etc, might work." Maybe someday we can have a "sync loop" that can magically creates new minions, that might be great, but we shouldn't try for that yet...

[thockin]

A lot of work has gone into nodes/minions - is this now obsolete?

[bgrant0607]

Yes.