Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dynamic Kubelet Configuration Proposal #29459

Closed
wants to merge 4 commits into from

Conversation

@mtaufen
Copy link
Contributor

commented Jul 22, 2016

This change is Reviewable

@derekwaynecarr

This comment has been minimized.

Copy link
Member

commented Jul 22, 2016

cc @kubernetes/sig-node

@vishh vishh assigned vishh and derekwaynecarr and unassigned brendandburns Jul 23, 2016

## Motivation

The Kubelet is currently configured via command-line flags. This is painful for a number of reasons:
- It makes it difficult to change the way Kubelets are configured in a running cluster, without tearing the cluster down.

This comment has been minimized.

Copy link
@vishh

vishh Jul 23, 2016

Member

Well tearing down is not necessary. It depends on the config management system that is being used.
I'd say that it requires changing the kubelet startup configuration which is often tedious.

The Kubelet is currently configured via command-line flags. This is painful for a number of reasons:
- It makes it difficult to change the way Kubelets are configured in a running cluster, without tearing the cluster down.
- It makes it difficult to manage different Kubelet configurations for different nodes, e.g. if you want to canary a new config.
- The current lack of versioned Kubelet configuration means that any time we change Kubelet flags, we risk breaking someone's setup.

This comment has been minimized.

Copy link
@vishh

vishh Jul 23, 2016

Member

There is also the use case of bootstrapping master nodes, before api-server is available.


- Add a well-defined, versioned configmap object for Kubelet configuration.
- Add/Remove/Update global and per-node Kubelet config on the fly via the API server.
- Provide a Kubelet endpoint to get the config currently in use by a given Kubelet.

This comment has been minimized.

Copy link
@vishh

vishh Jul 23, 2016

Member

nit I think you meant a REST API


## Motivation

The Kubelet is currently configured via command-line flags. This is painful for a number of reasons:

This comment has been minimized.

Copy link
@timothysc
@davidopp

This comment has been minimized.

Copy link
Member

commented Jul 25, 2016

@mtaufen mtaufen force-pushed the mtaufen:dynamic-kubelet-proposal branch from e7608dd to a548857 Aug 1, 2016


- Add a well-defined, versioned configmap object for Kubelet configuration.
- Add/Remove/Update global and per-node Kubelet config on the fly via the API server.
- Provide a REST API to get the config currently in use by a given Kubelet.

This comment has been minimized.

Copy link
@vishh

vishh Aug 2, 2016

Member

Why not place the configuration and the source of the configuration in NodeStatus?
I imagine a workflow where users can pull a new config object from a separate git repo, and have a controller apply the new config to nodes in a controlled manner, by waiting for a set of nodes to come up with the new configuration and be healthy, before proceeding with applying the new config to other nodes.
Bad config changes can be reverted via git based workflow on top of this functionality.

- `api-default`: The defaults provided for omitted/nil fields by the API server.
- `on-disk-config`: `KubeletConfiguration` provided on disk, used as a fallback when the Kubelet cannot cannot contact the API server.
- `minimal-config`: Conceptually, what the Kubelet needs to start up and contact the API server. This probably consists of the location of the API server and credentials to access the API server, and might be provided via flags or via `on-disk-config`.
- `global-kubelet-configuration`: the `KubeletConfiguration` configmap set via the API server which is shared by all Kubelets in your cluster except those which have a `<node-name>-kubelet-configuration`.

This comment has been minimized.

Copy link
@vishh

vishh Aug 2, 2016

Member

IIUC, the global config was introduced for the sake of convenience, but it is also too risky. Imagine a bad config rollout blowing up your entire cluster, which results in most of service outages today.
I'd recommend getting rid of the global configuration and support just per-node config.

This comment has been minimized.

Copy link
@aaronlevy

aaronlevy Aug 3, 2016

Member

Requiring <node-name>-kubelet-configuration could be pretty painful from a "rolling-update" perspective (must create a config object for every node in your cluster). But I agree the global-configuration is scary too (especially if a global config change results in all kubelets restarting).

Instead of being fixed to node name, what about allowing a reference to an arbitrary config object? Then a rolling update just requires configA and configB (or master vs worker).

Few ways this could be accomplished, but we were planning on using annotations for this in the interim, such that a utility app is responsible for pulling / checkpointing the configMap object which is referenced in the current node annotation.

This comment has been minimized.

Copy link
@mtaufen

mtaufen Aug 3, 2016

Author Contributor

Could you accomplish your goal by labeling nodes something like class: master/class: worker, defining master-kubelet-configuration and worker-kubelet-configuration configmaps, and having a controller deal with the rollout of the <node-name>-kubelet-configurations when you change either of those two?

This comment has been minimized.

Copy link
@aaronlevy

aaronlevy Aug 3, 2016

Member

Yup, this would also work.

This comment has been minimized.

Copy link
@derekwaynecarr

derekwaynecarr Aug 4, 2016

Member

is there any special interaction with the config map and kubelets whose name is sourced from a cloud provider?

This comment has been minimized.

Copy link
@derekwaynecarr

derekwaynecarr Aug 4, 2016

Member

@ingvagabund - do the existing ansible scripts have a similar global / node-specific kubelet configuration?

This comment has been minimized.

Copy link
@ingvagabund

ingvagabund Aug 5, 2016

Contributor

All nodes are configured from exactly the same config file [1]. Though, only a small portion of all kubelet arguments is used in the file. Basically, ansible mimics distribution rpm installation process with advantage of parametric templates to set network plugins, cluster dns, etc. Though, parameters are the same for all nodes. There is no concept of node-specific kubelet configuration so far.

[1] https://github.com/kubernetes/contrib/blob/master/ansible/roles/node/templates/kubelet.j2

This comment has been minimized.

Copy link
@mtaufen

mtaufen Aug 5, 2016

Author Contributor

@derekwaynecarr In the current example implementation, the Kubelet will try to use the Cloud Provider's implementation of CurrentNodeName to get the <node-name> portion of <node-name>-kubelet-configuration. If there isn't a kcfg.Cloud available, it will default to using the hostname (which is a uname -n unless a HostnameOverride was provided) for <node-name> when it looks for the configmap.

### Well-defined configmap type for Kubelet configuration

- There is an alpha version of the `KubeletConfiguration` type: external in `pkg/componentconfig/v1alpha1/types.go`, internal in `pkg/componentconfig/types.go`.
- We likely want to group related fields of the `KubeletConfiguration` into substructures. @timstclair and I did some brainstorming and came up with some rough potential categories:

This comment has been minimized.

Copy link
@vishh

vishh Aug 2, 2016

Member

This is great and very helpful in general, but can we punt on this for the MVP. We can keep this feature in alpha and the config versions as alpha until the grouping is done.

- Having the ability to pass config in the same format on-disk as via the API server will make it easier to bootstrap master nodes before the API server is available.


## Goals of this design:

This comment has been minimized.

Copy link
@vishh

vishh Aug 2, 2016

Member

Mention the deprecation policy. When can users expect a multi-release deprecation policy for this feature?

This comment has been minimized.

Copy link
@mtaufen

mtaufen Aug 3, 2016

Author Contributor

I'm thinking that as long as this is in alpha, we deprecate old stuff every release. Once it hits beta, we can start supporting the two most recent releases. How does that sound?


Initially, the Kubelet will store the most recent configmap it got from the API server internally, and compare that to new values it gets from the API server. It will restart when these configmaps differ in order to apply the new config.

This is a rudimentary solution, and @aaronlevy brought up some good reasons to checkpoint the config, and store it externally from the Kubelet:

This comment has been minimized.

Copy link
@vishh

vishh Aug 2, 2016

Member

Checkpointing is necessary for letting the kubelet run even with the api server is temporarily unreachable.

The fact that new configs might be incompatible with old configs is a problem that we should not try to optimize. In-compatible config changes might require a node reboot and that needs to be communicated to users as part of the release.
Adding checkpointing into this mix is too confusing in IMHO.


### Storing config locally (checkpointing)

Initially, the Kubelet will store the most recent configmap it got from the API server internally in memory, and compare that to new values it gets from the API server. It will restart when these configmaps differ in order to apply the new config.

This comment has been minimized.

Copy link
@aaronlevy

aaronlevy Aug 3, 2016

Member

So does "restart" mean applying the new config to the kubelet object without actually restarting the process?

This comment has been minimized.

Copy link
@mtaufen

mtaufen Aug 3, 2016

Author Contributor

We will kill the Kubelet process when we see new config and rely on the process manager to restart it.

This comment has been minimized.

Copy link
@aaronlevy

aaronlevy Aug 3, 2016

Member

Where is the new kubelet process sourcing the new config from if it is only stored internally in memory?

This comment has been minimized.

Copy link
@aaronlevy

aaronlevy Aug 3, 2016

Member

Sorry, I was misunderstanding the process here. So the kubelet will just see a diff of configs, throw it out, restart, pull down the new config and apply it early-ish in its initialization.

This comment has been minimized.

Copy link
@mtaufen

mtaufen Aug 3, 2016

Author Contributor

Yeah. Exactly. The pull and apply happens relatively early in its initialization.

This comment has been minimized.

Copy link
@ingvagabund

ingvagabund Aug 5, 2016

Contributor

./hack/local-up-cluster.sh has no process manager. Once you kill kubelet, it never gets restarted. Can we assume existence of the manager in every environment? Kubelet service vs. containerized kubelet vs. anything else.

This comment has been minimized.

Copy link
@mtaufen

mtaufen Aug 5, 2016

Author Contributor

This is a very good point, and something that I have not tested universally. The example implementation is currently using a modified ./hack/local-up-cluster.sh to make sure the Kubelet gets restarted.

@mtaufen

This comment has been minimized.

Copy link
Contributor Author

commented Aug 4, 2016

Tracking an example implementation of some basic functionality here: #30090.

mtaufen added a commit to mtaufen/kubernetes that referenced this pull request Aug 4, 2016
## Motivation

The Kubelet is currently configured via command-line flags. This is painful for a number of reasons:
- It makes it difficult to change the way Kubelets are configured in a running cluster, because it is often tedious to change the Kubelet startup configuration.

This comment has been minimized.

Copy link
@derekwaynecarr

derekwaynecarr Aug 4, 2016

Member

without a configuration management solution.

This comment has been minimized.

Copy link
@ingvagabund

ingvagabund Aug 5, 2016

Contributor

Ansible or salt can do that for you. You will have problems when you update kubelet configuration in apiserver which gets reflected in a kubelet vs. re-running ansible/running salt. With ansible, it is quote easy to skip configuration tasks (skip-tags flag) once the cluster is deployed and the ansible is re-run. So the new kubelet configuration does not get touched (still, need to keep that in mind). With salt, not sure if you can do anything here unless you stop the salt, change salt (e.g. skip configuration) and start salt.

Saying, some folks can be running ansible/salt already. When you start configuring the kubelet dynamically by default, you can get into trouble.

This comment has been minimized.

Copy link
@mtaufen

mtaufen Aug 12, 2016

Author Contributor

That's a good point. We should warn people about this in the documentation for the feature. With the current design, it would be pretty easy to avoid dynamically configuring the Kubelet -- just choose not to post kubelet-<node-name> configmaps in kube-system.

mtaufen added a commit to mtaufen/kubernetes that referenced this pull request Jun 19, 2017
Dynamic Kubelet Configuration
Alpha implementation of the Dynamic Kubelet Configuration feature.
See the proposal doc in kubernetes#29459.
mtaufen added a commit to mtaufen/kubernetes that referenced this pull request Jun 22, 2017
Dynamic Kubelet Configuration
Alpha implementation of the Dynamic Kubelet Configuration feature.
See the proposal doc in kubernetes#29459.
mtaufen added a commit to mtaufen/kubernetes that referenced this pull request Jul 6, 2017
Dynamic Kubelet Configuration
Alpha implementation of the Dynamic Kubelet Configuration feature.
See the proposal doc in kubernetes#29459.
mtaufen added a commit to mtaufen/kubernetes that referenced this pull request Jul 10, 2017
Dynamic Kubelet Configuration
Alpha implementation of the Dynamic Kubelet Configuration feature.
See the proposal doc in kubernetes#29459.
mtaufen added a commit to mtaufen/kubernetes that referenced this pull request Jul 10, 2017
Dynamic Kubelet Configuration
Alpha implementation of the Dynamic Kubelet Configuration feature.
See the proposal doc in kubernetes#29459.
mtaufen added a commit to mtaufen/kubernetes that referenced this pull request Jul 10, 2017
Dynamic Kubelet Configuration
Alpha implementation of the Dynamic Kubelet Configuration feature.
See the proposal doc in kubernetes#29459.
mtaufen added a commit to mtaufen/kubernetes that referenced this pull request Jul 10, 2017
Dynamic Kubelet Configuration
Alpha implementation of the Dynamic Kubelet Configuration feature.
See the proposal doc in kubernetes#29459.
mtaufen added a commit to mtaufen/kubernetes that referenced this pull request Jul 10, 2017
Dynamic Kubelet Configuration
Alpha implementation of the Dynamic Kubelet Configuration feature.
See the proposal doc in kubernetes#29459.
mtaufen added a commit to mtaufen/kubernetes that referenced this pull request Jul 10, 2017
Dynamic Kubelet Configuration
Alpha implementation of the Dynamic Kubelet Configuration feature.
See the proposal doc in kubernetes#29459.
mtaufen added a commit to mtaufen/kubernetes that referenced this pull request Jul 10, 2017
Dynamic Kubelet Configuration
Alpha implementation of the Dynamic Kubelet Configuration feature.
See the proposal doc in kubernetes#29459.
mtaufen added a commit to mtaufen/kubernetes that referenced this pull request Jul 19, 2017
Dynamic Kubelet Configuration
Alpha implementation of the Dynamic Kubelet Configuration feature.
See the proposal doc in kubernetes#29459.
mtaufen added a commit to mtaufen/kubernetes that referenced this pull request Jul 20, 2017
Dynamic Kubelet Configuration
Alpha implementation of the Dynamic Kubelet Configuration feature.
See the proposal doc in kubernetes#29459.
mtaufen added a commit to mtaufen/kubernetes that referenced this pull request Jul 20, 2017
Dynamic Kubelet Configuration
Alpha implementation of the Dynamic Kubelet Configuration feature.
See the proposal doc in kubernetes#29459.
mtaufen added a commit to mtaufen/kubernetes that referenced this pull request Jul 20, 2017
Dynamic Kubelet Configuration
Alpha implementation of the Dynamic Kubelet Configuration feature.
See the proposal doc in kubernetes#29459.
mtaufen added a commit to mtaufen/kubernetes that referenced this pull request Jul 21, 2017
Dynamic Kubelet Configuration
Alpha implementation of the Dynamic Kubelet Configuration feature.
See the proposal doc in kubernetes#29459.
mtaufen added a commit to mtaufen/kubernetes that referenced this pull request Jul 24, 2017
Dynamic Kubelet Configuration
Alpha implementation of the Dynamic Kubelet Configuration feature.
See the proposal doc in kubernetes#29459.
mtaufen added a commit to mtaufen/kubernetes that referenced this pull request Jul 24, 2017
Dynamic Kubelet Configuration
Alpha implementation of the Dynamic Kubelet Configuration feature.
See the proposal doc in kubernetes#29459.
mtaufen added a commit to mtaufen/kubernetes that referenced this pull request Jul 24, 2017
Dynamic Kubelet Configuration
Alpha implementation of the Dynamic Kubelet Configuration feature.
See the proposal doc in kubernetes#29459.
mtaufen added a commit to mtaufen/kubernetes that referenced this pull request Jul 25, 2017
Dynamic Kubelet Configuration
Alpha implementation of the Dynamic Kubelet Configuration feature.
See the proposal doc in kubernetes#29459.
mtaufen added a commit to mtaufen/kubernetes that referenced this pull request Jul 26, 2017
Dynamic Kubelet Configuration
Alpha implementation of the Dynamic Kubelet Configuration feature.
See the proposal doc in kubernetes#29459.
mtaufen added a commit to mtaufen/kubernetes that referenced this pull request Aug 1, 2017
Dynamic Kubelet Configuration
Alpha implementation of the Dynamic Kubelet Configuration feature.
See the proposal doc in kubernetes#29459.
mtaufen added a commit to mtaufen/kubernetes that referenced this pull request Aug 3, 2017
Dynamic Kubelet Configuration
Alpha implementation of the Dynamic Kubelet Configuration feature.
See the proposal doc in kubernetes#29459.
mtaufen added a commit to mtaufen/kubernetes that referenced this pull request Aug 8, 2017
Dynamic Kubelet Configuration
Alpha implementation of the Dynamic Kubelet Configuration feature.
See the proposal doc in kubernetes#29459.
mtaufen added a commit to mtaufen/kubernetes that referenced this pull request Aug 8, 2017
Dynamic Kubelet Configuration
Alpha implementation of the Dynamic Kubelet Configuration feature.
See the proposal doc in kubernetes#29459.
mtaufen added a commit to mtaufen/kubernetes that referenced this pull request Aug 8, 2017
Dynamic Kubelet Configuration
Alpha implementation of the Dynamic Kubelet Configuration feature.
See the proposal doc in kubernetes#29459.
mtaufen added a commit to mtaufen/kubernetes that referenced this pull request Aug 8, 2017
Dynamic Kubelet Configuration
Alpha implementation of the Dynamic Kubelet Configuration feature.
See the proposal doc in kubernetes#29459.
k8s-github-robot pushed a commit that referenced this pull request Aug 9, 2017
Kubernetes Submit Queue
Merge pull request #46254 from mtaufen/dkcfg
Automatic merge from submit-queue (batch tested with PRs 50016, 49583, 49930, 46254, 50337)

Alpha Dynamic Kubelet Configuration

Feature: kubernetes/enhancements#281

This proposal contains the alpha implementation of the Dynamic Kubelet Configuration feature proposed in ~#29459~ [community/contributors/design-proposals/dynamic-kubelet-configuration.md](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/dynamic-kubelet-configuration.md). 

Please note:
- ~The proposal doc is not yet up to date with this implementation, there are some subtle differences and some more significant ones. I will update the proposal doc to match by tomorrow afternoon.~
- ~This obviously needs more tests. I plan to write several O(soon). Since it's alpha and feature-gated, I'm decoupling this review from the review of the tests.~ I've beefed up the unit tests, though there is still plenty of testing to be done.
- ~I'm temporarily holding off on updating the generated docs, api specs, etc, for the sake of my reviewers 😄~ these files now live in a separate commit; the first commit is the one to review.

/cc @dchen1107 @vishh @bgrant0607 @thockin @derekwaynecarr 

```release-note
Adds (alpha feature) the ability to dynamically configure Kubelets by enabling the DynamicKubeletConfig feature gate, posting a ConfigMap to the API server, and setting the spec.configSource field on Node objects. See the proposal at https://github.com/kubernetes/community/blob/master/contributors/design-proposals/dynamic-kubelet-configuration.md for details.
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.