Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Convert all component command-line flags to versioned configuration #12245

Open
bgrant0607 opened this issue Aug 5, 2015 · 24 comments

Comments

@bgrant0607
Copy link
Member

commented Aug 5, 2015

Forked from #1627, #5472, #246, #12201, and other issues.

We configure Kubernetes components like apiserver and controller-manager with command-line flags currently. Every time we change these flags, we break everyone's turnup automation, at least for anything not in our repo: hosted products, other repos, blog posts, ...

OTOH, the scheduler and kubectl read configuration files. We should be treating all configuration similarly, as versioned APIs, with backward compatibility, deprecation policies, documentation, etc. How we distribute the configuration is a separate issue -- #1627.

cc @davidopp @thockin @dchen1107 @lavalamp

@bgrant0607

This comment has been minimized.

Copy link
Member Author

commented Aug 18, 2015

@bgrant0607

This comment has been minimized.

@bgrant0607

This comment has been minimized.

Copy link
Member Author

commented Aug 18, 2015

@stefwalter

This comment has been minimized.

Copy link
Contributor

commented Aug 20, 2015

@mikedanese I started yesterday to convert KubeletServer (which is poorly named) to use a JSON settings file. I'd be happy to follow the lead of your work done on the scheduler config, as it does cover a couple things I missed.

@mikedanese

This comment has been minimized.

Copy link
Member

commented Aug 20, 2015

I haven't done much with this yet. I've begun to add a component config api group in this commit:

a8b00ed

Is this a similar direction you are taking? Let's discuss how we want to redistribute the configuration work tomorrow.

@pmorie

This comment has been minimized.

Copy link
Member

commented Aug 20, 2015

@mikedanese @stefwalter Can you guys confirm that you intend to combine this with the mechanism described in #6477 ?

@mikedanese

This comment has been minimized.

Copy link
Member

commented Aug 20, 2015

@pmorie Yes it is. It is related but separate and can be done in parallel. The combination should close #1627.

@mikedanese

This comment has been minimized.

Copy link
Member

commented Aug 20, 2015

@stefwalter can you please point me to the branch where you are working on this?

@bgrant0607

This comment has been minimized.

Copy link
Member Author

commented Aug 20, 2015

All new configuration formats we create should go into new API groups #12951

@stefwalter

This comment has been minimized.

Copy link
Contributor

commented Aug 21, 2015

@mikedanese I stopped working on it once I saw your work. I did about half an hour ... before posting what I was doing ... https://github.com/stefwalter/kubernetes/tree/kubelet-settings

I'd be happy to help out with the tedious bits or wherever necessary.

@bgrant0607

This comment has been minimized.

Copy link
Member Author

commented Sep 9, 2015

Some additional motivation:

The current way in which we configure components via command-line flags is unnecessarily convoluted and hard to standardize, since some deployments run the components using containers, some use init, some use systemd, some use supervisord, etc.

Life of configuring a flag on GCE (if wrong, that's another indication of how convoluted it is; different for every cloud platform):

  1. Set an environment variable (e.g., ADMISSION_CONTROL) in a config-default.sh script that is invoked by some script invoked indirectly by kube-up.sh: https://github.com/kubernetes/kubernetes/blob/master/cluster/gce/config-default.sh
  2. Which is semi-manually translated to a YAML file and stuffed into VM metadata: https://github.com/kubernetes/kubernetes/blob/master/cluster/gce/debian/helper.sh
  3. Which is used to generate a Salt pillar on the master node: https://github.com/kubernetes/kubernetes/blob/master/cluster/gce/configure-vm.sh
  4. Which is used to template-substitute the apiserver's pod config: https://github.com/kubernetes/kubernetes/blob/master/cluster/saltbase/salt/kube-apiserver/kube-apiserver.manifest
@davidopp

This comment has been minimized.

Copy link
Member

commented Oct 6, 2015

Not for 1.1

@bogdando

This comment has been minimized.

Copy link

commented Jan 3, 2017

configure Kubernetes components like apiserver and controller-manager with command-line flags ...
is unnecessarily convoluted and hard to standardize, since some deployments run the components using containers, some use init, some use systemd, some use supervisord...

I strongly believe that truly immutable containers shall not only ship immutable binaries and dependencies bundled but as well behave always the same way, which is independent of the mounted
configuration files. I may be wrong and opinionated, but that's how I "feel" containers. And runtime flags do perfectly fit into that concept, unlike to config files.

With config files mounted RW into immutable containers, you always end up with risks to make those containers to behave unpredictable, which has nothing to real immutability. When deployed or mutated with bad configs, it becomes as well destructive if had been shipped with bad binaries or dependencies. An example failure mode to illustrate that:

  • By a mistake in the systemd unit file of a containerized kubelet service, let it be badly written teardown/init for post-stop/pre-start, create two running instances of the kubelet service with mounted config files in RW mode.
  • Both of the kublet service instances do checkpointing of its current state (see self-hosted kuberenetes's "checkpoint the updated config").
  • The RW mounted checkpointed state ends up in the corrupted file accessed by two instances of kubelet
  • ???
    (nothing good, but pure destruction is now delayed for the next kubelet service restart in order to take over the control plane to the self-hosted one).

Also, mutated config files must notify the related service on changes, this brings in extra moving parts, like classic config management tools and its agents etc. More things to fail.

Given that said, I kindly ask community to reconsider abandoning runtime flags to the versioned config files preference. Runtime flags make the given failure mode impossible and do not require notifications on the config changes, because of all options are self-contained in the service unit definition - just restart it to atomically mutate and apply. This brings none moving parts and none extra failure modes.

@mikedanese @bgrant0607 @jbeda

@bogdando

This comment has been minimized.

Copy link

commented Jan 3, 2017

And if one wants versioning, just please do versioning of the self-contained service units files containing all required runtime flags corresponding to the given version.

@bgrant0607

This comment has been minimized.

Copy link
Member Author

commented Jan 12, 2017

@bogdando Systemd unit files are configuration files in a less introspectable, less portable, less maintainable format. I don't think your argument is actually about flags vs configuration files at all. Read-only vs. read-write seems to be a fairly narrow/specific issue that could be easily addressed. Since this appears to be specifically about Kubelet, please raise your concerns on #29459 or #27980. As for checkpointing in Kubelet, that's inevitable due to availability and bootstrapping requirements. If you have specific suggestions for how to make checkpointing more robust and resilient, please comment on #489.

@bogdando

This comment has been minimized.

Copy link

commented Jan 13, 2017

@bgrant0607 thanks for the links, I will take a look. Note, the runtime flags vs config files bring less corner cases to handle when doing checkpointing, that was my main concern. Also, systemd unit files look like portable well enough as supported by all modern OS types, don't they? And maintainable with templating engines very well...

@kfox1111

This comment has been minimized.

Copy link

commented Mar 15, 2017

I'm not sure if this is relevant or not to the conversation, but if daemonsets are enhanced to support the linked feature, along with daemonset rolling upgrade support, maybe this will be useful:
#43112

@roberthbailey

This comment has been minimized.

Copy link
Member

commented Apr 21, 2017

@mikedanese is going to be driving the component configuration standardization for 1.7.

@fejta-bot

This comment has been minimized.

Copy link

commented Dec 23, 2017

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

@bgrant0607

This comment has been minimized.

Copy link
Member Author

commented Jan 22, 2018

/remove-lifecycle stale
/lifecycle frozen

@mikedanese mikedanese removed their assignment Jan 25, 2018

@mtaufen

This comment has been minimized.

Copy link
Contributor

commented Mar 28, 2018

@kfox1111

This comment has been minimized.

Copy link

commented Mar 28, 2018

doc is private

@mtaufen

This comment has been minimized.

Copy link
Contributor

commented Mar 29, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.