policy: setting single node defaults that don't enhance kubernetes

[dustymabe]

With the recent discussions about oomd and swapOnZram we've been touching on a larger topic of what makes sense for us to have as a default for certain settings on Fedora CoreOS. If a particular Fedora Change or feature (which we can choose to adopt or not (at a cost)) is not compatible with Kubernetes but would be beneficial to the single node FCOS use case, what should we do?

This topic started to bubble up in last week's meeting and in #859 (comment). It continued in today's meeting where we decided to create this issue to continue the discussion.

There seems to be a few camps of thought:

• We don't actually ship K8s and it has to be deployed on top. We should be OK to add in defaults for the single node use case.

  • Since K8s distros will need to configure the system anyway to get K8s up and running (using Ignition on first boot) they can easily disable/re-configure the few defaults that aren't palatable.
    • Good documentation for kubernetes distro integration would be useful here.
  • This would be somewhat of a policy change from our mission statement of "optimized for Kubernetes but also great without it".

• Can we somehow ship the defaults for single node, but have them dynamically disabled for Kubernetes

  • dynamically via systemd overrides (some condition checks for /etc/kubernetes or kubelet.service or something

So far I don't hear anyone rejecting the notion completely that we need to change something (i.e. stating that we should only make changes if they make sense for k8s). We will continue this discussion here in the ticket and in the coming meetings.

[jlebon]

We discussed this in the community meeting today.

No clear outcome yet, though there was some agreement that we should focus the discussion foremost on whether defaulting to single node makes sense. And then we can talk about how to make it easy for k8s configuration. I think the consensus on that was "yes", though we should verify that in the next meeting.

[bgilbert]

To clarify, I think we're really talking about setting defaults for the non-k8s case. Those defaults will generally be agnostic to whether an installation is single-node or non-k8s clustered.

For one way to handle dynamic feature enablement for k8s, see #892.

[dustymabe]

We discussed this in the meeting today.

2021-07-21 13:01:25  dustymabe  #agreed After an appropriate period of notice we implement the
                                single node optimized defaults for "new installs". We add documentation
                                for k8s distributors and users. If we automigrate existing nodes in the
                                future, we'll revisit the feature flags proposal first.

[bgilbert]

After the notice period expires, we should update the mission statement.