☂️ Introduce Gardener Node Agent (a.k.a. replace cloud-config-downloader)

[majst01]

How to categorize this issue?

/area quality robustness
/kind enhancement epic

What would you like to be added:

Replace the cloud-config-download/executor which are written in the bash programming language with a binary written in go which is a kubernetes controller based on controller-runtime.

This was implemented up to a working implementation during the Gardener Hackathon 2023 in Leverkusen

https://github.com/gardener-community/hackathon

Why is this needed:

With the new Architecture we gain a lot, let's describe the most important gains here.

Developer Productivity

Because we all develop in go day by day, writing business logic in bash is difficult, hard to maintain, almost impossible to test. Getting rid of almost all bash scripts which are currently in use for this very important part of the cluster creation process will enhance the speed of adding new features and removing bugs.

Speed

Until now, the cloud-config-downloader runs in a loop every 60sec to check if something changed on the shoot which requires modifications on the worker node. This produces a lot of unneeded traffic on the api-server and wastes time, it will sometimes take up to 60sec until a desired modification is started on the worker node.
By using the controller-runtime we can watch for the node, theOSC in the secret, and the shoot-access-token in the secret. If any of these object changed, and only then, the required action will take effect immediately.
This will speed up operations and will reduce the load on the api-server of the shoot dramatically.

Scalability

Actually the cloud-config-downloader add a random wait time before restarting the kubelet in case the kubelet was updated or a configuration change was made to it. This is required to reduce the load on the API server and the traffic on the internet uplink. It also reduces the overall downtime of the services in the cluster because every kubelet restart takes a node for several seconds into NotReady state which eventually interrupts service availability.

Decision was made to keep the existing jitter mechanism which calculates the kubelet-download-and-restart-delay-seconds on the controller itself.

Correctness

The configuration of the cloud-config-downloader is actually done by placing a file for every configuration item on the disk on the worker node. This was done because parsing the content of a single file and using this as a value in bash reduces to something like VALUE=$(cat /the/path/to/the/file). Simple but lacks validation, type safety and whatnot.
With the gardener-node-agent we introduce a new API which is then stored in the gardener-node-agent secret and stored on disc in a single yaml file for comparison with the previous known state. This brings all benefits of type safe configuration.
Because actual and previous configuration are compared, removed files and units are also removed and stopped on the worker if removed from the OSC.

Availability

Previously the cloud-config-downloader simply restarted the systemd-units on every change to the OSC, regardless which of the services changed. The gardener-node-agent first checks which systemd-unit was changed, and will only restart these. This will remove unneeded kubelet restarts.

Goals

• create a new shoot which only uses the new gardener-node-agent
• update existing workers from cloud-config-downloader to gardener-node-agent
• same behavior when applying the OSC as before.

Tasks (basic functionality)

• Initial skaffolding and introduction of new component
  • 🕵️ Introduce lib for gardener-node-agent #8249
  • [node-agent] Introduce cmd/gardener-node-agent/app.go with manager.Manager initialization #8627
• Business Logic
  • [node-agent] Introduce token Secret controller #8629
  • [node-agent] Introduce Node controller #8632
  • [node-agent] Introduce OperatingSystemConfig controller #8683
  • [node-agent] Refactor /pkg/node-agent/registry  #8678
  • [node-agent] Extend OperatingSystemConfig API to with unit files and imageRef file type #8707
    • [node-agent] controlplane mutator webhook also searches for files in units.files of OSC #8760
    • [node-agent] Refactor files handling in OperatingSystemConfig API #8814
    • Original OperatingSystemConfig components always specify filePaths in their units #8827
    • Deprecated OS common package ignores files with image ref #8825
    • [node-agent] Merge filePaths when merging units #8838
  • [node-agent] Do not try fetching node name on start-up #8826
  • [node-agent] Adapt token controller to fetch also other tokens (valitail) #8839
  • [node-agent] Add Lease controller for reporting health/liveness (similar to the Lease the kubelet writes) #8767
    • [node-agent] Restrict cache for Leases to kube-system namespace #8893
    • [node-agent] Restrict Lease cache when Node has already been registered #8901
  • [node-agent] Use temporary directory in /var/lib/gardener-node-agent/tmp #8894
  • [node-agent] Check for unit file existence before disabling+stopping deleted units #8898
  • [node-agent] dbus interface should not block and wait for result channel forever #8907
  • [node-agent] Stop waiting for systemd command results after 10s #8919
  • [node-agent] Disable and stop gardener-node-init.service when gardener-node-agent is active #9096
• Bootstrapping gardener-node-agent
  • [node-agent] Introduce gardener-node-init script and unit #8726
• Seed / Shoot Controllers Adaptation
  • [node-agent] Adapt extension operating system config components according to changes in OperatingSystemConfig API #8756
  • [node-agent] Add logging configuration for gardener-node-agent unit #8836
  • [node-agent] Integrate gardener-node-agent into gardenlet's Shoot controller #8847
  • Adapt gardenlet health checks to consider gardener-node-agent's Lease object for the EveryNodeReady condition #9073
• Adapt OS Extensions (check if they manipulate cloud-config-downloader unit (sample) - if they do, then try to remove these or enable gardener-node-init instead): Prepare OperatingSystemConfig API for changes needed for gardener-node-agent #8647
  • provider-local: Prepare OperatingSystemConfig API for changes needed for gardener-node-agent #8647 and [provider-local] Adapt to changes needed for gardener-node-agent #8834
  • os-gardenlinux: Adapt to changes needed for gardener-node-agent gardener-extension-os-gardenlinux#130, released with v0.22.0
  • os-suse-chost: Adapt to changes needed for gardener-node-agent gardener-extension-os-suse-chost#112, released with v1.24.0
  • os-coreos: Adapt to changes needed for gardener-node-agent gardener-extension-os-coreos#80, released with v1.19.0
  • os-ubuntu: Adapt to changes needed for gardener-node-agent gardener-extension-os-ubuntu#99, released with v1.24.0
  • os-metal: Adapt to changes needed for gardener-node-agent metal-stack/os-metal-extension#33, released with v? (PR is ready, but no need to wait for it as part of this umbrella issue, so let's consider it done)
  • os-k3os (not needed since it was never used anywhere and is outdated anyways)
• Miscellaneous
  • Enhance Decoder to cope with plaintext #7993 (this was found missing during the implementation of the gardener-node-agent)
  • Documentation: 🕵️ Introduce lib for gardener-node-agent #8249
  • Extra build job to publish gardener-node-agent and potentially kubelet/kubectl as OCI Image, see below section (not needed since we decided to use native containerd functionality for extracting binaries from images, see [node-agent] Refactor /pkg/node-agent/registry  #8678)
  • Build gardener-node-agent container image in Prow (multi-arch!) (we decided to drop Prow builds, see Stop building gardener/gardener dev-images in prow ci-infra#981 and Remove .ci/resources.yaml when concourse dev-builds are enabled #8740 for details)
  • [node-agent] Use linear mapping to distribute OperatingSystemConfig reconciliation delays evenly #8885
  • [node-agent] Restrict Secret permissions and cache #8857
  • [node-agent] Refactor {kubelet,containerd}-monitor health check bash scripts to Golang #8786
  • Enable gardener-node-agent in provider-extensions setup #9048 (follow-up of Explicitly disable UseGardenerNodeAgent feature gate in local extensions setup #8905)
  • [node-agent] Create gardener Linux user when nodes are bootstrapping #9077
  • [node-agent] Log hostname in gardener-node-agent #9114
  • [node-agent] Introduce hostname-check controller #9280
• Graduation & Rollout
  • Introduce new alpha UseGardenerNodeAgent feature gate in gardenlet controlling whether gardener-node-{init,agent} should be used: Prepare OperatingSystemConfig API for changes needed for gardener-node-agent #8647
  • Promote UseGardenerNodeAgent feature gate to beta #9161
  • Promote UseGardenerNodeAgent feature gate to GA #9208
  • Remove nerdctl workaround for local OS image gardener-attic/machine-controller-manager-provider-local#37
  • Drop UseGardenerNodeAgent feature gate, cloud-config-{downloader,executor}, and oscommon package 🧹 #9477
  • Cleanup migration code in OS extensions (planned after v1.92 got released)
    • os-gardenlinux: Cleanup legacy code for cloud-config-downloader scenario gardener-extension-os-gardenlinux#161
    • os-suse-chost: Cleanup legacy code for cloud-config-downloader scenario gardener-extension-os-suse-chost#138
    • os-coreos: Cleanup legacy code for cloud-config-downloader scenario gardener-extension-os-coreos#100
    • os-ubuntu: Cleanup legacy code for cloud-config-downloader scenario gardener-extension-os-ubuntu#126
    • provider-aws: Cleanup handling for disabled UseGardenerNodeAgent feature gate gardener-extension-provider-aws#920

OCI Image Support

A decision was made to go ahead with the current approach by extracting the binaries (kubelet, gardener-node-agent) out of docker images as currently implemented. Down the way towards GA of gardener-node-agent, the OCI artifact support is implemented and right before GA, this support is merged. The existing docker images of all still supported kubernetes/kubelet versions need to be pushed as OCI artifact and of course the binary of the gardener-node-agent.

Update: Not needed since we decided to use native containerd functionality for extracting binaries from images, see #8678.

No modifications are made to cloud-config-downloader to support OCI artifacts.

[Gerrit91]

👍 for this stuff!

I would like to propose another future goal, which can be turned into an own issue after this one was resolved. The idea is to drop the OSC with -original suffix (purpose reconcile) and migrate its contents into the NodeAgentConfiguration. The -downloader OSC can remain as is.

This way, we can move on to a more concise interface for gardener-node-agent, which has several advantages:

• The new interface enhances the visibility and testability for the features that are currently wrapped implicitly inside the OSC's file and unit definitions, making them easier to parametrize and to implement
• Gardener and Extensions should not need to struggle with OS-/node-internals, only the gardener-node-agent should implement this kind of code
• Extensions that want to hook into node-specific configuration do not need to agree on conventions (an example might be the registry-extension, which wants to add mirror configuration for containerd and has to put it to a specific path in order not to conflict with other extensions that want to modify the containerd configuration as well).
• The node-agent can directly read its configuration from the secret instead of the described approach where the configuration is wrapped inside the OSC, which the OSC controller writes to the disk, which will then notify other controllers do apply possible changes.

Here is an example of how the configuration could be extended:

type NodeAgentConfiguration struct {
	metav1.TypeMeta

	// APIServer contains the connection configuration for the gardener-node-agent to
	// access the shoot api server.
	APIServer APIServer `json:"apiServer"`

	// ConfigurationSecretName defines the name of the secret in the shoot cluster, which contains
	// the configuration for the gardener-node-agent.
	ConfigurationSecretName string `json:"configSecretName"`
	// TokenSecretName defines the name of the secret in the shoot cluster, which contains
	// the projected shoot access token for the gardener-node-agent.
	TokenSecretName string `json:"tokenSecretName"`

	// Image is the docker image reference to the gardener-node-agent.
	Image string `json:"image"`
	// HyperkubeImage is the docker image reference to the hyperkube containing kubelet.
	HyperkubeImage string `json:"hyperkubeImage"`

	// KubernetesVersion contains the kubernetes version of the kubelet, used for annotating
	// the corresponding node resource with a kubernetes version annotation.
	KubernetesVersion string `json:"kubernetesVersion"`

	// ContainerdConfiguration configures the containerd runtime according to the given configuration.
	// +optional
	ContainerdConfiguration *ContainerdConfiguration `json:"containerdConfiguration,omitempty`

	// ValitailConfiguration configures valitail according to the given configuration.
	// +optional
	ValitailConfiguration *ValitailConfiguration `json:"valitailConfiguration,omitempty`
    
	// KubeletDataVolumeSize sets the data volume size of an unformatted disk on the worker node,
	// which is the be used for /var/lib on the worker.
	// +optional
	KubeletDataVolumeSize *int64 `json:"kubeletDataVolumeSize,omitempty"`
}

type ContainerdConfiguration struct {
	// RegistryMirrors a list of registry mirror to configure for containerd.
	RegistryMirrors []string `json:"registryMirrors,omitempty`
	// MonitorEnabled deploys a service to monitor the containerd service.
	MonitorEnabled bool `json:"monitorEnabled`
	// LogRotationPeriod is cron schedule for when logs of containerd a rotated.
	LogRotationPeriod string `json:"logRotationPeriod"`

	// ... fields have to be worked out in the future
}

type ValitailConfiguration struct {
	// ... fields have to be worked out in the future
}

Do you think this is feasible?

[vknabel]

As promised I rebased all currently known future changes from the hackathon following the current PR.
Here you can have a look on the additional changes. As I could not come up with an ideal order of upcoming PRs based on the controllers yet, I did not create isolated commits (although it would be easier to cherry-pick and rebase a controller for a pr).

Sadly the codebase has diverged and I couldn't fix all compile issues by myself.

• compared to the current pr: metal-stack/gardener@gardener-node-agent-lib...gardener-node-agent-complete
• gardener-node-agent-complete on metal-stack/gardener
• the original, non-rebased version is gardener-node-agent-2. But due to the unrelated history it displays all changes - even the original and unrevisited ones in the current initial PR: metal-stack/gardener@gardener-node-agent-lib...gardener-node-agent-2

I hope this serves well as a starting point

[rfranzke]

/assign @oliver-goetz @rfranzke

[oliver-goetz]

/area ipcei

[rfranzke]

All tasks have been completed.
/close

[gardener-prow]

@rfranzke: Closing this issue.

In response to this:

All tasks have been completed.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.