Skip to content

Files

Latest commit

 

History

History
83 lines (64 loc) · 3.63 KB

node-labels.md

File metadata and controls

83 lines (64 loc) · 3.63 KB

Add support for node labels

Introduction

Problem: Within an EKS-A cluster, the user might want to run specific workloads only on certain nodes. A great way to do this is by assigning labels to the desired nodes. However, there is currently no way for users to add labels to their nodes during cluster creation. Users may assign labels to nodes post cluster creation, but those labels will not persist during processes like upgrading the cluster (which rolls new machines hence new nodes).

Goals and Objectives

As an EKS Anywhere user:

  • I want to have the ability to run specific workloads on specific nodes
  • I want to have the ability to assign one or more labels to a node during cluster creation
  • I want my node labels to persist during cluster operations such as upgrades

Overview of Solution

With this feature, a user can specify node labels in the node configuration. This info will be fetched and added to the kubeletExtraArgs in the kubelet configuration.

Solution Details

The user can specify the labels as key-value pairs under the node configurations.

Example in cluster config:

apiVersion: anywhere.eks.amazonaws.com/v1alpha1
kind: Cluster
metadata:
  name: cluster-name
spec:
   ...
   workerNodeGroupConfigurations:
   - count: 1
     labels:
       label1: foo
       label2: bar
     machineGroupRef:
       kind: VSphereMachineConfig
       name: my-cluster-machines
   ...

The labels field will be of type map[string]string to stay consistent with how labels are specified in the metadata for other k8s objects. However, kubeletExtraArgs expects a string of key-value pairs for the node-labels flag, so the above example would need to be converted to a string that looks like: "label1=foo,label2=bar". After converting that map to a string, we can proceed to add this information to kubeletExtraArgs.

Example:

kubeletExtraArgs:
  cloud-provider: external
  node-labels: label1=foo,label2=bar
  tls-cipher-suites: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256

After the extra args are applied and the cluster is created, the user will now be able to specify which nodes to run workloads on based on those labels. The user can also view the labels on their nodes using kubectl get nodes -A --show-labels.

Example output (labels applied to worker nodes only in this example):

NAME                      STATUS    ROLES                  AGE      LABELS
bob-kbf58                  Ready    control-plane,master   5m20s    beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=vsphere-vm.cpu-2.mem-8gb.os-ubuntu,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64
bob-md-0-56f765c98-pvwj4   Ready    <none>                 4m19s    beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=vsphere-vm.cpu-2.mem-8gb.os-ubuntu,beta.kubernetes.io/os=linux,label1=foo,label2=bar,kubernetes.io/arch=amd64
bob-md-0-56f765c98-vp87t   Ready    <none>                 4m19s    beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=vsphere-vm.cpu-2.mem-8gb.os-ubuntu,beta.kubernetes.io/os=linux,label1=foo,label2=bar,kubernetes.io/arch=amd64
bob-s9shf                  Ready    control-plane,master   4m9s     beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=vsphere-vm.cpu-2.mem-8gb.os-ubuntu,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64

Things to note:

  • For vSphere Ubuntu, adding these labels to kubeletExtraArgs is sufficient. However, we have to configure additional settings to add this extra customization when using Bottlerocket. Example user data for setting up labels with Bottlerocket:
[settings.kubernetes.node-labels]
"label1" = "foo"
"label2" = "bar"