waofed

Optimizes workload allocation and loadbalancing on KubeFed.

• Overview
• Getting Started
  • Installation
  • Deploy a WAOFedConfig resource
  • Schedule Optimization
    • Scheduling settings (RSPOptimizer)
    • Deploy a FederatedDeployment resource
    • See the generated ReplicaSchedulingPreference resource
  • Access Optimization
    • Loadbalancing settings (SLPOptimizer)
    • Deploy FederatedServices resources
    • See the generated ServiceLoadbalancingPreference resource
  • Uninstallation
• Developing
  • Prerequisites
  • Run development clusters with kind
  • Run tests on kind clusters
• License

Overview

WAOFed optimizes workload allocation and loadbalancing on KubeFed with the following components:

• RSPOptimizer: Optimizes FederatedDeployment weights across clusters by generating ReplicaSchedulingPreference using the specified method. KubeFed handles the actual scheduling according to ReplicaSchedulingPreference.
• SLPOptimizer: Optimizes FederatedService loadbalancing weights across clusters by generating ServiceLoadbalancingPreference using the specified method. A supported controller is required to handle the actual loadbalancing according to ServiceLoadbalancingPreference.

Getting Started

Supported Kubernetes versions: 1.19 or higher

💡 Mainly tested with 1.25, may work with the same versions that KubeFed supports (but may require some efforts).

Supported KubeFed APIs:

• FederatedDeployment [types.kubefed.io/v1beta1]
• ReplicaSchedulingPreference [scheduling.kubefed.io/v1alpha1]
• KubeFedCluster [core.kubefed.io/v1beta1]

New APIs Provided by WAOFed:

• WAOFedConfig [waofed.bitmedia.co.jp/v1beta1]
• ServiceLoadBalancingPreference [waofed.bitmedia.co.jp/v1beta1]

Installation

Make sure you have cert-manager deployed on the cluster where KubeFed control plane is deployed, as it is used to generate webhook certificates.

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.10.0/cert-manager.yaml

⚠️ You may have to wait a second for cert-manager to be ready.

Deploy the Operator with the following command. It creates waofed-system namespace and deploys CRDs, controllers and other resources.

kubectl apply -f https://github.com/Nedopro2022/waofed/releases/download/v0.4.0/waofed.yaml

Deploy a WAOFedConfig resource

WAOFedConfig is a resource for configuring WAOFed. Deploy it with the name default to the cluster where KubeFed control plane is deployed.

spec.kubefedNamespace specifies the namespace from which WAOFed gets member clusters.

apiVersion: waofed.bitmedia.co.jp/v1beta1
kind: WAOFedConfig
metadata:
  name: default # must be default
spec:
  kubefedNamespace: "kube-federation-system"
  scheduling:
    selector:
      hasAnnotation: waofed.bitmedia.co.jp/scheduling
    optimizer:
      method: "rr"
  loadbalancing:
    selector:
      hasAnnotation: waofed.bitmedia.co.jp/loadbalancing
    optimizer:
      method: "rr"

Schedule Optimization

Scheduling settings (RSPOptimizer)

RSPOptimizer watches the creation of FederatedDeployment resources and generates ReplicaSchedulingPreference resources with optimized workload allocation determined by the specified method.

Supported methods: rr (Round-robin, for testing purposes), wao (WAO-Estimator is required)

spec.scheduling.selector specifies the conditions for the FederatedDeployment resources that KubeFed watches.

💡 You can enable RSPOptimizer by default by setting spec.scheduling.selector.any to true.

   scheduling:
     selector:
-      hasAnnotation: waofed.bitmedia.co.jp/scheduling
+      any: true

Deploy a FederatedDeployment resource

💡 Ensure the namespace is federated by a FederatedNamespace resource before deploying FederatedDeployment resources.

apiVersion: types.kubefed.io/v1beta1
kind: FederatedNamespace
metadata:
  name: default
  namespace: default
spec:
  placement:
    clusterSelector: {}

When a FederatedDeployment with the annotation specified in WAOFedConfig is deployed, RSPOptimizer will detect the resource and generate an ReplicaSchedulingPreference.

apiVersion: types.kubefed.io/v1beta1
kind: FederatedDeployment
metadata:
  name: fdeploy-sample
  namespace: default
  annotations:
    waofed.bitmedia.co.jp/scheduling: ""
spec:
  template:
    metadata:
      labels:
        app: nginx
    spec:
      replicas: 9
      selector:
        matchLabels:
          app: nginx
      template:
        metadata:
          labels:
            app: nginx
        spec:
          containers:
            - image: nginx:1.23.2
              name: nginx
  placement:
    clusterSelector: {}

See the generated ReplicaSchedulingPreference resource

💡 You can see the resources with the following commands, and see the details by adding -oyaml.

$ kubectl get fdeploy
NAME             AGE
fdeploy-sample   12s

$ kubectl get rsp
NAME             AGE
fdeploy-sample   12s

The generated ReplicaSchedulingPreference has an owner reference indicating that it is controlled by the FederatedDeployment so that it will be deleted by GC when the FederatedDeployment is deleted.

spec.clusters includes all clusters specified in FederatedDeployment spec.placement (RSPOptimizer parses the selector and retrives clusters), and spec.clusters[name].weight is optimized by the method specified in WAOFedConfig. This sample uses rr so all clusters have a weight of 1.

apiVersion: scheduling.kubefed.io/v1alpha1
kind: ReplicaSchedulingPreference
metadata:
  name: fdeploy-sample
  namespace: default
  ownerReferences:
  - apiVersion: types.kubefed.io/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: FederatedDeployment
    name: fdeploy-sample
    ...
spec:
  clusters:
    cluster1:
      weight: 1
    cluster2:
      weight: 1
    cluster3:
      weight: 1
  intersectWithClusterSelector: true
  rebalance: true
  targetKind: FederatedDeployment
  totalReplicas: 9
...

💡 Since spec.intersectWithClusterSelector is set to true, the generated ReplicaSchedulingPreference does not overwrite anything in the FederatedDeployment, allowing RSPOptimizer to watch the FederatedDeployment easily.

⚠️ Edge cases not covered:

placement.clusters has 0 items

KubeFed ignores spec.placement.clusterSelector if spec.placement.clusters is provided, so no clusters will be selected for the following case (docs). However, RSPOptimizer currently does not recognize whether a list is nil (null) or has 0 items ([]), so it regards spec.placement.clusters as "not provided" and uses spec.placement.clusterSelector for scheduling.

spec:
  placement:
    clusters: []
    clusterSelector:
      matchExpressions:
        - { key: mylabel, operator: Exists }

Access Optimization

Loadbalancing settings (SLPOptimizer)

⚠️ NOTE: A supported controller is required to handle the actual loadbalancing according to ServiceLoadbalancingPreference.

SLPOptimizer watches the creation of FederatedService resources and generates ServiceLoadbalancingPreference resources with optimized workload allocation determined by the specified method.

Supported methods: rr (Round-robin, for testing purposes)

spec.loadbalancing.selector specifies the conditions for the FederatedService resources that KubeFed watches.

💡 You can enable SLPOptimizer by default by setting spec.loadbalancing.selector.any to true.

   loadbalancing:
     selector:
-      hasAnnotation: waofed.bitmedia.co.jp/loadbalancing
+      any: true

Deploy FederatedServices resources

💡 Ensure the namespace is federated by a FederatedNamespace resource before deploying FederatedService resources.

apiVersion: types.kubefed.io/v1beta1
kind: FederatedNamespace
metadata:
  name: default
  namespace: default
spec:
  placement:
    clusterSelector: {}

When a FederatedService with the annotation specified in WAOFedConfig is deployed, SLPOptimizer will detect the resource and generate an ServiceLoadbalancingPreference.

apiVersion: types.kubefed.io/v1beta1
kind: FederatedService
metadata:
  name: fsvc-sample
  namespace: default
  annotations:
    waofed.bitmedia.co.jp/loadbalancing: ""
spec:
  template:
    spec:
      selector:
        app: nginx
      ports:
        - name: http
          port: 80
  placement:
    clusterSelector: {}

See the generated ServiceLoadbalancingPreference resource

💡 You can see the resources with the following commands, and see the details by adding -oyaml.

$ kubectl get fsvc
NAME             AGE
fsvc-sample   12s

$ kubectl get slp
NAME             AGE
fsvc-sample   12s

The generated ServiceLoadbalancingPreference has an owner reference indicating that it is controlled by the FederatedService so that it will be deleted by GC when the FederatedService is deleted.

spec.clusters includes all clusters specified in FederatedService spec.placement (SLPOptimizer parses the selector and retrives clusters), and spec.clusters[name].weight is optimized by the method specified in WAOFedConfig. This sample uses rr so all clusters have a weight of 1.

apiVersion: waofed.bitmedia.co.jp/v1beta1
kind: ServiceLoadbalancingPreference
metadata:
  name: fsvc-sample
  namespace: default
  ownerReferences:
  - apiVersion: types.kubefed.io/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: FederatedService
    name: fsvc-sample
    ...
spec:
  clusters:
    cluster1:
      weight: 1
    cluster2:
      weight: 1
    cluster3:
      weight: 1
...

⚠️ Edge cases not covered:

placement.clusters has 0 items Same as RSPOptimizer

Uninstallation

Delete the Operator and resources with the following command.

kubectl delete -f https://github.com/Nedopro2022/waofed/releases/download/v0.4.0/waofed.yaml

Developing

This Operator uses Kubebuilder (v3.8.0), so we basically follow the Kubebuilder way. See the Kubebuilder Documentation for details.

Prerequisites

Make sure you have the following tools installed:

• Git
• Make
• Go
• Docker

Run development clusters with kind

The script creates K8s clusters kind-waofed-[0123], deploys KubeFed control plane on kind-waofed-0 and let the remaining clusters join as member clusters.

./hack/dev-kind-reset-clusters.sh
./hack/dev-kind-deploy.sh

⚠️ NOTE: Currently it is needed to re-create the clusters on every reboot as the script does not set static IPs to Docker containers.

Run tests on kind clusters

The script creates K8s clusters kind-waofed-test-[01], deploys KubeFed control plane on kind-waofed-test-0, let all clusters join as member clusters and runs integration tests.

# setup test clusters
./test/reset-clusters.sh

# basic tests (tests both scheduling and loadbalancing with "rr" method)
./test/run-basic-tests.sh

# test spec.scheduling.optimizer.method="wao"
./test/rspoptimizer-wao-setup.sh
./test/rspoptimizer-wao-run-tests.sh

License

Copyright 2022 Bitmedia Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.