Skip to content
Cluster API Bootstrap Provider Kubeadm: giving your cluster API clusters a kubeadm kick
Go Shell Python Makefile Dockerfile
Branch: master
Clone or download
k8s-ci-robot Merge pull request #282 from chuckha/psa
📖 Inform users of move
Latest commit 2cdce45 Oct 18, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.github Add prow image building Sep 23, 2019
api/v1alpha2 Update generated code Oct 11, 2019
cloudinit remove command single quote Oct 17, 2019
config-capi Bump capi + crds Aug 28, 2019
config add well-known tolerations to released CABPK components Oct 4, 2019
controllers fix: ensure there's a token to refresh Oct 3, 2019
docs Add additional support for external etcd Oct 1, 2019
hack 🏃 Update module dependencies / controller-tools Sep 27, 2019
internal Separate certificate logic for joins Oct 2, 2019
kubeadm Disambiguate v1 import Aug 21, 2019
third_party/forked Update rerun-process-wrapper Aug 6, 2019
.gitignore Ignore sed backups Aug 5, 2019
.golangci.yml Invert enabled/disabled linter configuration. Sep 12, 2019
CONTRIBUTING.md Update PR template Aug 28, 2019
Dockerfile Update controller-runtime golang cluster-api Oct 11, 2019
Dockerfile.dev Update controller-runtime golang cluster-api Oct 11, 2019
LICENSE Initial commit Jun 24, 2019
Makefile 🏃 Align Makefile with other CAP* repos Sep 27, 2019
OWNERS Update approvers Aug 1, 2019
OWNERS_ALIASES Promote sataqiu to a maintainer Sep 25, 2019
PROJECT chore: add KubeadmConfigTemplate type Aug 19, 2019
README.md Inform users of move Oct 17, 2019
RELEASE.md Initial commit Jun 24, 2019
SECURITY_CONTACTS Update OWNERS, OWNERS_ALIASES and SECURITY_CONTACTS Jun 24, 2019
cloudbuild.yaml 🏃 Align Makefile with other CAP* repos Sep 27, 2019
code-of-conduct.md Initial commit Jun 24, 2019
go.mod Update controller-runtime golang cluster-api Oct 11, 2019
go.sum Update controller-runtime golang cluster-api Oct 11, 2019
main.go hard coding LeaderElectionID Oct 7, 2019

README.md

⚠️ Legacy repository. Current code has moved

This repository is only accepting bug fixes. All new development is being done in the cluster-api repository.

Cluster API bootstrap provider kubeadm

What is the Cluster API bootstrap provider kubeadm?

Cluster API bootstrap provider Kubeadm (CABPK) is a component of Cluster API that is responsible of generating a cloud-init script to turn a Machine into a Kubernetes Node; this implementation uses kubeadm for kubernetes bootstrap.

Resources

How to build, deploy and test CABPK

CABPK is built using kubebuilder. Please refer to the The Kubebuilder Book for build and deploy in a cluster.

CABPK is only part of a whole and does not function in isolation. It requires coordination with the core provider, Cluster API (CAPI), and an infrastructure provider; see the CAEP document for more information. For the remainder of this document we will be referring to cluster-api-provider-docker (CAPD) as our infrastructure provider.

A convenient way to set up infrastructure for testing is to use kind as the platform cluster to install the controllers and CRDs into. Then use a tool like tilt or skaffold to manage your dev environment. A minimal example of a Tiltfile looks like this:

allow_k8s_contexts('kubernetes-admin@kubernetes')

controllers = {
    'capi': {
        'path': '../cluster-api',
        'image': 'gcr.io/k8s-staging-cluster-api/cluster-api-controller:dev',
    },
    'cabpk': {
        'path': './',
        'image': 'gcr.io/k8s-staging-cluster-api/cluster-api-bootstrap-provider-kubeadm:dev',
    },
    'capd': {
        'path': '../cluster-api-provider-docker',
        'image': 'gcr.io/k8s-staging-cluster-api/cluster-api-provider-docker:dev',
    },
}

for name, controller in controllers.items():
    command = '''sed -i'' -e 's@image: .*@image: '"{}"'@' ./{}/config/default/manager_image_patch.yaml'''.format(controller['image'], controller['path'])
    local(command)

    k8s_yaml(local('kustomize build ' + controller['path'] + '/config/default'))

    docker_build(controller['image'], controller['path'])

See capi-dev for an example of a more complex developemt environment using tilt.

How does CABPK work?

Once your test environment is in place, create a Cluster object and its corresponding DockerCluster infrastructure object.

kind: DockerCluster
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha2
metadata:
  name: my-cluster-docker
---
kind: Cluster
apiVersion: cluster.x-k8s.io/v1alpha2
metadata:
  name: my-cluster
spec:
  infrastructureRef:
    kind: DockerCluster
    apiVersion: infrastructure.cluster.x-k8s.io/v1alpha2
    name: my-cluster-docker

Now you can start creating machines by defining a Machine, its corresponding DockerMachine object, and the KubeadmConfig bootstrap object.

kind: KubeadmConfig
apiVersion: bootstrap.cluster.x-k8s.io/v1alpha2
metadata:
  name: my-control-plane1-config
spec:
  initConfiguration:
    nodeRegistration:
      kubeletExtraArgs:
        eviction-hard: nodefs.available<0%,nodefs.inodesFree<0%,imagefs.available<0%
  clusterConfiguration:
    controllerManager:
      extraArgs:
        enable-hostpath-provisioner: "true"
---
kind: DockerMachine
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha2
metadata:
  name: my-control-plane1-docker
---
kind: Machine
apiVersion: cluster.x-k8s.io/v1alpha2
metadata:
  name: my-control-plane1
  labels:
    cluster.x-k8s.io/cluster-name: my-cluster
    cluster.x-k8s.io/control-plane: "true"
    set: controlplane
spec:
  bootstrap:
    configRef:
      kind: KubeadmConfig
      apiVersion: bootstrap.cluster.x-k8s.io/v1alpha2
      name: my-control-plane1-config
  infrastructureRef:
    kind: DockerMachine
    apiVersion: infrastructure.cluster.x-k8s.io/v1alpha2
    name: my-control-plane1-docker
  version: "v1.14.2"

CABPK's main responsibility is to convert a KubeadmConfig bootstrap object into a cloud-init script that is going to turn a Machine into a Kubernetes Node using kubeadm.

The cloud-init script will be saved into the KubeadmConfig.Status.BootstrapData and then the infrastructure provider (CAPD in this example) will pick up this value and proceed with the machine creation and the actual bootstrap.

KubeadmConfig objects

The KubeadmConfig object allows full control of Kubeadm init/join operations by exposing raw InitConfiguration, ClusterConfiguration and JoinConfiguration objects.

CABPK will fill in some values if they are left empty with sensible defaults:

KubeadmConfig field Default
clusterConfiguration.KubernetesVersion Machine.Spec.Version [1]
clusterConfiguration.clusterName Cluster.metadata.name
clusterConfiguration.controlPlaneEndpoint Cluster.status.apiEndpoints[0]
clusterConfiguration.networking.dnsDomain Cluster.spec.clusterNetwork.serviceDomain
clusterConfiguration.networking.serviceSubnet Cluster.spec.clusterNetwork.service.cidrBlocks[0]
clusterConfiguration.networking.podSubnet Cluster.spec.clusterNetwork.pods.cidrBlocks[0]
joinConfiguration.discovery a short lived BootstrapToken generated by CABPK

IMPORTANT! overriding above defaults could lead to broken Clusters.

[1] if both clusterConfiguration.KubernetesVersion and Machine.Spec.Version are empty, the latest Kubernetes version will be installed (as defined by the default kubeadm behavior).

Examples

Valid combinations of configuration objects are:

  • at least one of InitConfiguration and ClusterConfiguration for the first control plane node only
  • JoinConfiguration for worker nodes and additional control plane nodes

Bootstrap control plane node:

kind: KubeadmConfig
apiVersion: bootstrap.cluster.x-k8s.io/v1alpha2
metadata:
  name: my-control-plane1-config
spec:
  initConfiguration:
    nodeRegistration:
      kubeletExtraArgs:
        eviction-hard: nodefs.available<0%,nodefs.inodesFree<0%,imagefs.available<0%
  clusterConfiguration:
    controllerManager:
      extraArgs:
        enable-hostpath-provisioner: "true"

Additional control plane nodes:

kind: KubeadmConfig
apiVersion: bootstrap.cluster.x-k8s.io/v1alpha2
metadata:
  name: my-control-plane2-config
spec:
  joinConfiguration:
    nodeRegistration:
      kubeletExtraArgs:
        eviction-hard: nodefs.available<0%,nodefs.inodesFree<0%,imagefs.available<0%
    controlPlane: {}

worker nodes:

kind: KubeadmConfig
apiVersion: bootstrap.cluster.x-k8s.io/v1alpha2
metadata:
  name: my-worker1-config
spec:
  joinConfiguration:
    nodeRegistration:
      kubeletExtraArgs:
        eviction-hard: nodefs.available<0%,nodefs.inodesFree<0%,imagefs.available<0%

Bootstrap Orchestration

CABPK supports multiple control plane machines initing at the same time. The generation of cloud-init scripts of different machines is orchestrated in order to ensure a cluster bootstrap process that will be compliant with the correct Kubeadm init/join sequence. More in detail:

  1. cloud-config-data generation starts only after Cluster.InfrastructureReady flag is set to true.
  2. at this stage, cloud-config-data will be generated for the first control plane machine even if multiple control plane machines are ready (kubeadm init).
  3. after Cluster.metadata.Annotations[cluster.x-k8s.io/control-plane-ready] is set to true, the cloud-config-data for all the other machines are generated (kubeadm join/join —control-plane).

Certificate Management

The user can choose two approaches for certificate management:

  1. provide required certificate authorities (CAs) to use for kubeadm init/kubeadm join --control-plane; such CAs should be provided as a Secrets objects in the management cluster.
  2. let CABPK to generate the necessary Secrets objects with a self-signed certificate authority for kubeadm

TODO: Add more info about certificate secrets

Additional Features

The KubeadmConfig object supports customizing the content of the config-data:

  • KubeadmConfig.Files specifies additional files to be created on the machine
  • KubeadmConfig.PreKubeadmCommands specifies a list of commands to be executed before kubeadm init/join
  • KubeadmConfig.PostKubeadmCommands same as above, but after kubeadm init/join
  • KubeadmConfig.Users specifies a list of users to be created on the machine
  • KubeadmConfig.NTP specifies NPT settings for the machine

Versioning, Maintenance, and Compatibility

  • We follow Semantic Versioning (semver).
  • Cluster API bootstrap provider kubeadm versioning is syncronized with Cluster API.
  • The master branch is where development happens, this might include breaking changes.
  • The release-X branches contain stable, backward compatible code. A new release-X branch is created at every major (X) release.

Get involved!

  • Join the Cluster API discuss forum.

  • Join the sig-cluster-lifecycle Google Group for access to documents and calendars.

  • Join our Cluster API working group sessions

  • Provider implementer office hours

    • Weekly on Tuesdays @ 12:00 PT (Zoom) and Wednesdays @ 15:00 CET (Zoom)
    • Previous meetings: [ notes ]
  • Chat with us on Slack: #cluster-api

You can’t perform that action at this time.