Branch: master
Clone or download
Pull request Compare This branch is 21 commits ahead, 110 commits behind cloudfoundry-incubator:master.
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Type Name Latest commit message Commit time
Failed to load latest commit information.

Cloud Foundry Container Runtime

A BOSH release for Kubernetes. Formerly named kubo.


  • A BOSH Director configured with UAA, Credhub, and BOSH DNS. We recommend using BOSH Bootloader for this.

  • Latest kubo-deployment tarball

  • Accessing the master:

    • Single Master: Set up a DNS name pointing to your master's IP address
    • Multiple Masters: A TCP load balancer for your master nodes.
      • Use a TCP load balancer configured to connect to the master nodes on port 8443.
      • Add healthchecks using either a TCP dial or HTTP by looking for a 200 OK response from /healthz.
  • Cloud Config with

    • vm_types named minimal, small, and small-highmem (See cf-deployment for reference)
    • network named default
    • three availability zones azs named z1,z2,z3

    Note: the cloud-config properties can be customized by applying ops-files. See manifests/ops-files for some examples.

    If using loadbalancers then apply the vm_extension called cfcr-master-loadbalancer to the cloud-config to add the instances to your loadbalancers. See BOSH documentation for information on how to configure loadbalancers.

Hardware Requirements

Kubernetes uses etcd as its datastore. The official infrastructure requirements and example configurations for the etcd cluster can be found here.

Deploying CFCR

  1. Upload the latest Xenial stemcell to the director.

  2. Untar the kubo-deployment tarball and rename it kubo-deployment

  3. Deploy

    Option 1. Single Master
    cd kubo-deployment
    bosh deploy -d cfcr manifests/cfcr.yml \
      -o manifests/ops-files/misc/single-master.yml \
      -o manifests/ops-files/add-hostname-to-master-certificate.yml \
      -v api-hostname=[DNS-NAME]
    Option 2. Three Masters
    cd kubo-deployment
    bosh deploy -d cfcr manifests/cfcr.yml \
      -o manifests/ops-files/add-vm-extensions-to-master.yml \
      -o manifests/ops-files/add-hostname-to-master-certificate.yml \
      -v api-hostname=[LOADBALANCER-ADDRESS]

    Note: Loadbalancer address should be the external address (hostname or IP) of the loadbalancer you have configured.

    Check additional configurations, such as setting Kubernetes cloud provider, in docs.

  4. Add Kubernetes system components

    bosh -d cfcr run-errand apply-specs
  5. Run the following to confirm the cluster is operational

    bosh -d cfcr run-errand smoke-tests


CFCR clusters on BOSH Lite are intended for development. We run the deploy_cfcr_lite script to provision a cluster with the latest stemcell and master of kubo-release.

cd kubo-deployment

Accessing the CFCR Cluster with kubectl

  1. Login to the Credhub Server that stores the cluster's credentials:
    credhub login
  2. Find the director name by running
    bosh env
  3. Configure the kubeconfig for your kubectl client:
    cd kubo-deployment
    ./bin/set_kubeconfig <DIRECTOR_NAME>/cfcr https://[DNS-NAME-OR-LOADBALANCER-ADDRESS]:8443

Backup & Restore

We use BBR to perform backups and restores of the etcd node within a CFCR cluster. Our backup currently takes an etcd snapshot without interruptions to the cluster. However, for restore we take both the kube-apiserver and etcd offline to restore the cluster with the specified snapshot. Restore is a destructive operation that will completely overwrite any existing data on the cluster. For a closer look at the bbr scripts, check out:

To run the bbr cli against a CFCR cluster, follow the steps under "BOSH Deployment" on the BBR documentation page.

Note: this feature is currently available for single master deployments only. Work for BBR support on multi-master clusters is underway.


Follow the recommendations in etcd's documentation for monitoring etcd metrics.


Deployment scripts and docs

CFCR had a set of scripts, including deploy_bosh and deploy_k8s, that were the primary mechanism we supported to deploy BOSH and Kubernetes clusters. We no longer support these and have removed the corresponding documentation from

The BOSH oriented method documented in this is the supported method to deploy Kubernetes clusters with CFCR.


K8s 1.11 release kicked off the deprecation timeline for the Heapster component, see here for more info. As a result, we're in the process of replacing Heapster with Metrics Server in the upcoming releases of kubo-release.