cfcr
Latest commit e5ac622 Oct 15, 2018

README.md

Cloud Foundry Container Runtime

A BOSH release for Kubernetes. Formerly named kubo.

Prerequisites

  • A BOSH Director configured with UAA, Credhub, and BOSH DNS. We recommend using BOSH Bootloader for this.

  • Latest kubo-deployment tarball

  • Accessing the master:

    • Single Master: Set up a DNS name pointing to your master's IP address
    • Multiple Masters: A TCP load balancer for your master nodes.
      • Use a TCP load balancer configured to connect to the master nodes on port 8443.
      • Add healthchecks using either a TCP dial or HTTP by looking for a 200 OK response from /healthz.
  • Cloud Config with

    • vm_types named minimal, small, and small-highmem (See cf-deployment for reference)
    • network named default
    • three availability zones azs named z1,z2,z3

    Note: the cloud-config properties can be customized by applying ops-files. See manifests/ops-files for some examples.

    If using loadbalancers then apply the vm_extension called cfcr-master-loadbalancer to the cloud-config to add the instances to your loadbalancers. See BOSH documentation for information on how to configure loadbalancers.

Hardware Requirements

Kubernetes uses etcd as its datastore. The official infrastructure requirements and example configurations for the etcd cluster can be found here.

Deploying CFCR

  1. Upload the latest Xenial stemcell to the director.

  2. Untar the kubo-deployment tarball and rename it kubo-deployment

  3. Deploy

    Option 1. Single Master
    cd kubo-deployment
    
    bosh deploy -d cfcr manifests/cfcr.yml \
      -o manifests/ops-files/misc/single-master.yml \
      -o manifests/ops-files/add-hostname-to-master-certificate.yml \
      -v api-hostname=[DNS-NAME]
    Option 2. Three Masters
    cd kubo-deployment
    
    bosh deploy -d cfcr manifests/cfcr.yml \
      -o manifests/ops-files/add-vm-extensions-to-master.yml \
      -o manifests/ops-files/add-hostname-to-master-certificate.yml \
      -v api-hostname=[LOADBALANCER-ADDRESS]

    Note: Loadbalancer address should be the external address (hostname or IP) of the loadbalancer you have configured.

    Check additional configurations, such as setting Kubernetes cloud provider, in docs.

  4. Add Kubernetes system components

    bosh -d cfcr run-errand apply-specs
  5. Run the following to confirm the cluster is operational

    bosh -d cfcr run-errand smoke-tests

BOSH Lite

CFCR clusters on BOSH Lite are intended for development. We run the deploy_cfcr_lite script to provision a cluster with the latest stemcell and master of kubo-release.

cd kubo-deployment
./bin/deploy_cfcr_lite

Accessing the CFCR Cluster with kubectl

  1. Login to the Credhub Server that stores the cluster's credentials:
    credhub login
    
  2. Find the director name by running
    bosh env
    
  3. Configure the kubeconfig for your kubectl client:
    cd kubo-deployment
    
    ./bin/set_kubeconfig <DIRECTOR_NAME>/cfcr https://[DNS-NAME-OR-LOADBALANCER-ADDRESS]:8443
    

Backup & Restore

We use BBR to perform backups and restores of the etcd node within a CFCR cluster. Our backup currently takes an etcd snapshot without interruptions to the cluster. However, for restore we take both the kube-apiserver and etcd offline to restore the cluster with the specified snapshot. Restore is a destructive operation that will completely overwrite any existing data on the cluster. For a closer look at the bbr scripts, check out:

To run the bbr cli against a CFCR cluster, follow the steps under "BOSH Deployment" on the BBR documentation page.

Note: this feature is currently available for single master deployments only. Work for BBR support on multi-master clusters is underway.

Monitoring

Follow the recommendations in etcd's documentation for monitoring etcd metrics.

Deprecations

Deployment scripts and docs

CFCR had a set of scripts, including deploy_bosh and deploy_k8s, that were the primary mechanism we supported to deploy BOSH and Kubernetes clusters. We no longer support these and have removed the corresponding documentation from https://docs-cfcr.cfapps.io

The BOSH oriented method documented in this README.md is the supported method to deploy Kubernetes clusters with CFCR.

Heapster

K8s 1.11 release kicked off the deprecation timeline for the Heapster component, see here for more info. As a result, we're in the process of replacing Heapster with Metrics Server in the upcoming releases of kubo-release.