Cloud Foundry Container Runtime
- Slack: #cfcr on https://slack.cloudfoundry.org
- Pivotal Tracker: https://www.pivotaltracker.com/n/projects/2093412
A BOSH Director configured with UAA, Credhub, and BOSH DNS. We recommend using BOSH Bootloader for this.
Accessing the master:
- Single Master: Set up a DNS name pointing to your master's IP address
- Multiple Masters: A TCP load balancer for your master nodes.
- Use a TCP load balancer configured to connect to the master nodes on port 8443.
- Add healthchecks using either a TCP dial or HTTP by looking for a
200 OKresponse from
Cloud Config with
small-highmem(See cf-deployment for reference)
- three availability zones
Note: the cloud-config properties can be customized by applying ops-files. See
manifests/ops-filesfor some examples.
If using loadbalancers then apply the
cfcr-master-loadbalancerto the cloud-config to add the instances to your loadbalancers. See BOSH documentation for information on how to configure loadbalancers.
Kubernetes uses etcd as its datastore. The official infrastructure requirements and example configurations for the etcd cluster can be found here.
Upload the latest Xenial stemcell to the director.
Untar the kubo-deployment tarball and rename it
Option 1. Single Master
cd kubo-deployment bosh deploy -d cfcr manifests/cfcr.yml \ -o manifests/ops-files/misc/single-master.yml \ -o manifests/ops-files/add-hostname-to-master-certificate.yml \ -v api-hostname=[DNS-NAME]
Option 2. Three Masters
cd kubo-deployment bosh deploy -d cfcr manifests/cfcr.yml \ -o manifests/ops-files/add-vm-extensions-to-master.yml \ -o manifests/ops-files/add-hostname-to-master-certificate.yml \ -v api-hostname=[LOADBALANCER-ADDRESS]
Note: Loadbalancer address should be the external address (hostname or IP) of the loadbalancer you have configured.
Check additional configurations, such as setting Kubernetes cloud provider, in docs.
Add Kubernetes system components
bosh -d cfcr run-errand apply-specs
Run the following to confirm the cluster is operational
bosh -d cfcr run-errand smoke-tests
CFCR clusters on BOSH Lite are intended for development. We run the deploy_cfcr_lite script to provision a cluster with the latest stemcell and master of kubo-release.
cd kubo-deployment ./bin/deploy_cfcr_lite
Accessing the CFCR Cluster with kubectl
- Login to the Credhub Server that stores the cluster's credentials:
- Find the director name by running
- Configure the
cd kubo-deployment ./bin/set_kubeconfig <DIRECTOR_NAME>/cfcr https://[DNS-NAME-OR-LOADBALANCER-ADDRESS]:8443
Backup & Restore
We use BBR to perform backups and restores of the etcd node within a CFCR cluster. Our backup currently takes an etcd snapshot without interruptions to the cluster. However, for restore we take both the kube-apiserver and etcd offline to restore the cluster with the specified snapshot. Restore is a destructive operation that will completely overwrite any existing data on the cluster. For a closer look at the bbr scripts, check out:
To run the
bbr cli against a CFCR cluster, follow the steps under "BOSH Deployment" on the BBR documentation page.
Note: this feature is currently available for single master deployments only. Work for BBR support on multi-master clusters is underway.
Follow the recommendations in etcd's documentation for monitoring etcd metrics.
Deployment scripts and docs
CFCR had a set of scripts, including
deploy_k8s, that were the primary mechanism we supported to deploy BOSH and Kubernetes clusters. We no longer support these and have removed the corresponding documentation from https://docs-cfcr.cfapps.io
The BOSH oriented method documented in this README.md is the supported method to deploy Kubernetes clusters with CFCR.
K8s 1.11 release kicked off the deprecation timeline for the Heapster component, see here for more info. As a result, we're in the process of replacing Heapster with Metrics Server in the upcoming releases of kubo-release.