Creating a Kubernetes Cluster

It4lik edited this page Jul 20, 2017 · 21 revisions
Clone this wiki locally

Table of Contents


Setting up a Kubernetes cluster to run a web application demonstrates the power of Kubernetes as a service. Photon Platform supports Kubernetes clusters with multiple masters, persistent volumes, and authentication. A Kubernetes cluster can use traditional networking or virtualized networking powered by VMware NSX.

Deploying a Kubernetes cluster involves two main steps: creating resources for a tenant and spinning them up into a cluster with the photon service create command. On Photon Controller, the following primitives are the building blocks of clusters:

  • Images
  • Virtual machines
  • Disks
  • Quotas


A Kubernetes cluster carries several requirements:

  • A static IP address to assign as the master IP address of the cluster.
  • A static IP address for the load balancer.
  • At least one static IP address for an etcd node in the cluster. For high availability of etcd, you'll need three static IP addresses.
  • The Kubernetes image for Photon Controller.

The instructions assume that you are using a Linux workstation with the Photon Controller CLI connected to a deployment of Photon Controller on ESXi without VMware NSX. You can also use a Mac or Windows workstation with the Photon Controller CLI installed, but you will have to adapt some of the commands to your operating system and environment.

For information about photon commands, subcommands, and options, see the help for the Photon Controller CLi on your Linux workstation; examples:

photon --help
photon tenant create --help
photon service create -h

The VMware NSX option: If you deployed Photon Platform with NSX, you can set up a Kubernetes cluster with a virtualized network; see Setting Up a Kubernetes Cluster with NSX.

Obtaining, Uploading, and Enabling the Kubernetes Image

Download the Kubernetes disk image for Photon Controller from the following URL to the workstation on which you are running the Photon Controller command-line utility:

The Kubernetes image is packaged as an OVA; it has a file name that looks like this, though the version, build, and other numbers might be slightly different:


Upload the the Kubernetes image to Photon Controller by running the following commands as the system administrator, replacing the variables with the IP address of the load balancer for the Photon Controller management plane:

photon target set https://mgmt-ip-address:443
photon image create kubernetes-1.6.0-pc-1.2-5caa892.ova -n kube1 -i EAGER

The upload takes a few minutes. When it finishes, enable the image by obtaining its ID with the photon image list command and using it to replace the variable in the enable-cluster-type command:

photon image list
photon deployment enable-cluster-type -k KUBERNETES -i <Kubernetes_image_ID>

As the Photon Controller system administrator, you typically need to enable a cluster type only once. Enabling the cluster type sets Photon Controller to use the given image to deploy each of your Kubernetes clusters. You should, however, disable the cluster type for a given image before you delete the image.

Creating a Tenant and a Project for the Cluster

A tenant is a unit of administrative control allocated a quota for projects, such as a Kubernetes cluster. The following sequence of commands creates a tenant, sets a quota to give it a pool of resources, and creates a project that uses use all the resources in the pool.

photon tenant create plato
photon tenant quota set plato \
         --limits 'vm.count 100 COUNT, vm.memory 1000 GB, vm.cpu 500 COUNT'
photon project create --tenant "plato" --name "plato-prjt" \
    --limits "vm.memory 100 GB, vm.cpu 100 COUNT, vm 100 COUNT, \
    persistent-disk 100 COUNT, persistent-disk.capacity 200 GB, \
    ephemeral-disk 100 COUNT, ephemeral-disk.capacity 200 GB" 
photon tenant set "plato"
photon project set "plato-prjt"

Creating Resources for Use in the Cluster

The following command creates a flavor for a small VM that you'll use when you create the Kubernetes cluster.

photon flavor create --name cluster-small -k vm \
--cost "vm 1 COUNT, vm.cpu 1 COUNT, vm.memory 2 GB"

If you are setting up a Kubernetes cluster for production purposes in an environment without memory constraints, you can omit this command because Photon Controller includes the following default flavors for clusters:

  • cluster-master-vm. Photon Controller supplies this flavor as the default for a Kubernetes master node.
  • cluster-other-vm. This flavor is for etcd nodes and Kubernetes workers.
  • cluster-vm-disk. This flavor is the default ephemeral disk for a Kubernetes cluster.

When you create a Kubernetes cluster, Photon Controller uses the default flavors unless you specify other flavors.

Creating Resources for Containerized Applications

The following sequence of commands provisions some resources for applications. You can skip these commands if you want, but the resources they create might come in handy later when you deploy applications. For more information, see Flavors.

photon -n flavor create --name "vm-basic" --kind "vm" \
       --cost "vm 1 COUNT, vm.cpu 2 COUNT, vm.memory 2 GB" \
photon -n flavor create --name "disk-eph" --kind "ephemeral-disk" \
       --cost "ephemeral-disk 1 COUNT"
photon -n flavor create --name "disk-persist" --kind "persistent-disk" \
       --cost "persistent-disk 1 COUNT"

Setting Up a Network for Use with the Cluster

Finally, make a network for the cluster and set it as the default:

photon subnet create --name "vm-network" --portgroups "VM Network" 

From the output of the photon subnet create command, note the network ID and then use it to set the default network in the following command:

photon subnet set-default <network_ID>

Creating the Kubernetes Cluster

You are now ready to create a Kubernetes cluster by running the following command. Replace the example IP addresses with those from your ESXi and network environments. The IP address for the master-ip option should contain the static IP address that you want to assign to the Kubernetes cluster. The etcd option should also contain a static IP address.

photon service create -n kube-socrates -k KUBERNETES --master-ip \
    --etcd1 --container-network --dns \
    --gateway --netmask -c 1 --vm_flavor cluster-small 

The service create command prompts you for several inputs. You can press Enter to accept the defaults and type 1 to for a worker node, or you can specify the options that you want.

The photon service create Command

The photon service create --help command shows descriptions of the options' values. Here's the output of its help:

photon service create -h
   photon service create - Create a new service
   photon service create [command options]  
   Create a new Kubernetes service or Harbor Docker registry. 
   photon service create -n k8-service -k KUBERNETES --dns \ 
     --gateway --netmask --master-ip \ 
     --container-network --etcd1 \ 
     -c 1 -v cluster-vm -d small-disk --ssh-key ~/.ssh/
   --tenant value, -t value            Tenant name
   --project value, -p value           Project name
   --name value, -n value              Service name
   --type value, -k value              Service type (KUBERNETES or HARBOR)
   --vm_flavor value, -v value         VM flavor name for master and worker
   --master-vm-flavor value, -m value  Override master VM flavor
   --worker-vm-flavor value, -W value  Override worker VM flavor
   --disk_flavor value, -d value       Disk flavor name
   --subnet_id value, -w value         VM subnet ID
   --image-id value, -i value          Image ID
   --worker_count value, -c value      Worker count (default: 0)
   --dns value                         VM network DNS server IP address
   --gateway value                     VM network gateway IP address
   --netmask value                     VM network netmask
   --number-of-masters value           Number of Kubernetes masters 
     (required for Kubernetes services with virtual networking) (default: 0)
   --master-ip value                   Kubernetes master IP address 
     (required for Kubernetes services with physical networking)
   --master-ip2 value                  Static IP address with which 
                                       to create Kubernetes node 2
   --load-balancer-ip value            Kubernetes load balancer IP address 
     (required for Kubernetes services with physical networking)
   --container-network value           CIDR representation of the container network, 
     e.g. '' (required for Kubernetes services)
   --number-of-etcds value             Number of Etcd instances for Kubernetes 
     (required for Kubernetes services with virtual networking) (default: 0)
   --etcd1 value                       Static IP address with which to create etcd node 1 
     (required for Kubernetes services with physical networking)
   --etcd2 value                       Static IP address with which to create etcd node 2
   --etcd3 value                       Static IP address with which to create etcd node 3
   --ssh-key value                     The file path of the SSH key
   --registry-ca-cert value            The file path of the file containing the CA 
     certificate for a docker registry (optional)
   --admin-password value              The Harbor registry admin password (optional). 
     The password needs to have at least 7 characters with 1 lowercase letter, 1 capital 
     letter and 1 numeric character. If not specified, the default user name is admin 
     and the password is Harbor12345
   --batchSize value                   Batch size for expanding worker nodes (default: 0)
   --wait-for-ready                    Wait synchronously for the service to become 
                                       ready and expanded fully

Checking the Cluster's Resources and Status

After provisioning the cluster and its resources, you can log in to the Photon Controller web interface to check them out:


Take a few moments to click through the web UI. Examine the tenant who owns the project running Kubernetes:

A Tenant in Photon Platform

In the list of the tenant's projects, you can click a project to view information about it:

A Project in Photon Platform

And once you've reached the project, you can click Clusters and then click the name of a cluster to see its settings:

A Kubernetes Cluster

Opening the Kubernetes Dashboard

To open the Kubernetes web interface in a browser, you first must obtain information about authentication for the Kubernetes cluster from Photon Controller and then you must add it to your kubectl configuration file. Here's how:

First, get the kubectl authentication information from Photon Controller by running the following command on your workstation. Replace the username and password with your Photon Controller username and password:

photon service get-kubectl-auth -u <username> -p <password> <service-id>

Here's and example:

photon service get-kubectl-auth -u \
         -p 'MySecret1!' 446c5839-992c-456a-b271-bc2aa48e88fc

Second, copy the command's output and add it to the following command to insert it into the kubectl configuration. Here's an example with sample output:

kubectl config set-credentials \
--auth-provider=oidc \
--auth-provider-arg=idp-issuer-url= \
--auth-provider-arg=client-id=fc50e7cd-7166-4025-907b-c2af210fb902 \
--auth-provider-arg=client-secret=fc50e7cd-7166-4025-907b-c2af210fb902 \
--auth-provider-arg=refresh-token=eyJhbGciOiJSUzI1NiJ9.eyJzdWIi ... \
--auth-provider-arg=id-token=eyJhbGciOiJSUzI1NiJ9.eyJzdWIiOiJhZ ... \

Next, set the cluster and server:

kubectl config set-cluster <cluster-name> \
--server <k8s-load-balancer-address> \


kubectl config set-cluster default-cluster \
--server \

Next, set the context in the kubectl config:

kubectl config set-context <context-name> \
--cluster <cluster-name> \
--user <username>


kubectl config set-context default-system \
--cluster default-cluster \

Now make the context the current context:

kubectl config use-context <context-name>


kubectl config use-context default-system

Verify that you set it up correctly and that it works:

kubectl get pods --all-namespaces

Finally, launch the Kubernetes user interface and go to it by entering the following URL in your web browser:

kubectl proxy

Now you're ready to load and run an application in your Kubernetes cluster.

Deploying an nginx Web Server

Now that you've got a Kubernetes cluster up and running on Photon Controller, you can run a application as a service. This example deploys an nginx web server to demonstrate how to launch an application.

You'll need the Kubernetes command-line interface, kubectl, which you can quickly download through the Photon Platform web interface, as the following image illustrates:

Install kubectl

Or you can download the version of kubectl for a 64-bit Linux machine from the Kubernetes web site; see the Kubernetes user guide for instructions on how to download versions for other operating systems.

After you download kubectl, install it by changing its mode bits so that it is executable and then moving it to /usr/local/bin. Here's an example:

cd ~/Downloads/
chmod +x kubectl 
sudo mv kubectl /usr/local/bin

Now you can check your nodes' status, and you can omit the server option in the commands because the default is set:

kubectl get nodes
NAME            STATUS    AGE     Ready     25d   Ready     25d

You're now ready to deploy the nginx web server as a service. Copy the following block of code into a file named nginx.yml. The code block contains the configuration for running the nginx web server on Kubernetes.

By setting the type field to NodePort, this YAML file instructs the Kubernetes master to allocate a port from the default node port range of 30000 to 32767. Each node proxies the same port to your service. A node port exposes a node's IP address; for more information, see the Kubernetes documentation.

apiVersion: v1
kind: Service
  name: nginx-demo-service
    app: nginx-demo
  type: NodePort
  - port: 80
    protocol: TCP
    name: http
    app: nginx-demo
apiVersion: v1
kind: ReplicationController
  name: nginx-demo
  replicas: 3
        app: nginx-demo
      - name: nginx-demo
        image: nginx
        - containerPort: 80

Run the following command to create the nginx service on the Kubernetes cluster; you might need to modify the path the YAML file if it's not in your current working directory:

kubectl create -f nginx.yml

service "nginx-demo-service" created
replicationcontroller "nginx-demo" created

Now view the services running on the cluster:

kubectl get svc
NAME                 CLUSTER-IP   EXTERNAL-IP   PORT(S)     AGE
kubernetes      <none>        443/TCP     3d
nginx-demo-service   <nodes>       80/TCP      5h

Finally, run the following command to obtain the NodePort on which the nginx web service is exposed as a service:

kubectl describe svc nginx-demo-service

Name:			nginx-demo-service
Namespace:		default
Labels:			app=nginx-demo
Selector:		app=nginx-demo
Type:			NodePort
Port:			http	80/TCP
NodePort:		http	30786/TCP
Endpoints:,, + 3 more...
Session Affinity:	None

You can also see the nginx workload running in the Kubernetes web interface:

Kubernetes Workloads

That's it. Now you can connect to the nginx web server by launching a web browser and pointing it at the IP address of the Kubernetes master plus the NodePort on which the service is running, which is 30786 in this case:

You should see the nginx welcome screen:

nginx Welcome Screen

Upgrading to New Version of Kubernetes

You can upgrade to a new version of Kubernetes by running the photon service change-version command. Here's the help output for the command:

photon service change-version -h
   photon service change-version - Configure the service 
   to use specified image id
   photon service change-version [command options] service-id
   Example: photon service change-version \
   9b159e92-9495-49a4-af58-53ad4764f616 \
   -i 2aeaf034-3b02-4873-a6fc-f92615dca849
   --image-id value, -i value  Image ID
   --wait-for-ready            Wait synchronously for the 
   service to become ready and fully upgraded

Troubleshooting Cluster Creation

If the creation of the Kubernetes cluster failed, run the following photon command to clean up clusters that might be in an error state.

Keep in mind that this command deletes all your clusters, so you should run it only if you are trying to deploy your first one:

photon -n service list | awk '{print $1}' | xargs -n 1 photon -n service delete

Another troubleshooting method is to log into the VMs that Photon Controller creates for the Kubernetes master and the etcd nodes. The default password for the root account is changeme. You should change it the first time you log in.

After logging in, examine the system's status and logs as well as the Docker networking configuration.

You can get the ID of the cluster with the following the photon service list command; example:

photon service list
ID                                    Name           Type        State  Worker Count
dd0118d1-3b91-473d-8ecf-c25a373e75d1  kube-socrates  KUBERNETES  READY  1
Total: 1

And then you can view more information about the cluster by running adding the ID to the photon service show command; example with abridged output:

photon -n service show dd0118d1-3b91-473d-8ecf-c25a373e75d1
dd0118d1-3b91-473d-8ecf-c25a373e75d1	kube-socrates	READY	KUBERNETES	1
cluster_version:v1.6.0 g

Restoring Default Flavors

If you mistakenly deleted Photon's default flavors for Kubernetes clusters, you can restore them by running the following commands as the system administrator:

photon -n flavor create --name "cluster-master-vm" --kind "vm" \
                        --cost "vm 1 COUNT, vm.cpu 4 COUNT, vm.memory 4 GB"
photon -n flavor create --name "cluster-other-vm" --kind "vm" \
                        --cost "vm 1 COUNT, vm.cpu 1 COUNT, vm.memory 2 GB"
photon -n flavor create --name "cluster-vm-disk" --kind "ephemeral-disk" \
                        --cost "ephemeral-disk 1 COUNT" 

Dealing with "Not Enough Memory Resource" Errors

The default flavors for a Kubernetes cluster show you how much memory a default Kubernetes cluster requires:

  • A master node requires 4 CPUs and 4 GB of RAM.
  • Each worker node that's running as its own VM takes 1 CPU and 2 GB of RAM.
  • Each etcd node that's running as its own VM takes 1 CPU and 2 GB or RAM.

So, if you were to use the default flavors to build a Kubernetes cluster with a master, three workers, and three etcd nodes all running on their own VMs, the total memory requirement would be 10 CPUs and 16 GB of RAM.

If you're not tracking the memory usage of the ESXi machines in your Photon Controller, you might get an error code saying "NotEnoughMemoryResource." There are several ways to address the error:

  • Reduce the size of the cluster you are trying to create.
  • Free up resources by deleting unused virtual machines.
  • Add additional memory or ESXi machines to the cluster.
  • Delete unused Kubernetes clusters.

Another alternative for working in an environment with memory constraints is to make a small flavor and use it when you build a cluster, as this article did earlier when it demonstrated how to create a cluster.

Deleting a Cluster to Reclaim Space

To delete a cluster to free up space for a new cluster in an environment with memory constraints, run this command:

photon service delete <cluster_ID>

Deleting a Kubernetes Image

If you want to wipe out the Kubernetes image for some reason, such as recovering the storage space for an image that's old or no longer in use, you can delete it, You should, however, make sure that it's disabled as the cluster type before you delete it:

photon deployment disable-cluster-type -k KUBERNETES -i <Kubernetes_image_ID>
photon image delete <kubernetes_image_filename>