clusterapi vsphere e2e test (kubernetes-sigs#137)

this is the first layout of e2e test: it has two stages 1) deploy a bootstrap cluster 2) apply secret to bootstrap cluster and apply job to bootstrap cluster. the job on bootstrap can be expanded with more regards to 1) target cluster topology. 2) target cluster verification. addressed review comments
jayunit100 · Dec 3, 2018 · f3b56bd · f3b56bd
1 parent dfcb4b4
commit f3b56bd
Show file tree

Hide file tree

Showing 17 changed files with 1,770 additions and 0 deletions.
diff --git a/scripts/e2e/README.md b/scripts/e2e/README.md
@@ -0,0 +1,91 @@
+Test cluster-api-provider-vsphere
+
+***Integration with Prow***   
+apply hack/secret.yml to Prow cluster/local cluster   
+apply hack/job.yml at Prow cluster/local cluster   
+note: the actual Prow job definition file will be at k8s.io/test-infra   
+
+```
+            +-----------------------------------------------------+
+            |                                                     |
+            |                                                     |
+            |        container running on Prow cluster:           |
+            |                                                     |
+            |        create bootstrap cluster (on VMC)            |
+            |        transfer secret from Prow to bootstrap       |
+            |        launch a ci job at bootstrap                 |
+            |        monitor job status                           |
+            |                                                     |
+            |                                                     |
+            |                             +---------------------+ |
+            |                             |  secret             | |
+            |                             +---------------------+ |
+            +-----------------------------------------------------+
+
+           +-------------------------------------------------------+
+           |        +--------------------------------------------+ |
+           |        |  secret: target VM SSH, bootstrap cluster  | |
+           |        |  kubeconfig, vsphere info                  | |
+           |        |                                            | |
+           |        +--------------------------------------------+ |
+           |                                                       |
+           |                             +-----------------------+ |
+           |                             |                       | |
+           |                             |     CI job:           | |
+           |                             | create target cluster | |
+           |                             | on VMC                | |
+           |                             +-----------------------+ |
+           |                                                       |
+           |        BOOTSTRAP CLUSTER (on VMC)                     |
+           |                                                       |
+           +-------------------------------------------------------+
+``` 
+
+
+***Launch CI from travis-ci***  
+```
+docker run \
+  --rm \
+  -v $HOME/.ssh:/root/ssh \
+  -e GOVC_URL=$GOVC_URL \
+  -e GOVC_USERNAME=$GOVC_USERNAME \
+  -e GOVC_PASSWORD=$GOVC_PASSWORD \
+  -e JUMPHOST=$JUMPHOST \
+  -e GOVC_INSECURE="true" \
+  -e VSPHERE_MACHINE_CONTROLLER_REGISTRY=$VSPHERE_MACHINE_CONTROLLER_REGISTRY \
+  -ti luoh/cluster-api-provider-vsphere-travis-ci:latest
+```
+note: set `$VSPHER_MACHINE_CONTROLLER_REGISTRY` if you want to test your local build controller
+
+
+***Architecture***  
+```
+
+                                             +-----------------------------------+
+      +----------------------+               |          VMC Infra                |
+      |   travis-ci env      |               |-----------------------------------|
+      |----------------------|               |+----+ +--------------------------+|
+      |                      |               ||    | |  bootstrap cluster       ||
+      |                      |               ||    | |                          ||
+      | cluster-api-vsphere- |               ||JUMP| |  cluster-api-vsphere-ci  ||
+      | travis-ci            |  SSH + HTTP   ||HOST| |  (a k8s job)             ||
+      |                      | +-----------> ||    | |                          ||
+      |                      | <-----------+ ||    | |                          ||
+      |                      |               ||    | +--------------------------+|
+      |                      |               ||    |                             |
+      |                      |               ||    | +--------------------------+|
+      |                      |               ||    | |  target cluster          ||
+      |                      |               ||    | |                          ||
+      |                      |               ||    | |                          ||
+      |                      |               |+----+ +--------------------------+|
+      +----------------------+               +-----------------------------------+
+                                             
+```
+      
+***Containers***  
+the vsphere-machine-controller containers for CI purpose are hosted at   
+`gcr.io/cnx-cluster-api/vsphere-cluster-api-provider`   
+the cluster-api-provider-vsphere-travis-ci hosted at   
+`luoh/cluster-api-provider-vsphere-travis-ci`   
+the cluster-api-provider-vsphere-ci hosted at   
+`gcr/cnx-cluster-api/cluster-api-provider-vsphere-ci`   
diff --git a/scripts/e2e/bootstrap_job.yml b/scripts/e2e/bootstrap_job.yml
@@ -0,0 +1,65 @@
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: cluster-api-provider-vsphere-ci
+spec:
+  backoffLimit: 4
+  template:
+    spec:
+      tolerations:
+      - effect: NoSchedule
+        key: node-role.kubernetes.io/master
+      containers:
+      - name: cluster-api-provider-vsphere-ci
+        image: gcr.io/cnx-cluster-api/cluster-api-provider-vsphere-ci:latest
+        env:
+          - name: TARGET_VM_SSH
+            valueFrom:
+              secretKeyRef:
+                name: clusterapi-provider-vsphere-ci-prow
+                key: target-vm-ssh
+          - name: TARGET_VM_SSH_PUB
+            valueFrom:
+              secretKeyRef:
+                name: clusterapi-provider-vsphere-ci-prow
+                key: target-vm-ssh-pub
+          - name: VSPHERE_CONTROLLER_VERSION
+            valueFrom:
+              secretKeyRef:
+                name: clusterapi-provider-vsphere-ci-prow
+                key: vsphere-controller-version
+          - name: PROVIDER_COMPONENT_SPEC
+            valueFrom:
+              secretKeyRef:
+                name: clusterapi-provider-vsphere-ci-prow
+                key: provider-component-spec
+          - name: VSPHERE_SERVER
+            valueFrom:
+              secretKeyRef:
+                name: clusterapi-provider-vsphere-ci-prow
+                key: vsphere-server
+          - name: VSPHERE_USERNAME
+            valueFrom:
+              secretKeyRef:
+                name: clusterapi-provider-vsphere-ci-prow
+                key: vsphere-username
+          - name: VSPHERE_PASSWORD
+            valueFrom:
+              secretKeyRef:
+                name: clusterapi-provider-vsphere-ci-prow
+                key: vsphere-password
+        volumeMounts:
+          - name: kube
+            mountPath: /root/.kube
+          - name: sshkeys
+            mountPath: /root/.ssh
+        command:
+        - "./clusterctl.sh"
+      volumes:
+      - name: kube
+        hostPath:
+          path: /home/vmware/.kube
+      - name: sshkeys
+        hostPath:
+          path: /home/vmware/.ssh
+      restartPolicy: Never
diff --git a/scripts/e2e/bootstrap_job/Dockerfile b/scripts/e2e/bootstrap_job/Dockerfile
@@ -0,0 +1,12 @@
+FROM photon:2.0
+LABEL maintainer="Hui Luo <luoh@vmware.com>"
+
+RUN tdnf install -y iputils wget openssh
+
+COPY *.sh /clusterapi/
+COPY bin /clusterapi/bin
+COPY spec /clusterapi/spec
+
+WORKDIR /clusterapi/
+CMD ["shell"]
+ENTRYPOINT ["/clusterapi/clusterctl.sh"]
diff --git a/scripts/e2e/bootstrap_job/Makefile b/scripts/e2e/bootstrap_job/Makefile
@@ -0,0 +1,24 @@
+# Makefile
+
+VERSION ?= $(shell git describe --exact-match 2> /dev/null || \
+                 git describe --match=$(git rev-parse --short=8 HEAD) --always --dirty --abbrev=8)
+REGISTRY ?=gcr.io/cnx-cluster-api/cluster-api-provider-vsphere-ci
+
+all: build upload clean
+.PHONY : all
+
+.PHONY : build
+build: 
+	docker build . --tag $(REGISTRY):$(VERSION)
+	docker tag $(REGISTRY):$(VERSION) $(REGISTRY):latest
+
+upload:
+	@echo "logging into gcr.io registry with key file"
+	@echo $$GCR_KEY_FILE | docker login -u _json_key --password-stdin gcr.io
+	docker push $(REGISTRY):$(VERSION)
+	docker push $(REGISTRY):latest
+	@echo docker logout gcr.io
+
+clean:
+	docker image rm -f $(REGISTRY):$(VERSION)
+	docker image rm -f $(REGISTRY):latest
diff --git a/scripts/e2e/bootstrap_job/clusterctl.sh b/scripts/e2e/bootstrap_job/clusterctl.sh
@@ -0,0 +1,56 @@
+#!/bin/sh
+
+# this script takes care of everything after bootstrap cluster created, it means
+# bootstrap need be created beforehand.
+
+# specs requires following enviroments variables:
+# VSPHERE_SERVER
+# VSPHERE_USERNAME
+# VSPHERE_PASSWORD
+# VSPHERE_CONTROLLER_VERSION
+# TARGET_VM_SSH  (base64 encoded)
+# TARGET_VM_SSH_PUB (base64 encoded)
+
+
+# base64 encode SSH keys (k8s secret automatically decode it)
+export TARGET_VM_SSH=$(echo -n "${TARGET_VM_SSH}" | base64 -w 0)
+export TARGET_VM_SSH_PUB=$(echo -n "${TARGET_VM_SSH_PUB}" | base64 -w 0)
+
+for filename in spec/*.template; do
+  newfilename="$(echo "$filename" | sed 's/template/yml/g')"
+  rm -f "$newfilename" temp.sh  
+  ( echo "cat <<EOF >$newfilename";
+    cat "$filename";
+    echo "EOF";
+  ) >temp.sh
+  chmod +x temp.sh
+  ./temp.sh
+done
+rm temp.sh
+
+# download kubectl binary
+retry=20
+until [ "$(ping www.google.com -c 1)" ]
+do
+   sleep 6
+   retry=$((retry - 1))
+   if [ $retry -lt 0 ]
+   then
+      echo "can not access internet"
+      exit 1
+   fi
+done
+wget https://storage.googleapis.com/kubernetes-release/release/v1.10.2/bin/linux/amd64/kubectl \
+     -O /usr/local/bin/kubectl
+chmod +x /usr/local/bin/kubectl
+
+# run clusterctl
+echo "test ${PROVIDER_COMPONENT_SPEC}"
+./bin/clusterctl create cluster --existing-bootstrap-cluster-kubeconfig ~/.kube/config -c ./spec/cluster.yml \
+    -m ./spec/machines.yml \
+    -p ./spec/${PROVIDER_COMPONENT_SPEC} \
+    --provider vsphere \
+    -v 6
+
+# cleanup the cluster
+# TODO (clusterctl delete is not working)
diff --git a/scripts/e2e/bootstrap_job/spec/cluster.template b/scripts/e2e/bootstrap_job/spec/cluster.template
@@ -0,0 +1,18 @@
+apiVersion: "cluster.k8s.io/v1alpha1"
+kind: Cluster
+metadata:
+  name: test1
+spec:
+    clusterNetwork:
+        services:
+            cidrBlocks: ["10.96.0.0/12"]
+        pods:
+            cidrBlocks: ["10.244.0.0/16"]
+        serviceDomain: "cluster.local"
+    providerConfig:
+      value:
+        apiVersion: "vsphereproviderconfig/v1alpha1"
+        kind: "VsphereClusterProviderConfig"
+        vsphereUser: "$VSPHERE_USERNAME"
+        vspherePassword: "$VSPHERE_PASSWORD"
+        vsphereServer: "$VSPHERE_SERVER"
diff --git a/scripts/e2e/bootstrap_job/spec/machines.template b/scripts/e2e/bootstrap_job/spec/machines.template
@@ -0,0 +1,32 @@
+items:
+- apiVersion: "cluster.k8s.io/v1alpha1"
+  kind: Machine
+  metadata:
+    generateName: clusterapi-prow-
+    labels:
+      set: master
+  spec:
+    providerConfig:
+      value:
+        apiVersion: "vsphereproviderconfig/v1alpha1"
+        kind: "VsphereMachineProviderConfig"
+        machineSpec:
+          datacenter: "SDDC-Datacenter"
+          datastore: "WorkloadDatastore"
+          resourcePool: "clusterapi"
+          vmFolder: "clusterapi"
+          networks:
+          - networkName: "sddc-cgw-network-3"
+            ipConfig:
+              networkType: dhcp
+          numCPUs: 2
+          memoryMB: 2048
+          template: "ubuntu-16.04-server-cloudimg-amd64"
+          disks:
+          - diskLabel: "Hard disk 1"
+            diskSizeGB: 15
+    versions:
+      kubelet: 1.11.2
+      controlPlane: 1.11.2
+    roles:
+    - Master
diff --git a/scripts/e2e/bootstrap_job/spec/machineset.template b/scripts/e2e/bootstrap_job/spec/machineset.template
@@ -0,0 +1,37 @@
+apiVersion: "cluster.k8s.io/v1alpha1"
+kind: MachineSet
+metadata:
+  name: clusterapi-machineset-1
+spec:
+  replicas: 2
+  selector:
+    matchLabels:
+      node-type: worker-node
+  template:
+    metadata:
+      labels:
+        node-type: worker-node
+    spec:
+      providerConfig:
+        value:
+          apiVersion: "vsphereproviderconfig/v1alpha1"
+          kind: "VsphereMachineProviderConfig"
+          machineSpec:
+            datacenter: "SDDC-Datacenter"
+            datastore: "WorkloadDatastore"
+            resourcePool: "clusterapi"
+            vmFolder: "clusterapi"
+            networks:
+            - networkName: "sddc-cgw-network-3"
+              ipConfig:
+                networkType: dhcp
+            numCPUs: 2
+            memoryMB: 2048
+            template: "ubuntu-16.04-server-cloudimg-amd64"
+            disks:
+            - diskLabel: "Hard disk 1"
+              diskSizeGB: 15
+      versions:
+        kubelet: 1.11.2
+      roles:
+      - Node