use KIND multinode cluster and other improvements

for networking development and testing, using single node
environments hide a big part of the problems.

We should use multinode by defalt if possible.

Another improvements:

* the apiserver autodiscovers the API address, so we don't
  need to pass the kubeconfig

* fix multus webook problem

* bump KIND version to 0.8.1

* allow to configure IPv6 environments (experimental)

* use kubectl wait instead of bash loops

Signed-off-by: Antonio Ojea <aojea@redhat.com>

hack/KIND_CNO.md

@@ -7,18 +7,48 @@ fix in an environment that can be brought up locally and within a few minutes.
 ## How does it work?
 
 At a high level, the deployment will create a docker container per K8S node. This docker container will host its own
-containerd instance, and pods, kubelet, etc will all run inside of these docker containers. The ``kind.yaml`` file
-declares how many nodes will be deployed. Currently only a single node is supported. There are plans to support HA as
-well as more nodes. ``ovn-kind-cno.sh`` is the script that will handle deploying K8S, CNO, and consequently
-Multus/OVN-K8S.
+containerd instance, and pods, kubelet, etc will all run inside of these docker containers.
+
+``ovn-kind-cno.sh`` is the script that will handle deploying K8S, CNO, and consequently Multus/OVN-K8S.
+By default it will use the following configuration for kind, and IPv4 multinode cluster:
+
+```yaml
+# config for 1 control plane node and 2 workers (necessary for conformance)
+kind: Cluster
+apiVersion: kind.x-k8s.io/v1alpha4
+networking:
+  ipFamily: ${IP_FAMILY:-ipv4}
+  disableDefaultCNI: true
+  podSubnet: ${CLUSTER_CIDR:-10.244.0.0/16}
+  serviceSubnet: ${SERVICE_CIDR:-10.96.0.0/12}
+nodes:
+- role: control-plane
+- role: worker
+- role: worker
+```
+
+It accepts environment variables to customize the environment, with the following defaults:
+
+```sh
+K8S_VERSION=${K8S_VERSION:-v1.18.2}
+BUILD_OVN=${BUILD_OVN:-false}
+BUILD_CNO=${BUILD_CNO:-false}
+BUILD_MULTUS=${BUILD_MULTUS:-false}
+CNO_PATH=${CNO_PATH:-$GOPATH/src/github.com/openshift/cluster-network-operator}
+OVN_K8S_PATH=${OVN_K8S_PATH:-$GOPATH/src/github.com/ovn-org/ovn-kubernetes}
+KIND_CONFIG=${KIND_CONFIG:-$HOME/kind-ovn-config.yaml}
+export KUBECONFIG=${HOME}/kube-ovn.conf
+NUM_MASTER_NODES=1
+OVN_KIND_VERBOSITY=${OVN_KIND_VERBOSITY:-5}
+```
 
 ## Requirements
 
 The deployment should work locally on a laptop for a single node. To deploy, it is required to have golang
 (1.11 or later), kubectl, and docker installed. The KIND deployment requires at least Kubernetes v1.17 (installed by
 default) so building corresponding kubectl client is best.
 
-It is also required to install KIND before deploying. As of this writing the latest KIND release is v0.7.0, and
+It is also required to install KIND before deploying. As of this writing the latest KIND release is v0.8.1, and
 installation instructions can be found at https://github.com/kubernetes-sigs/kind#installation-and-usage.
 
 ## Usage
@@ -36,32 +66,138 @@ restarted post deployment, the changes will be lost, and OVN will stop functioni
 
 To build OVN K8S:
 
-````
+```
 $ BUILD_OVN=true ovn-kind-cno.sh
-````
+```
 
 This will build an OVN K8S docker image to use with the deployment from a local git workspace. By default the script will use
 a path relative to your GOPATH. To override this, specify `CNO_PATH` or `OVN_K8S_PATH` when executing the above.
 
+### Post deployment
+
 Post deployment you can simply use kubectl to interact with your cluster. Additionally you may wish to exec into the
 docker node and poke around. Inside the docker container you can see all the running pods via:
 
-````
-root@ovn-control-plane:/# ctr  --namespace k8s.io containers list
-````
+```sh
+docker exec -it ovn-control-plane crictl ps
+CONTAINER           IMAGE               CREATED             STATE               NAME                      ATTEMPT             POD ID
+fe1fe79be1a73       e4b1ab4cbf659       2 hours ago         Running             kube-multus               0                   91dad15b65f72
+3a69b9c3b6456       e9b443d218dc7       2 hours ago         Running             ovnkube-master            0                   6e24b8c012e49
+394730b38dc46       e9b443d218dc7       2 hours ago         Running             sbdb                      0                   6e24b8c012e49
+504d7fbb09129       e9b443d218dc7       2 hours ago         Running             nbdb                      0                   6e24b8c012e49
+29a497add0ee8       e9b443d218dc7       2 hours ago         Running             ovnkube-node              0                   d2ef48bd1b871
+c9232a2cabdb1       e9b443d218dc7       2 hours ago         Running             northd                    0                   6e24b8c012e49
+d1e1de2d2b763       e9b443d218dc7       2 hours ago         Running             ovn-controller            0                   d2ef48bd1b871
+884a0ea94c0ff       e9b443d218dc7       2 hours ago         Running             ovs-daemons               0                   7fbfcf5dbd924
+eac57b49af7ab       4ec1801e760a8       2 hours ago         Running             network-operator          0                   f7821d3420787
+29c039335d1b8       dd61f68ee6f73       3 hours ago         Running             kube-proxy                0                   1554057b10ecd
+e6fca3ad23f05       303ce5db0e90d       3 hours ago         Running             etcd                      0                   f6a03d6d1ce11
+43d6cfc99f9f0       06f726e5bab40       3 hours ago         Running             kube-apiserver            0                   42834ae09e8ce
+898e5e0a7aa64       4b2a99ce99208       3 hours ago         Running             kube-controller-manager   0                   a5bf81537cd99
+3d761203df63c       5e7eb76f91581       3 hours ago         Running             kube-scheduler            0                   34ef9a3da0f2d
+```
 
 Kubelet logs can also be found via journalctl.
 
+```sh
+-- Logs begin at Sun 2020-06-07 09:48:19 UTC, end at Sun 2020-06-07 12:28:37 UTC. --
+Jun 07 09:48:21 ovn-control-plane systemd[1]: Started kubelet: The Kubernetes Node Agent.
+Jun 07 09:48:24 ovn-control-plane kubelet[151]: Flag --fail-swap-on has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs
+/tasks/administer-cluster/kubelet-config-file/ for more information.
+Jun 07 09:48:24 ovn-control-plane kubelet[151]: F0607 09:48:24.108531     151 server.go:199] failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file "/var/lib/kubelet/config.yaml", error: open /var/lib/kubelet/config.yaml: no such file or directory
+```
+
+### Using the cluster
+
+You can use now the cluster, deploy applications and expose them.
+
+Let's create a Deployment:
+
+```sh
+kubectl create deployment hello-node --image=k8s.gcr.io/echoserver:1.4
+deployment.apps/hello-node created
+```
+
+and expose it using a NodePort Service.
+
+```sh
+kubectl expose deployment hello-node --type NodePort --port=8080
+service/hello-node exposed
+```
+
+you can check the port used by the NodePort service:
+
+```sh
+ kubectl get service hello-node -o wide
+NAME         TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE   SELECTOR
+hello-node   NodePort   10.111.137.68   <none>        8080:32591/TCP   52s   app=hello-node
+```
+
+you should be able to access the service in any of the cluster nodes IP, that you can obtain using:
+
+```sh
+kubectl get nodes -o wide
+NAME                STATUS   ROLES    AGE     VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE           KERNEL-VERSION     CONTAINER-RUNTIME
+ovn-control-plane   Ready    master   4h29m   v1.18.2   172.18.0.3    <none>        Ubuntu 20.04 LTS   5.4.0-33-generic   containerd://1.3.3-14-g449e9269
+ovn-worker          Ready    <none>   4h28m   v1.18.2   172.18.0.2    <none>        Ubuntu 20.04 LTS   5.4.0-33-generic   containerd://1.3.3-14-g449e9269
+ovn-worker2         Ready    <none>   4h28m   v1.18.2   172.18.0.4    <none>        Ubuntu 20.04 LTS   5.4.0-33-generic   containerd://1.3.3-14-g449e9269
+```
+
+if everything is ok, and using this examples IP 172.18.0.2 and Port 32591, we should obtain an answer like this:
+
+```sh
+curl http://172.18.0.2:32591
+CLIENT VALUES:
+client_address=100.64.2.1
+command=GET
+real path=/
+query=nil
+request_version=1.1
+request_uri=http://172.18.0.2:8080/
+
+SERVER VALUES:
+server_version=nginx: 1.10.0 - lua: 10001
+
+HEADERS RECEIVED:
+accept=*/*
+host=172.18.0.2:32591
+user-agent=curl/7.68.0
+BODY:
+```
+
+### Running kubernetes e2e test
+
+Once you have your cluster working, you can use it to debug Kubernetes e2e tests.
+
+In order to do that you just have to checkout the desired Kubernetes branch and
+compile the e2e.test binary:
+
+```
+cd $GOPATH/src/k8s.io/kubernetes
+bazel build //test/e2e:e2e.test
+```
+
+Execute your tests using the KIND cluster kubeconfig file:
+
+```
+bazel-bin/test/e2e/e2e.test -kubeconfig $HOME/kube-ovn.conf -ginkgo.focus="\[sig-network\].*Conformance" -num-nodes 2
+```
+
+### Cleaning up
+
 In order to clean up your environment and remove your KIND cluster, simply execute:
 
-````
+```sh
 $ kind delete cluster --name ovn
-````
+```
 
 ## Todo
 
 * Add support for building custom Multus
-* Fix multus-admission-controller not coming up
 
+* Run IPv6 only cluster (experimental)
 
+```sh
+$ IP_FAMILY=ipv6 ovn-kind-cno.sh
+```
 

hack/ovn-kind-cno.sh

@@ -2,17 +2,27 @@
 set -euo pipefail
 
 # Version v1.17.0 or higher is required
-K8S_VERSION=${K8S_VERSION:-v1.17.0}
+K8S_VERSION=${K8S_VERSION:-v1.18.2}
 BUILD_OVN=${BUILD_OVN:-false}
 BUILD_CNO=${BUILD_CNO:-false}
 BUILD_MULTUS=${BUILD_MULTUS:-false}
 CNO_PATH=${CNO_PATH:-$GOPATH/src/github.com/openshift/cluster-network-operator}
 OVN_K8S_PATH=${OVN_K8S_PATH:-$GOPATH/src/github.com/ovn-org/ovn-kubernetes}
-CLUSTER_CIDR=${CLUSTER_CIDR:-"172.16.0.0/16"}
-SERVICE_NETWORK=${SERVICE_NETWORK:-"172.30.0.0/16"}
-# Skip the comment lines and retrieve the number of Master nodes from kind.yaml file.
-NUM_MASTER_NODES=`grep "^[^#]" kind.yaml | grep -c "role\: control-plane"`
-OVN_KIND_VERBOSITY=${OVN_KIND_VERBOSITY:-0}
+KIND_CONFIG=${KIND_CONFIG:-$HOME/kind-ovn-config.yaml}
+export KUBECONFIG=${HOME}/kube-ovn.conf
+NUM_MASTER_NODES=1
+OVN_KIND_VERBOSITY=${OVN_KIND_VERBOSITY:-5}
+
+# Default networks (same as in KIND)
+if [ "${IP_FAMILY:-ipv4}" = "ipv6" ]; then
+  CLUSTER_CIDR=${CLUSTER_CIDR:-"fd00:10:244::/48"}
+  SERVICE_NETWORK=${SERVICE_NETWORK:-"fd00:10:96::/112"}
+  HOST_PREFIX=64
+else
+  CLUSTER_CIDR=${CLUSTER_CIDR:-"10.244.0.0/16"}
+  SERVICE_NETWORK=${SERVICE_NETWORK:-"10.96.0.0/12"}
+  HOST_PREFIX=24
+fi
 
 # Check for docker
 if ! command -v docker; then
@@ -26,38 +36,30 @@ if ! command -v kubectl; then
   exit 1
 fi
 
-# Detect IP to use as API server
-API_IP=$(ip -4 addr | grep -oP '(?<=inet\s)\d+(\.\d+){3}' | grep -v "127.0.0.1" | head -n 1)
-if [ -z "$API_IP" ]; then
-  echo "Error detecting machine IP to use as API server"
-  exit 1
-fi
-
-sed -i "s/apiServerAddress.*/apiServerAddress: ${API_IP}/" kind.yaml
-
 # Ensure reachability to host via Docker network
 if ! sudo iptables -C DOCKER-USER -j ACCEPT > /dev/null 2>&1; then 
   sudo iptables -I DOCKER-USER -j ACCEPT
 fi
 
+ # create the config file
+  cat <<EOF > ${KIND_CONFIG}
+# config for 1 control plane node and 2 workers (necessary for conformance)
+kind: Cluster
+apiVersion: kind.x-k8s.io/v1alpha4
+networking:
+  ipFamily: ${IP_FAMILY:-ipv4}
+  disableDefaultCNI: true
+  podSubnet: ${CLUSTER_CIDR:-10.244.0.0/16}
+  serviceSubnet: ${SERVICE_NETWORK:-10.96.0.0/12}
+nodes:
+- role: control-plane
+- role: worker
+- role: worker
+EOF
+
+
 # Create KIND cluster
-kind create cluster --name ovn --kubeconfig ${HOME}/admin.conf --image kindest/node:${K8S_VERSION} --config=./kind.yaml -v ${OVN_KIND_VERBOSITY}
-export KUBECONFIG=${HOME}/admin.conf
-mkdir -p /tmp/kind
-sudo chmod 777 /tmp/kind
-count=0
-until kubectl get secrets -o jsonpath='{.items[].data.ca\.crt} &> /dev/null'
-do
-  if [ $count -gt 10 ]; then
-    echo "Failed to get k8s crt/token"
-    exit 1
-  fi
-  count=$((count+1))
-  echo "secrets not available on attempt $count"
-  sleep 5
-done
-kubectl get secrets -o jsonpath='{.items[].data.ca\.crt}' > /tmp/kind/ca.crt
-kubectl get secrets -o jsonpath='{.items[].data.token}' > /tmp/kind/token
+kind create cluster --name ovn --image kindest/node:${K8S_VERSION} --config=${KIND_CONFIG} -v ${OVN_KIND_VERBOSITY}
 
 echo -e "\n"
 
@@ -102,14 +104,15 @@ fi
 NODES=$(docker ps | grep "kindest/node" | awk '{ print $1 }')
 for n in $NODES; do
   echo "Modifying node $n"
-  echo "Copying kubeconfig"
-  docker cp ~/admin.conf $n:/etc/kubernetes/kubeconfig
-  docker exec $n chmod 777 /etc/kubernetes/kubeconfig
   echo "Modifying os-release for Multus"
   # required for Multus platform check
   docker exec $n sed -i 's/ID=.*/ID=rhcos/' /etc/os-release
 done
 
+# openshift-network-operator need read access to the kubeconfig
+# TODO: support multiple master nodes
+docker exec ovn-control-plane chmod 666 /etc/kubernetes/admin.conf
+
 # Create Proxy resource
 kubectl create -f https://raw.githubusercontent.com/openshift/api/e7fa4b871a25985ef0cc36c2fbd9f2cb4445dc9c/config/v1/0000_03_config-operator_01_proxy.crd.yaml
 
@@ -145,22 +148,9 @@ if [ "$BUILD_OVN" = true ] || [ "$BUILD_CNO" = true ]; then
   popd
 fi
 
-count=1
-until kubectl get pod -n openshift-network-operator -o jsonpath='{.items[0].metadata.name}' --field-selector status.phase=Running &> /dev/null ;do
-  if [ $count -gt 15 ]; then
-    echo "Network operator not running"
-    exit 1
-  fi
-  echo "Network operator pod not available yet on attempt $count"
-  count=$((count+1))
-  sleep 10
-
-done
-
-CNO_POD=$(kubectl get pod -n openshift-network-operator -o jsonpath='{.items[0].metadata.name}' --field-selector status.phase=Running)
-if [ -z "$CNO_POD" ]; then
-    echo "Cannot find running CNO pod"
-    exit 1
+if ! kubectl wait -n openshift-network-operator --for condition=available deployment network-operator --timeout=120s; then
+  echo "Network operator not running"
+  exit 1
 fi
 
 if [ "$BUILD_CNO" != true ]; then
@@ -192,25 +182,12 @@ metadata:
 spec:
   clusterNetwork:
   - cidr: ${CLUSTER_CIDR}
-    hostPrefix: 24
+    hostPrefix: ${HOST_PREFIX}
   networkType: OVNKubernetes
   serviceNetwork:
   - ${SERVICE_NETWORK}
 EOF
 
-count=1
-until kubectl get pod -n openshift-ovn-kubernetes -o jsonpath="{.items[0].status.phase}" 2> /dev/null | grep Running &> /dev/null;do
-  if [ $count -gt 15 ]; then
-    echo "OVN-k8s pods are not running"
-    exit 1
-  fi
-  echo "OVN pod not available yet on attempt $count"
-  count=$((count+1))
-  sleep 10
-done
-sleep 10
-
-
 for n in $NODES; do
   echo "Sym-linking cni dirs for node $n"
   docker exec $n rm -rf /opt/cni/bin
@@ -220,4 +197,20 @@ for n in $NODES; do
   docker exec $n ln -s /etc/kubernetes/cni/net.d /etc/cni/
 done
 
+# wait until resources are created
+sleep 30
+
+if ! kubectl wait -n openshift-ovn-kubernetes --for=condition=ready pods --all --timeout=300s ; then
+  echo "OVN-k8s pods are not running"
+  exit 1
+fi
+
+# Configuring secret for multus-admission-webhook
+# https://raw.githubusercontent.com/openshift/multus-admission-controller/master/hack/webhook-create-signed-cert.sh
+$CNO_PATH/hack/webhook-create-signed-cert.sh --service multus-admission-controller --namespace openshift-multus --secret multus-admission-controller-secret
+if ! kubectl wait -n openshift-multus --for=condition=ready pods --all --timeout=300s ; then
+  echo "multus pods are not running"
+  exit 1
+fi
+
 echo "Deployment Complete!"