Skip to content

Commit

Permalink
test: Run kubectl in test VM for older K8s releases
Browse files Browse the repository at this point in the history 
kubectl is guaranteed to be compatible with limited number of earlier
releases:

> kubectl is supported within one minor version (older or newer) of
> kube-apiserver.

So far we have been using the latest kubectl in the host (now 1.18) to
control clusters from K8s 1.11 to 1.18. This did not work any more
with 'istioctl', which complained about "kubectl not being found in
$PATH". When pairing istioctl with the same version of kubectl as the
cluster this started working again.

Downgrading kubectl in the test hosts may not be practical, but the CI
infra also supports running kubectl in the cluster's master node
(k8s1). This is triggered via the value of the Ginkgo
'cilium.kubeconfig' option. When 'cilium.kubeconfig' is non-empty, it
is assumed to be a path to a valid kubeconfig for connecting kubectl
in the host to the test cluster. When 'cilium.kubeconfig' is empty, CI
Ginkgo helpers assume that kubectl should be run on
"k8s1". 'test/vagrant-ci-start.sh' expects KUBECONFIG environment
variable to be set to the path to the file into which the kubeconfig
fetched from the test cluster should be stored. Modify
'test/vagrant-ci-start.sh' to accept undefined KUBECONFIG to signify
the need to run kubectl in the test cluster's master node (k8s1).

Finally, remove setting of both the KUBECONFIG environment variable
before calling 'vagrant-ci-start.sh' and the Ginkgo option
'cilium.kubeconfig' in 'ginkgo-kubernetes-all.Jenkinsfile' which is
used to run the CI K8s test suite on k8s versions from 1.11 to
1.17. This way we always use the kubectl installed as part of the
testing cluster itself, in the master node. This solves the
compatibility problem with istioctl and should help guard that we have
not introduced any kubectl syntax that would not be compatible with
the target k8s version.

Signed-off-by: Jarno Rajahalme <jarno@covalent.io>
  • Loading branch information
@jrajahalme @nebril
jrajahalme authored and nebril committed Apr 27, 2020
1 parent 29f0b34 commit 5cdde10
Show file tree
Hide file tree
Showing 2 changed files with 19 additions and 32 deletions.
30 changes: 6 additions & 24 deletions jenkinsfiles/ginkgo-kubernetes-all.Jenkinsfile
View file
Original file line number Diff line number Diff line change
Expand Up @@ -121,8 +121,6 @@ pipeline {
K8S_VERSION="1.12"
GOPATH="${WORKSPACE}/${K8S_VERSION}-gopath"
TESTDIR="${GOPATH}/${PROJ_PATH}/test"
KUBECONFIG="vagrant-kubeconfig"
//setting it here so we don't compile Cilium in vagrant nodes (already done on local node registry)
}
steps {
sh 'mkdir -p ${GOPATH}/src/github.com/cilium'
Expand Down Expand Up @@ -150,8 +148,6 @@ pipeline {
K8S_VERSION="1.13"
GOPATH="${WORKSPACE}/${K8S_VERSION}-gopath"
TESTDIR="${GOPATH}/${PROJ_PATH}/test"
KUBECONFIG="vagrant-kubeconfig"
//setting it here so we don't compile Cilium in vagrant nodes (already done on local node registry)
}
steps {
sh 'mkdir -p ${GOPATH}/src/github.com/cilium'
Expand Down Expand Up @@ -189,10 +185,9 @@ pipeline {
K8S_VERSION="1.12"
GOPATH="${WORKSPACE}/${K8S_VERSION}-gopath"
TESTDIR="${GOPATH}/${PROJ_PATH}/test"
KUBECONFIG="${TESTDIR}/vagrant-kubeconfig"
}
steps {
sh 'cd ${TESTDIR}; ginkgo --focus=" K8s*" -v --failFast=${FAILFAST} -- -cilium.provision=false -cilium.timeout=${GINKGO_TIMEOUT} -cilium.kubeconfig=${KUBECONFIG} -cilium.registry=$(./print-node-ip.sh)'
sh 'cd ${TESTDIR}; ginkgo --focus=" K8s*" -v --failFast=${FAILFAST} -- -cilium.provision=false -cilium.timeout=${GINKGO_TIMEOUT} -cilium.registry=$(./print-node-ip.sh)'
}
post {
always {
Expand All @@ -216,10 +211,9 @@ pipeline {
K8S_VERSION="1.13"
GOPATH="${WORKSPACE}/${K8S_VERSION}-gopath"
TESTDIR="${GOPATH}/${PROJ_PATH}/test"
KUBECONFIG="${TESTDIR}/vagrant-kubeconfig"
}
steps {
sh 'cd ${TESTDIR}; ginkgo --focus=" K8s*" -v --failFast=${FAILFAST} -- -cilium.provision=false -cilium.timeout=${GINKGO_TIMEOUT} -cilium.kubeconfig=${KUBECONFIG} -cilium.registry=$(./print-node-ip.sh)'
sh 'cd ${TESTDIR}; ginkgo --focus=" K8s*" -v --failFast=${FAILFAST} -- -cilium.provision=false -cilium.timeout=${GINKGO_TIMEOUT} -cilium.registry=$(./print-node-ip.sh)'
}
post {
always {
Expand Down Expand Up @@ -258,8 +252,6 @@ pipeline {
K8S_VERSION="1.14"
GOPATH="${WORKSPACE}/${K8S_VERSION}-gopath"
TESTDIR="${GOPATH}/${PROJ_PATH}/test"
KUBECONFIG="vagrant-kubeconfig"
//setting it here so we don't compile Cilium in vagrant nodes (already done on local node registry)
}
steps {
sh 'mkdir -p ${GOPATH}/src/github.com/cilium'
Expand Down Expand Up @@ -287,8 +279,6 @@ pipeline {
K8S_VERSION="1.15"
GOPATH="${WORKSPACE}/${K8S_VERSION}-gopath"
TESTDIR="${GOPATH}/${PROJ_PATH}/test"
KUBECONFIG="vagrant-kubeconfig"
//setting it here so we don't compile Cilium in vagrant nodes (already done on local node registry)
}
steps {
sh 'mkdir -p ${GOPATH}/src/github.com/cilium'
Expand Down Expand Up @@ -326,10 +316,9 @@ pipeline {
K8S_VERSION="1.14"
GOPATH="${WORKSPACE}/${K8S_VERSION}-gopath"
TESTDIR="${GOPATH}/${PROJ_PATH}/test"
KUBECONFIG="${TESTDIR}/vagrant-kubeconfig"
}
steps {
sh 'cd ${TESTDIR}; ginkgo --focus=" K8s*" -v --failFast=${FAILFAST} -- -cilium.provision=false -cilium.timeout=${GINKGO_TIMEOUT} -cilium.kubeconfig=${KUBECONFIG} -cilium.registry=$(./print-node-ip.sh)'
sh 'cd ${TESTDIR}; ginkgo --focus=" K8s*" -v --failFast=${FAILFAST} -- -cilium.provision=false -cilium.timeout=${GINKGO_TIMEOUT} -cilium.registry=$(./print-node-ip.sh)'
}
post {
always {
Expand All @@ -353,10 +342,9 @@ pipeline {
K8S_VERSION="1.15"
GOPATH="${WORKSPACE}/${K8S_VERSION}-gopath"
TESTDIR="${GOPATH}/${PROJ_PATH}/test"
KUBECONFIG="${TESTDIR}/vagrant-kubeconfig"
}
steps {
sh 'cd ${TESTDIR}; ginkgo --focus=" K8s*" -v --failFast=${FAILFAST} -- -cilium.provision=false -cilium.timeout=${GINKGO_TIMEOUT} -cilium.kubeconfig=${KUBECONFIG} -cilium.registry=$(./print-node-ip.sh)'
sh 'cd ${TESTDIR}; ginkgo --focus=" K8s*" -v --failFast=${FAILFAST} -- -cilium.provision=false -cilium.timeout=${GINKGO_TIMEOUT} -cilium.registry=$(./print-node-ip.sh)'
}
post {
always {
Expand Down Expand Up @@ -394,8 +382,6 @@ pipeline {
K8S_VERSION="1.16"
GOPATH="${WORKSPACE}/${K8S_VERSION}-gopath"
TESTDIR="${GOPATH}/${PROJ_PATH}/test"
KUBECONFIG="vagrant-kubeconfig"
//setting it here so we don't compile Cilium in vagrant nodes (already done on local node registry)
}
steps {
sh 'mkdir -p ${GOPATH}/src/github.com/cilium'
Expand Down Expand Up @@ -423,8 +409,6 @@ pipeline {
K8S_VERSION="1.17"
GOPATH="${WORKSPACE}/${K8S_VERSION}-gopath"
TESTDIR="${GOPATH}/${PROJ_PATH}/test"
KUBECONFIG="vagrant-kubeconfig"
//setting it here so we don't compile Cilium in vagrant nodes (already done on local node registry)
}
steps {
sh 'mkdir -p ${GOPATH}/src/github.com/cilium'
Expand Down Expand Up @@ -462,10 +446,9 @@ pipeline {
K8S_VERSION="1.16"
GOPATH="${WORKSPACE}/${K8S_VERSION}-gopath"
TESTDIR="${GOPATH}/${PROJ_PATH}/test"
KUBECONFIG="${TESTDIR}/vagrant-kubeconfig"
}
steps {
sh 'cd ${TESTDIR}; ginkgo --focus=" K8s*" -v --failFast=${FAILFAST} -- -cilium.provision=false -cilium.timeout=${GINKGO_TIMEOUT} -cilium.kubeconfig=${KUBECONFIG} -cilium.registry=$(./print-node-ip.sh)'
sh 'cd ${TESTDIR}; ginkgo --focus=" K8s*" -v --failFast=${FAILFAST} -- -cilium.provision=false -cilium.timeout=${GINKGO_TIMEOUT} -cilium.registry=$(./print-node-ip.sh)'
}
post {
always {
Expand All @@ -489,10 +472,9 @@ pipeline {
K8S_VERSION="1.17"
GOPATH="${WORKSPACE}/${K8S_VERSION}-gopath"
TESTDIR="${GOPATH}/${PROJ_PATH}/test"
KUBECONFIG="${TESTDIR}/vagrant-kubeconfig"
}
steps {
sh 'cd ${TESTDIR}; ginkgo --focus=" K8s*" -v --failFast=${FAILFAST} -- -cilium.provision=false -cilium.timeout=${GINKGO_TIMEOUT} -cilium.kubeconfig=${KUBECONFIG} -cilium.registry=$(./print-node-ip.sh)'
sh 'cd ${TESTDIR}; ginkgo --focus=" K8s*" -v --failFast=${FAILFAST} -- -cilium.provision=false -cilium.timeout=${GINKGO_TIMEOUT} -cilium.registry=$(./print-node-ip.sh)'
}
post {
always {
Expand Down
21 changes: 13 additions & 8 deletions test/vagrant-ci-start.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,25 +17,30 @@ for i in $(seq 1 $K8S_NODES); do
vagrant up k8s${i}-${K8S_VERSION} --provision
done

echo "getting vagrant kubeconfig from provisioned vagrant cluster"
./get-vagrant-kubeconfig.sh > vagrant-kubeconfig
if [ -n "${KUBECONFIG}" ]; then
echo "getting vagrant kubeconfig from provisioned vagrant cluster"
./get-vagrant-kubeconfig.sh > ${KUBECONFIG}
KUBECTL="kubectl"
else
echo "using vagrant ssh k8s1-${K8S_VERSION} for kubectl"
# No kube config, run kubectl in k8s1 via ssh
KUBECTL="vagrant ssh k8s1-${K8S_VERSION} -- kubectl"
fi

echo "checking whether kubeconfig works for vagrant cluster"

NEXT_WAIT_TIME=0
until kubectl get nodes || [ $NEXT_WAIT_TIME -eq 12 ]; do
until ${KUBECTL} get nodes || [ $NEXT_WAIT_TIME -eq 12 ]; do
((NEXT_WAIT_TIME++))
sleep 5
done

export HOME=${GOPATH}
kubectl get nodes
${KUBECTL} get nodes

echo "adding local docker registry to cluster"
helm template registry-adder k8sT/manifests/registry-adder --set IP="$(./print-node-ip.sh)" > registry-adder.yaml
kubectl apply -f registry-adder.yaml
helm template registry-adder k8sT/manifests/registry-adder --set IP="$(./print-node-ip.sh)" | ${KUBECTL} apply -f -

echo "labeling nodes"
for i in $(seq 1 $K8S_NODES); do
kubectl label node k8s${i} cilium.io/ci-node=k8s${i}
${KUBECTL} label node k8s${i} cilium.io/ci-node=k8s${i} --overwrite
done

0 comments on commit 5cdde10

Please sign in to comment.