Skip to content

Commit

Permalink
CI: Handle GKE lock release in a trap
Browse files Browse the repository at this point in the history
We are interrupted by jenkins on timeouts, and this can happen while we
scale the cluster down. We now unlock in an exit trap and this should,
hopefully, leave the cluster reusable, if not well cleaned.

Signed-off-by: Ray Bejjani <ray@isovalent.com>
  • Loading branch information
raybejjani committed Apr 14, 2020
1 parent 976fad7 commit 4d60573
Showing 1 changed file with 12 additions and 6 deletions.
18 changes: 12 additions & 6 deletions test/gke/release-cluster.sh
@@ -1,6 +1,16 @@
#!/bin/bash

export KUBECONFIG=gke-kubeconfig
cluster=$(cat cluster-name)

# Create a function to unlock the cluster. We then execute this on script exit.
# This should occur even if the script is interrupted, by a jenkins timeout,
# for example.
unlock() {
echo "releasing cluster lock from $cluster"
kubectl annotate deployment lock lock-
}
trap unlock EXIT

# We leak istio pods for an unknown reason (these tests do cleanup). This may
# be related to timeouts or other failures. In any case, we delete them here to
Expand All @@ -13,12 +23,8 @@ echo "deleting terminating namespaces"
./delete-terminating-namespaces.sh

set -e

cluster=$(cat cluster-name)

echo "scaling $cluster ng to 0"
yes | gcloud container clusters resize $cluster --node-pool default-pool --num-nodes 0 --zone $GKE_ZONE

echo "releasing cluster lock from $cluster"
kubectl annotate deployment lock lock-

rm cluster-name
rm -f cluster-name

0 comments on commit 4d60573

Please sign in to comment.