Adding canary 'monitor' mode. #241

templecloud · 2018-09-06T13:46:05Z

The canary team requested some changes to the CCM (and volume-provisioner and flexvolume-driver) canary. The requested changes were:

Provide a mode where the canary test container entrypoint itself runs in an infinite loop and executes the test on each invocation.
The entry point is made available on path: '/oci/scripts/ccm-canary-entrypoint.sh'

This commit attempts to provide this functionality. The default behaviour of the ccm-canary is now to run in an infinite loop with a pause of 30s before each invocation of the ccm e2e canary test. The pause period can be configured via the 'MONITOR_PERIOD' environment variable. The 'mode' can be configured via the 'CANARY_MODE' ('monitor'|'run-once') environment variable.

The canary validation test has been altered to test the result from the canary for 'CANARY_RUNS' invocations - configured via an environment variable in the CI or locally for dev.

templecloud · 2018-09-06T13:50:54Z

Makefile


-# Validate the generated canary test image.
+# Run the canary tests - in monitor (infinite loop) mode.
+.PHONY: canary-monitor


New monitor mode.

templecloud · 2018-09-06T13:51:04Z

Makefile

-canary:
-	@./hack/test-canary.sh
+# Run the canary tests - in single run mode.
+.PHONY: canary-run-once


Original mode renamed.

templecloud · 2018-09-06T13:54:13Z

hack/ccm-canary-entrypoint.sh

@@ -0,0 +1,29 @@
+#!/bin/bash


This script is for the new entrypoint. The easiest solution (rather than mess with GOPATH) is for the entrypoint script to move to the original project root and execute the canary test from there as before.

templecloud · 2018-09-06T13:54:37Z

hack/test-canary.sh

        -focus "\[Canary\]" \
        test/e2e \
-        -- --kubeconfig=${KUBECONFIG} --delete-namespace=false \
+        -- --kubeconfig=${KUBECONFIG} --delete-namespace=true \


This is a better default for the canary tests.

templecloud · 2018-09-06T13:55:45Z

hack/test-canary.sh

 function clean() {
-    kubectl get pods --all-namespaces | grep ccm | awk '{print $1}' | xargs kubectl delete ns
-    rm "${TEST_DIR}/${TEST_PREFIX}*"
+    echo "ensuring fresh \$START."


In monitor mode we no longer run once and exit - so we need a function to clean up resources between each iteration.

templecloud · 2018-09-06T13:59:39Z

hack/test-canary.sh

+}
+
+# Run the tests in loop with the specified wait period.
+function monitor() {


The new 'monitor' method just sits in an infinite loop and calls the original 'run-once' method on each iteration. It also sleeps for the specified period. This will give a window of opportunity for any sidecars to grab the results from filesystem. They will be deleted on the next iteration of the tests and the current results lost.

templecloud · 2018-09-06T14:01:59Z

hack/validate-canary.sh

-        value: /metrics/output.json
-      - name: KUBECONFIG_VAR
-        value: $(cat ${KUBECONFIG} | openssl enc -base64 -A)
+    - name: KUBECONFIG_VAR


The validation test also shows expected usage of the canary in a pod.

templecloud · 2018-09-06T14:04:11Z

hack/validate-canary.sh

    image: iad.ocir.io/oracle/oci-cloud-controller-manager-ci-e2e:1.0.1
    command: ["/bin/bash"]
-    args: ["-ec", "touch \$METRICS_FILE; while [ -z \$(cat \$METRICS_FILE | grep 'end_time' | cut -d':' -f 1) ]; do sleep 1;  done; cat \$METRICS_FILE"]
+    args: ["-ec", "while true; do sleep 10; cat \$METRICS_FILE; done"]


The canary validation tests now has a dumb reporter that just sits in an infinite loop and cats the contents of the metrics file to std out.

templecloud · 2018-09-06T14:10:39Z

hack/validate-canary.sh

-    else
-        exit 0
-    fi
+


The validation test itself is a little more involved now. We wait for canary to run CANARY_RUNS times. If we have any failures we consider it FAILED. If we have all successes then PASSED. On completion we destroy the canary pod.

We let the CI destroy long running tests. This should also destroy any running canaries.

templecloud · 2018-09-06T14:11:44Z

test/e2e/framework/framework.go

 	f.cleanupHandle = AddCleanupAction(f.AfterEach)

-	if f.Client == nil {
+	// Create an OCI client if the cloudConfig has been specified.


A series of tweaks to allow users to avoid specifying the cloud config if not required. The canary runner might not want to do this.

templecloud · 2018-09-06T14:13:01Z

wercker.yml

    - script:
      name: validate canary tests
-      code: make validate-canary
+      code: make validate-canary 


Whoops will get rid of that trailing ws...

templecloud · 2018-09-06T14:13:34Z

wercker.yml

-
+    - script:
+        name: configure oci canary entrypoint
+        code: mkdir -p /oci/scripts/ && cp ./hack/ccm-canary-entrypoint.sh /oci/scripts/ccm-canary-entrypoint.sh 


We now mount in the new entry point into the container.

templecloud · 2018-09-06T14:46:24Z

hack/validate-canary.sh

    - mountPath: /metrics
      name: metrics-volume
-  
+      


kristenjacobs

All looks good to me.

kristenjacobs · 2018-09-07T09:43:25Z

hack/validate-canary.sh

+            local num_pass=$(echo "${logs}"| grep '"create_lb": "1"' | uniq | wc -l)
+            local num_fail=$(echo "${logs}"| grep '"create_lb": "0"' | uniq | wc -l)
+            if [ "${num_fail}" -gt "0" ]; then 
+                echo "FAILED"


Maybe dump the test logs here for the failure case, i.e. it might make debugging a failure easier?

owainlewis · 2018-09-07T14:53:54Z

Currently failing E2E tests so can't merge.

templecloud · 2018-09-07T18:14:51Z

Thanks Owain - I should have been monitoring this. The e2e test 'failed to provision a load balancer within 20m'. We only have 3 (100MB) load balancers available in the the test tenancy region (spinnaker / IAD). I had a look and there were some sick ones sitting around in the 'general' compartment which I deleted.

I ran the test again with no code changes and it passed second time around - so I am going to put it down to it correctly failing due to having no load balancers available.

I dont know if there is anything we can do to alleviate this issue in future?

owainlewis · 2018-09-10T08:10:18Z

hack/ccm-canary-entrypoint.sh

+    export CANARY_MODE="monitor" 
+fi
+
+# "/go/src/github.com/oracle/oci-cloud-controller-manager"


Comment can be removed

owainlewis · 2018-09-10T08:11:49Z

hack/ccm-canary-entrypoint.sh

+fi
+
+# "/go/src/github.com/oracle/oci-cloud-controller-manager"
+export CCM_DIR="${GOPATH}/src/github.com/oracle/oci-cloud-controller-manager"


Could just simply:

pushd "${GOPATH}/src/github.com/oracle/oci-cloud-controller-manager"

owainlewis

Minor comments. OK to merge.

… to allow the test pods to commuicate with worker nodes. This fixes that issues by making the nodePort portion of the tests configurable.

Adding canary monitor mode.

Adding canary 'monitor' node.

0fff8e6

templecloud requested review from owainlewis and prydie September 6, 2018 13:46

templecloud commented Sep 6, 2018

View reviewed changes

hack/validate-canary.sh

- mountPath: /metrics

name: metrics-volume

Copy link

Contributor Author

templecloud Sep 6, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Whoops.

templecloud requested a review from kristenjacobs September 6, 2018 15:00

kristenjacobs approved these changes Sep 7, 2018

View reviewed changes

Added review change to print out error log on failure.

0dc45a1

templecloud changed the title ~~Adding canary 'monitor' node.~~ Adding canary 'monitor' mode. Sep 7, 2018

owainlewis reviewed Sep 10, 2018

View reviewed changes

owainlewis approved these changes Sep 10, 2018

View reviewed changes

templecloud added 3 commits September 11, 2018 11:33

PR requested changes.

0e9dae9

Fixed bad JSON formating.

eb981a7

By default OKE clusters do not have required the ingress seclist rule…

cc2c230

… to allow the test pods to commuicate with worker nodes. This fixes that issues by making the nodePort portion of the tests configurable.

owainlewis removed the request for review from prydie September 14, 2018 08:28

owainlewis added the testing label Sep 14, 2018

owainlewis merged commit b36103d into master Sep 14, 2018

owainlewis deleted the trjl/infinite-canary branch September 14, 2018 08:32

ayushverma14 pushed a commit to ayushverma14/oci-cloud-controller-manager that referenced this pull request Jul 18, 2022

Adding canary 'monitor' mode. (oracle#241)

e3b3635

Adding canary monitor mode.

Adding canary 'monitor' mode. #241

Adding canary 'monitor' mode. #241

Uh oh!

Conversation

templecloud commented Sep 6, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

templecloud Sep 6, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

templecloud Sep 6, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

templecloud Sep 6, 2018 • edited by owainlewis Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

templecloud Sep 6, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kristenjacobs left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

owainlewis commented Sep 7, 2018

Uh oh!

templecloud commented Sep 7, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

owainlewis left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

templecloud commented Sep 6, 2018 •

edited

Loading

templecloud Sep 6, 2018 •

edited

Loading

templecloud Sep 6, 2018 •

edited

Loading

templecloud Sep 6, 2018 •

edited by owainlewis

Loading

templecloud Sep 6, 2018 •

edited

Loading

templecloud commented Sep 7, 2018 •

edited

Loading