Add integration tests for grafana deployment #1156

ggriffiths · 2023-07-20T23:26:12Z

What this PR does / why we need it:
Adds a few tests to the operator integration test suite for validating grafana deployments, svcs, configmaps, and airgapped support.

Which issue(s) this PR fixes (optional)
Closes #

Special notes for your reviewer:

ggriffiths · 2023-07-20T23:42:06Z

Ran the test suite a few times, works as expected:

--- PASS: TestGrafana (173.22s)
    --- PASS: TestGrafana/GrafanaInstall (173.15s)
    --- PASS: TestGrafana/GrafanaInstallAirgapped (0.07s)
PASS
==================================================
--- PASS:
    GrafanaInstall [XXX XXX]
    GrafanaInstallAirgapped [XXX XXX]
--- SKIP:
--- FAIL:
==================================================

DONE 3 tests in 173.491s
Tests passed

nikolaypopov

Some changes are needed here, let me know if you want to sync up

pkg/util/test/util.go

ggriffiths · 2023-07-25T20:09:32Z

Tested again after addressing comments:

time="2023-07-25T19:53:35Z" level=info msg="Using version extracted from image name: 2.13.6"
time="2023-07-25T19:53:35Z" level=info msg="Update StorageCluster px-test-grafana in kube-system"
2023/07/25 19:53:35 grafana is not installed with image docker.io/grafana/grafana:8.5.27. number of pods: 0 Next retry in: 10s
time="2023-07-25T19:53:45Z" level=info msg="Grafana installed successfully with image docker.io/grafana/grafana:8.5.27"
--- PASS: TestGrafana (158.30s)
    --- PASS: TestGrafana/GrafanaInstall (143.24s)
    --- PASS: TestGrafana/GrafanaInstallVersionManifest (15.06s)
PASS
==================================================
--- PASS:
    GrafanaInstall [XXX XXX]
    GrafanaInstallVersionManifest [XXX XXX]
--- SKIP:
--- FAIL:
==================================================

DONE 3 tests in 158.637s
Tests passed

pkg/util/test/util.go

nikolaypopov

Overall looks much better, please address comments and build and image so I can run it in one of the basic jobs

nikolaypopov · 2023-07-26T16:34:49Z

pkg/util/test/util.go

+func ValidateGrafanaDeploymentImage(image string) error {
+	check := func() (interface{}, bool, error) {
+		pods, err := coreops.Instance().GetPods("kube-system", map[string]string{"app": "grafana"})
+		if err != nil {
+			return "", true, err
+		}
+
+		podCount := len(pods.Items)
+		if podCount > 0 && len(pods.Items[0].Spec.Containers) > 0 {
+			actualImage := pods.Items[0].Spec.Containers[0].Image
+			if actualImage == image {
+				logrus.Infof("Grafana installed successfully with image %s", image)
+				return "", false, nil
+			} else {
+				logrus.Errorf("Grafana not yet ready with image %s", image)
+				return "", true, fmt.Errorf("grafana is not installed with expected image %s. actual: %s", image, actualImage)
+			}
+		} else {
+			return "", true, fmt.Errorf("grafana is not installed with image %s. number of pods: %v", image, podCount)
+		}
+	}
+
+	if _, err := task.DoRetryWithTimeout(check, 1*time.Minute, 10*time.Second); err != nil {
+		return err
+	}
+
+	return nil
+}


Remove this since you using validateContainerImageInsidePods() and we don't need this any more

nikolaypopov · 2023-07-26T16:36:30Z

pkg/util/test/util.go

+
+func ValidateGrafanaDeployment(cluster *corev1.StorageCluster, shouldBeInstalled bool, pxImageList map[string]string) error {
+	check := func() (interface{}, bool, error) {
+		pods, err := coreops.Instance().GetPods("kube-system", map[string]string{"app": "grafana"})


Don't hardcode kube-system, this can be installed into any namespace, let's get namespace from cluster.Namespace and use that instead

nikolaypopov · 2023-07-26T16:38:28Z

pkg/util/test/util.go

+			if len(pods.Items) == 0 || pods.Items[0].Status.Phase != v1.PodRunning {
+				logrus.Errorf("Grafana not yet ready")
+				return "", true, fmt.Errorf("grafana is not installed when it should be")
+			}


Is grafana a deployment? Maybe just using sched-ops function ValidateDeployment should be enough here as it would do all the pod validation, if its not and its just a pod, this should be ok

nikolaypopov · 2023-07-26T16:40:40Z

pkg/util/test/util.go

+				logrus.Errorf("Grafana image is invalid yet ready")
+				return "", true, fmt.Errorf("grafana is not installed when it should be")


Let's not log error and then return it, let's just return verbose error instead

nikolaypopov · 2023-07-26T16:42:42Z

pkg/util/test/util.go

+				logrus.Errorf("Grafana not yet uninstalled")
+				return "", true, fmt.Errorf("grafana is installed when it is expected to be uninstalled")


Same here and all other place you have this

nikolaypopov

LGTM, let's run op-basic job once with this private image before merging. Let me know when you have an image ready

ggriffiths · 2023-07-26T20:16:08Z

Thanks, sent the test image.

Signed-off-by: Grant Griffiths <ggriffiths@purestorage.com>

github-actions bot added area/integration-test size/m labels Jul 20, 2023

ggriffiths requested review from nikolaypopov and a team July 20, 2023 23:35

github-actions bot added the stage/in-review label Jul 20, 2023

ggriffiths force-pushed the grafana_integration_test branch 4 times, most recently from ed0168d to baac997 Compare July 24, 2023 23:43

nikolaypopov reviewed Jul 25, 2023

View reviewed changes

pkg/util/test/util.go Outdated Show resolved Hide resolved

pkg/util/test/util.go Outdated Show resolved Hide resolved

pkg/util/test/util.go Outdated Show resolved Hide resolved

pkg/util/test/util.go Outdated Show resolved Hide resolved

pkg/util/test/util.go Outdated Show resolved Hide resolved

ggriffiths force-pushed the grafana_integration_test branch 2 times, most recently from 6fbadde to 3fc01a0 Compare July 25, 2023 19:54

ggriffiths requested a review from nikolaypopov July 25, 2023 20:08

nikolaypopov reviewed Jul 25, 2023

View reviewed changes

pkg/util/test/util.go Outdated Show resolved Hide resolved

ggriffiths force-pushed the grafana_integration_test branch 3 times, most recently from dff6bce to fab127e Compare July 26, 2023 00:15

nikolaypopov reviewed Jul 26, 2023

View reviewed changes

nikolaypopov approved these changes Jul 26, 2023

View reviewed changes

ggriffiths force-pushed the grafana_integration_test branch 7 times, most recently from 4039639 to 032506a Compare July 27, 2023 18:43

nikolaypopov self-requested a review July 27, 2023 22:01

ggriffiths force-pushed the grafana_integration_test branch 3 times, most recently from e85c1e9 to c6efb64 Compare July 28, 2023 00:00

github-actions bot added the area/vendor label Jul 28, 2023

ggriffiths force-pushed the grafana_integration_test branch 2 times, most recently from cf214db to 98eda84 Compare July 28, 2023 01:34

nikolaypopov approved these changes Jul 28, 2023

View reviewed changes

Grant Griffiths added 9 commits July 28, 2023 01:44

Add integration tests for grafana deployment

cad072a

Signed-off-by: Grant Griffiths <ggriffiths@purestorage.com>

PR Feedback

1794434

Signed-off-by: Grant Griffiths <ggriffiths@purestorage.com>

Refactor test to basic_test.go

532b175

Signed-off-by: Grant Griffiths <ggriffiths@purestorage.com>

PR feedback

531c315

Signed-off-by: Grant Griffiths <ggriffiths@purestorage.com>

Handle test conflict cases

225bf54

Signed-off-by: Grant Griffiths <ggriffiths@purestorage.com>

skip for op 23.7 and earlier

2c4b73f

Signed-off-by: Grant Griffiths <ggriffiths@purestorage.com>

add skip log line

c561fd6

Signed-off-by: Grant Griffiths <ggriffiths@purestorage.com>

Add test debugging

baf4b36

Signed-off-by: Grant Griffiths <ggriffiths@purestorage.com>

Check for grafana running properly

7e6da52

Signed-off-by: Grant Griffiths <ggriffiths@purestorage.com>

ggriffiths force-pushed the grafana_integration_test branch from 98eda84 to 7e6da52 Compare July 28, 2023 01:44

ggriffiths merged commit 8078b73 into libopenstorage:master Jul 28, 2023
7 checks passed

ggriffiths deleted the grafana_integration_test branch July 28, 2023 04:01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add integration tests for grafana deployment #1156

Add integration tests for grafana deployment #1156

ggriffiths commented Jul 20, 2023

ggriffiths commented Jul 20, 2023

nikolaypopov left a comment

ggriffiths commented Jul 25, 2023

nikolaypopov left a comment

nikolaypopov Jul 26, 2023

nikolaypopov Jul 26, 2023

ggriffiths Jul 26, 2023

nikolaypopov Jul 26, 2023

nikolaypopov Jul 26, 2023

nikolaypopov Jul 26, 2023

nikolaypopov left a comment

ggriffiths commented Jul 26, 2023

		logrus.Errorf("Grafana image is invalid yet ready")
		return "", true, fmt.Errorf("grafana is not installed when it should be")

		logrus.Errorf("Grafana not yet uninstalled")
		return "", true, fmt.Errorf("grafana is installed when it is expected to be uninstalled")

Add integration tests for grafana deployment #1156

Add integration tests for grafana deployment #1156

Conversation

ggriffiths commented Jul 20, 2023

ggriffiths commented Jul 20, 2023

nikolaypopov left a comment

Choose a reason for hiding this comment

ggriffiths commented Jul 25, 2023

nikolaypopov left a comment

Choose a reason for hiding this comment

nikolaypopov Jul 26, 2023

Choose a reason for hiding this comment

nikolaypopov Jul 26, 2023

Choose a reason for hiding this comment

ggriffiths Jul 26, 2023

Choose a reason for hiding this comment

nikolaypopov Jul 26, 2023

Choose a reason for hiding this comment

nikolaypopov Jul 26, 2023

Choose a reason for hiding this comment

nikolaypopov Jul 26, 2023

Choose a reason for hiding this comment

nikolaypopov left a comment

Choose a reason for hiding this comment

ggriffiths commented Jul 26, 2023