CSPL-949: Refactor of secret test cases #298

pdhanoya-splunk · 2021-03-26T05:37:29Z

Refactored Secret Test Cases and added support for server.conf secret verification

Split test case's into individual files based on SVA
Created method to fetch Secret Object through API, eliminating need to encode/decode secrets
Added method to fetch splunk pods in a namespace
Added method to fetch versioned secrets in a namespace
Refactored Secret Verification Methods
Added additional verification for verifying shc_secret, idxc_secret, pass4SymmKey on Splunk Instance
Modified existing util to parse splunk config file based on stanza
Added commons methods to remove duplicated code in multiple test Cases
Marked one S1 test case as smoke test
Renamed some variable to match naming conventions used across Splunk Operator Code
Fixed a logic bug in mcutil.go

Passing Test Runs

S1 SVA: secret update is applied to standalone

[2] Secret Test for SVA S1
[2] /Users/pdhanoya/splunk-operator-git/automation/splunk-operator/test/secret/secret_s1_test.go:24
[2]   Standalone deployment (S1) with LM
[2]   /Users/pdhanoya/splunk-operator-git/automation/splunk-operator/test/secret/secret_s1_test.go:44
[2]     secret: Secret update on a standalone instance
[2]     /Users/pdhanoya/splunk-operator-git/automation/splunk-operator/test/secret/secret_s1_test.go:45
[2] ------------------------------
[2] {"level":"info","ts":1616737584.6970189,"msg":"testenv deleted.\n","testenv":"secret-dw2"}
[2] 
[2] JUnit report was created: /Users/pdhanoya/splunk-operator-git/automation/splunk-operator/test/secret/secret-dw2_junit.xml
[2] 
[2] Ran 1 of 1 Specs in 539.048 seconds
[2] SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 0 Skipped
[2] PASS

S1 SVA: Secret Object data is repopulated in secret object on passing empty Data map and new secrets are applied to Splunk Pods

[2] Secret Test for SVA S1
[2] /Users/pdhanoya/splunk-operator-git/automation-test/splunk-operator/test/secret/secret_s1_test.go:26
[2]   Standalone deployment (S1)
[2]   /Users/pdhanoya/splunk-operator-git/automation-test/splunk-operator/test/secret/secret_s1_test.go:205
[2]     secret,smoke: Secret Object data is repopulated in secret object on passing empty Data map and new secrets are applied to Splunk Pods
[2]     /Users/pdhanoya/splunk-operator-git/automation-test/splunk-operator/test/secret/secret_s1_test.go:206
[2] ------------------------------
[2] {"level":"info","ts":1616706688.443281,"msg":"testenv deleted.\n","testenv":"secret-vj"}
[2] 
[2] JUnit report was created: /Users/pdhanoya/splunk-operator-git/automation-test/splunk-operator/test/secret/secret-vj_junit.xml
[2] 
[2] Ran 1 of 2 Specs in 388.682 seconds
[2] SUCCESS! -- 1 Passed | 0 Failed | 1 Pending | 0 Skipped
[2] PASS

Secret Object is recreated on delete and new secrets are applied to Splunk Pods

[1] • [SLOW TEST:555.138 seconds]
[1] Secret Test for SVA S1
[1] /Users/pdhanoya/splunk-operator-git/automation-test/splunk-operator/test/secret/secret_s1_test.go:26
[1]   Standalone deployment (S1) with LM
[1]   /Users/pdhanoya/splunk-operator-git/automation-test/splunk-operator/test/secret/secret_s1_test.go:46
[1]     secret: Secret update on a standalone instance
[1]     /Users/pdhanoya/splunk-operator-git/automation-test/splunk-operator/test/secret/secret_s1_test.go:47
[1] ------------------------------
[1] {"level":"info","ts":1616708352.028694,"msg":"testenv deleted.\n","testenv":"secret-rn"}
[1] 
[1] JUnit report was created: /Users/pdhanoya/splunk-operator-git/automation-test/splunk-operator/test/secret/secret-rn_junit.xml
[1] 
[1] Ran 1 of 2 Specs in 562.198 seconds
[1] SUCCESS! -- 1 Passed | 0 Failed | 1 Pending | 0 Skipped
[1] PASS

Ginkgo ran 1 suite in 9m28.570545562s
Test Suite Passed

C3 SVA : secret update on indexers and search head cluster

[1] • [SLOW TEST:2026.156 seconds]
[1] secret test
[1] /Users/pdhanoya/splunk-operator-git/automation/splunk-operator/test/secret/secret_c3_test.go:24
[1]   Clustered deployment (C3 - clustered indexer, search head cluster)
[1]   /Users/pdhanoya/splunk-operator-git/automation/splunk-operator/test/secret/secret_c3_test.go:44
[1]     secret: secret update on indexers and search head cluster
[1]     /Users/pdhanoya/splunk-operator-git/automation/splunk-operator/test/secret/secret_c3_test.go:45
[1] ------------------------------
[1] {"level":"info","ts":1616731745.991369,"msg":"testenv deleted.\n","testenv":"secret-hpc"}
[1] 
[1] JUnit report was created: /Users/pdhanoya/splunk-operator-git/automation/splunk-operator/test/secret/secret-hpc_junit.xml
[1] 
[1] Ran 1 of 1 Specs in 2032.645 seconds
[1] SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 0 Skipped
[1] PASS

M4 SVA: secret update on multisite indexers and search head cluster

[2] Secret Test for M4 SVA
[2] /Users/pdhanoya/splunk-operator-git/automation/splunk-operator/test/secret/secret_m4_test.go:24
[2]   Multisite cluster deployment (M13 - Multisite indexer cluster, Search head cluster)
[2]   /Users/pdhanoya/splunk-operator-git/automation/splunk-operator/test/secret/secret_m4_test.go:44
[2]     secret: secret update on multisite indexers and search head cluster
[2]     /Users/pdhanoya/splunk-operator-git/automation/splunk-operator/test/secret/secret_m4_test.go:45
[2] ------------------------------
[2] {"level":"info","ts":1616736128.170295,"msg":"testenv deleted.\n","testenv":"secret-fal"}
[2] 
[2] JUnit report was created: /Users/pdhanoya/splunk-operator-git/automation/splunk-operator/test/secret/secret-fal_junit.xml
[2] 
[2] Ran 1 of 3 Specs in 2159.138 seconds
[2] SUCCESS! -- 1 Passed | 0 Failed | 2 Pending | 0 Skipped
[2] PASS

test/secret/secret_m4_test.go

akondur · 2021-03-26T23:32:33Z

.circleci/config.yml

            find ./test -name "*junit.xml" -exec cp {} /tmp/test-results \;
          environment: 
-            TEST_FOCUS: "smoke|ingest_search|monitoring_console|smartstore|licensemaster|scaling_test|crcrud|secret"
+            # TEST_FOCUS: "smoke|ingest_search|monitoring_console|smartstore|licensemaster|scaling_test|crcrud|secret"


Do we need the commented out line?

This is just for reference I will remove it in future.

akondur · 2021-03-26T23:34:08Z

.circleci/config.yml

          environment: 
-            TEST_FOCUS: "smoke|ingest_search|monitoring_console|smartstore|licensemaster|scaling_test|crcrud|secret"
+            # TEST_FOCUS: "smoke|ingest_search|monitoring_console|smartstore|licensemaster|scaling_test|crcrud|secret"
+            TEST_FOCUS: "integration"


Are we able to control which tests we can run with this change?

Yes we are coming up with keyword to define different test and this is a change in that direction. We should be able to run different test based on keyword.

akondur · 2021-03-26T23:36:06Z

test/ingest_search/ingest_search_test.go


 	Context("Standalone deployment (S1)", func() {
-		It("ingest_search: can search internal logs for standalone instance", func() {
+		It("ingest_search,integration: can search internal logs for standalone instance", func() {


Nit: ingest_search, integration an extra space between the ingest_search and integration?

Similarly in other occurrences below.

akondur · 2021-03-26T23:36:53Z

test/monitoring_console/monitoring_console_test.go


 	Context("Standalone deployment (S1)", func() {
-		It("monitoring_console: can deploy a MC with standalone instance and update MC with new standalone deployment", func() {
+		It("monitoring_console,integration: can deploy a MC with standalone instance and update MC with new standalone deployment", func() {


Nit: Same comment as above.

akondur · 2021-03-26T23:36:57Z

test/monitoring_console/monitoring_console_test.go


 	Context("Clustered deployment (C3 - clustered indexer, search head cluster)", func() {
-		It("monitoring_console: MC can configure SHC, indexer instances after scale up and standalone in a namespace", func() {
+		It("monitoring_console,integration: MC can configure SHC, indexer instances after scale up and standalone in a namespace", func() {


Nit: Same comment as above.

akondur · 2021-03-26T23:56:49Z

test/smartstore/smartstore_test.go

-	Context("Configure multiple indexes on standlaone deployment using CR Spec", func() {
-		It("smartstore: Can configure multiple indexes through app", func() {
+	Context("Standalone Deployment (S1)", func() {
+		It("smartstore,integration: Can configure multiple indexes through app", func() {


Nit: Same comment as above

akondur · 2021-03-26T23:57:31Z

test/smartstore/smartstore_test.go

-	Context("Confiugre volume in default settings on standlaone deployment using CR Spec", func() {
-		It("smartstore: Can configure indexes which use default volumes through app", func() {
+	Context("Standalone Deployment (S1)", func() {
+		It("smartstore,integration: Can configure indexes which use default volumes through app", func() {


Nit: Same comment as above

akondur · 2021-03-26T23:57:36Z

test/smartstore/smartstore_test.go

-	Context("Confiugre indexes and volumes on Indexer Cluster using CR Spec", func() {
-		It("smartstore: Can configure indexes and volumes on Multisite Indexer Cluster through app", func() {
+	Context("Multisite Indexer Cluster with Search Head Cluster (M4)", func() {
+		It("smartstore,integration: Can configure indexes and volumes on Multisite Indexer Cluster through app", func() {


Nit: Same comment as above

akondur · 2021-03-26T23:58:26Z

test/testenv/verificationutils.go

+func VerifySplunkServerConfSecrets(deployment *Deployment, testenvInstance *TestEnv, verificationPods []string, data map[string][]byte, match bool) {
+	for _, podName := range verificationPods {
+		keysToMatch := GetKeysToMatch(podName)
+		testenvInstance.Log.Info("Verificaton Keys Set", "Pod Name", podName, "Keys To Compare", keysToMatch)


Nit: Do we need the log for production?

Useful for DEBUG. Autogenerated secrets should not cause any security issues. Online test runs are behind secured credentials anyways.

test/testenv/util.go

…ightly integration runs

pdhanoya-splunk requested review from tpereirasplunk, akondur, jambrosiano and smohan-splunk March 26, 2021 05:38

pdhanoya-splunk force-pushed the CSPL-949-secret-server-conf-verification branch from da80a5a to 4e3ff93 Compare March 26, 2021 05:57

CSPL-949: Refacotor of secret test cases

91245ca

pdhanoya-splunk force-pushed the CSPL-949-secret-server-conf-verification branch 2 times, most recently from 9c16114 to d433760 Compare March 26, 2021 19:45

tpereirasplunk reviewed Mar 26, 2021

View reviewed changes

test/secret/secret_m4_test.go Outdated Show resolved Hide resolved

tpereirasplunk approved these changes Mar 26, 2021

View reviewed changes

pdhanoya-splunk force-pushed the CSPL-949-secret-server-conf-verification branch from d433760 to 1f78f8c Compare March 26, 2021 21:07

jambrosiano approved these changes Mar 26, 2021

View reviewed changes

smohan-splunk approved these changes Mar 26, 2021

View reviewed changes

akondur reviewed Mar 27, 2021

View reviewed changes

pdhanoya-splunk force-pushed the CSPL-949-secret-server-conf-verification branch from 1f78f8c to 81c7444 Compare March 27, 2021 00:21

Introduced integration keyword to reduce number of test executed on n…

2a9e978

…ightly integration runs

pdhanoya-splunk force-pushed the CSPL-949-secret-server-conf-verification branch from 81c7444 to 2a9e978 Compare March 27, 2021 00:25

smohan-splunk merged commit 0b34f28 into develop Mar 27, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

CSPL-949: Refactor of secret test cases #298

CSPL-949: Refactor of secret test cases #298

Uh oh!

pdhanoya-splunk commented Mar 26, 2021 •

edited

Loading

Uh oh!

Uh oh!

akondur Mar 26, 2021

Uh oh!

pdhanoya-splunk Mar 27, 2021

Uh oh!

akondur Mar 26, 2021

Uh oh!

pdhanoya-splunk Mar 27, 2021

Uh oh!

akondur Mar 26, 2021

Uh oh!

akondur Mar 26, 2021

Uh oh!

akondur Mar 26, 2021

Uh oh!

akondur Mar 26, 2021

Uh oh!

akondur Mar 26, 2021

Uh oh!

akondur Mar 26, 2021

Uh oh!

akondur Mar 26, 2021

Uh oh!

pdhanoya-splunk Mar 27, 2021

Uh oh!

Uh oh!

Uh oh!

CSPL-949: Refactor of secret test cases #298

CSPL-949: Refactor of secret test cases #298

Uh oh!

Conversation

pdhanoya-splunk commented Mar 26, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Refactored Secret Test Cases and added support for server.conf secret verification

Passing Test Runs

S1 SVA: secret update is applied to standalone

S1 SVA: Secret Object data is repopulated in secret object on passing empty Data map and new secrets are applied to Splunk Pods

Secret Object is recreated on delete and new secrets are applied to Splunk Pods

C3 SVA : secret update on indexers and search head cluster

M4 SVA: secret update on multisite indexers and search head cluster

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

pdhanoya-splunk commented Mar 26, 2021 •

edited

Loading