CSPL-1219: Adding app framework boundary tests #470

jambrosiano · 2021-08-20T01:45:48Z

1 new test added, 1 test updated:

Updated test: The 1st test is updated to perform several apps operation at the same time instead of just updating apps:
This test uses the following apps:
App A: microsoft-azure-add-on-for-splunk_300.tgz
App B: palo-alto-networks-add-on-for-splunk_660.tgz
App C: splunk-common-information-model-cim_4181.tgz
App D: splunk-es-content-update_3200.tgz
App E: add-on-for-ldap_400.tgz

At 1st, it will install A,B,C and D
After install is done, the following operations are performed in a 2nd time:

Update A and B
No changes for C
Remove D from the list of apps
Install E

Successful runs:
S1:
[2] • [SLOW TEST:1768.474 seconds]
[2] s1appfw test
[2] /Users/jambrosiano/splunk-operator/CSPL-1219/splunk-operator/test/s1/appframework/appframework_test.go:30
[2] appframework Standalone deployment (S1) with App Framework
[2] /Users/jambrosiano/splunk-operator/CSPL-1219/splunk-operator/test/s1/appframework/appframework_test.go:64
[2] smoke, s1, appframework: can deploy a standalone instance with App Framework enabled
[2] /Users/jambrosiano/splunk-operator/CSPL-1219/splunk-operator/test/s1/appframework/appframework_test.go:65
[2] ------------------------------
[2] {"level":"info","ts":1629941209.110615,"msg":"testenv deleted.\n","testenv":"s1appfw-rh3"}
[2]
[2] JUnit report was created: /Users/jambrosiano/splunk-operator/CSPL-1219/splunk-operator/test/s1/appframework/s1appfw-rh3_junit.xml
[2]
[2] Ran 1 of 1 Specs in 1783.687 seconds
[2] SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 0 Skipped
[2] PASS

C3:
[2] • [SLOW TEST:5337.306 seconds]
[2] c3appfw test
[2] /Users/jambrosiano/splunk-operator/CSPL-1219/splunk-operator/test/c3/appframework/appframework_test.go:29
[2] Single Site Indexer Cluster with SHC (C3) with App Framework
[2] /Users/jambrosiano/splunk-operator/CSPL-1219/splunk-operator/test/c3/appframework/appframework_test.go:63
[2] appfwint, c3, appframework: can deploy a C3 SVA with App Framework enabled
[2] /Users/jambrosiano/splunk-operator/CSPL-1219/splunk-operator/test/c3/appframework/appframework_test.go:64
[2] ------------------------------
[2] {"level":"info","ts":1629939057.600124,"msg":"testenv deleted.\n","testenv":"c3appfw-pde"}
[2] {"level":"info","ts":1629939057.6001651,"msg":"testenv deleted.\n","testenv":"c3appfw-pde"}
[2]
[2] JUnit report was created: /Users/jambrosiano/splunk-operator/CSPL-1219/splunk-operator/test/c3/appframework/c3appfw-pde_junit.xml
[2]
[2] Ran 1 of 1 Specs in 5352.625 seconds
[2] SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 0 Skipped
[2] PASS

New test: Installing a bigger amount of apps at the same time
Successful runs:
S1:
[1] • [SLOW TEST:464.035 seconds]
[1] s1appfw test
[1] /Users/jambrosiano/splunk-operator/CSPL-1219/splunk-operator/test/s1/appframework/appframework_test.go:30
[1] appframework Standalone deployment (S1) with App Framework
[1] /Users/jambrosiano/splunk-operator/CSPL-1219/splunk-operator/test/s1/appframework/appframework_test.go:339
[1] smoke, s1, appframework: can deploy a standalone instance with App Framework enabled and install around 350MB of apps at once
[1] /Users/jambrosiano/splunk-operator/CSPL-1219/splunk-operator/test/s1/appframework/appframework_test.go:340
[1] ------------------------------
[1] {"level":"info","ts":1629422367.995358,"msg":"testenv deleted.\n","testenv":"s1appfw-kex"}
[1]
[1] JUnit report was created: /Users/jambrosiano/splunk-operator/CSPL-1219/splunk-operator/test/s1/appframework/s1appfw-kex_junit.xml
[1]
[1] Ran 1 of 4 Specs in 480.768 seconds
[1] SUCCESS! -- 1 Passed | 0 Failed | 3 Pending | 0 Skipped
[1] PASS

C3:
[1]
[1] • [SLOW TEST:597.546 seconds]
[1] c3appfw test
[1] /Users/jambrosiano/splunk-operator/CSPL-1219/splunk-operator/test/c3/appframework/appframework_test.go:29
[1] Clustered deployment (C3 - clustered indexer, search head cluster)
[1] /Users/jambrosiano/splunk-operator/CSPL-1219/splunk-operator/test/c3/appframework/appframework_test.go:685
[1] smoke, c3, appframework: can deploy a C3 SVA instance with App Framework enabled and install above 200MB of apps at once
[1] /Users/jambrosiano/splunk-operator/CSPL-1219/splunk-operator/test/c3/appframework/appframework_test.go:686
[1] ------------------------------
[1] {"level":"info","ts":1629421191.313681,"msg":"testenv deleted.\n","testenv":"c3appfw-wzu"}
[1] {"level":"info","ts":1629421191.313709,"msg":"testenv deleted.\n","testenv":"c3appfw-wzu"}
[1]
[1] JUnit report was created: /Users/jambrosiano/splunk-operator/CSPL-1219/splunk-operator/test/c3/appframework/c3appfw-wzu_junit.xml
[1]
[1] Ran 1 of 6 Specs in 613.485 seconds
[1] SUCCESS! -- 1 Passed | 0 Failed | 5 Pending | 0 Skipped
[1] PASS

Note:
2 apps (Splunk_TA_nix and the splunk_app_stream suite) used for the S1 test are not used in the C3 one as they lead to test failure, probably due to extra setup needed.

sgontla · 2021-08-31T01:08:39Z

Thanks, @jambrosiano for covering these boundary test cases. In addition to these, can we also cover:

Re-add the App, that was deleted
Upgrade the app:
2.1. Using the same app package name
2.2. Using a modified app package name
(Not sure if we already have, otherwise)In standalone case
3.1 Start CR with replicas > 1, and apps
3.2 Scale up the replicas, and also have the add/modify/delete combination, and make sure that the apps are the same across all the Pods.

jambrosiano · 2021-08-31T23:36:13Z

Thanks, @jambrosiano for covering these boundary test cases. In addition to these, can we also cover:

Re-add the App, that was deleted

Upgrade the app:
2.1. Using the same app package name
2.2. Using a modified app package name

(Not sure if we already have, otherwise)In standalone case
3.1 Start CR with replicas > 1, and apps
3.2 Scale up the replicas, and also have the add/modify/delete combination, and make sure that the apps are the same across all the Pods.

Thanks ,@sgontla for bringing these test ideas.
Here's the status for each of them:
1: this test could be added in a new PR.
2.1 this is about having 2 files with exact same name, but different content/hash, like an updated .conf file for ex. This could be added also in a new PR.
2.2 This is handled by the current update tests we have.
3.1: This could be done by a PR verifying CSPL-1271, I'll create a ticket for that.
3.2: This is covered by the updated test in this PR, as it goes this way: install V1-> Upgrade to V2 -> Scale -> Downgrade to V1 -> do the add/modify/delete combination

pdhanoya-splunk · 2021-09-02T16:17:15Z

test/s1/appframework/appframework_test.go

 	})
+
+	Context("appframework Standalone deployment (S1) with App Framework", func() {
+		It("smoke, s1, appframework: can deploy a standalone instance with App Framework enabled and install around 350MB of apps at once", func() {


Nit: This is redundant as ES app is already 2GB +?

ES app is a special case, and installed alone. Here we install several apps together, more than just the basic apps

pdhanoya-splunk · 2021-09-02T16:18:03Z

test/c3/appframework/appframework_test.go

 			testenvInstance.Log.Info("Verify Apps are installed on the pods by running Splunk CLI commands for app", " version", appVersion)
 			testenv.VerifyAppInstalled(deployment, testenvInstance, testenvInstance.GetName(), allPodNames, appListV1, true, "enabled", false, true)

+			// Delete apps on S3 for new Apps


Nit: The run time of this test is already 30 mins + .. this would take it into 45 min mark.

This test becomes indeed very long, I would suggest this to be the very last addition to it.

Ideally this should not be added to the test however we have separate task to break these test down.

Thankfully its integration and not smoke!!

smohan-splunk · 2021-09-02T20:18:39Z

test/c3/appframework/appframework_test.go

 			testenvInstance.Log.Info("Verify Apps are installed on the pods by running Splunk CLI commands for app", " version", appVersion)
 			testenv.VerifyAppInstalled(deployment, testenvInstance, testenvInstance.GetName(), allPodNames, appListV1, true, "enabled", false, true)

+			// Delete apps on S3 for new Apps


Thankfully its integration and not smoke!!

* move m4 test to integration to clear smoke run (#487) * Feature circleci migration (#490) * Migration of Unit Test and Smoke Test pipelines from CircleCi to Github Actions (#441) * Update cron schedule * Fix a bug where standalone with replicas>1 won't come up (#489) * Added int test workflow and nightly workflows. (#493) * Modify monitoring console selection name to avoid eks cluster creation failure (#494) * Update the name of int test step (#496) * CSPL-1219 (#470) * [CSPL-1283] Fix AWS & minio S3 client code to support App framework on GCS (#498) * Fix minio S3 client code & incorrect minio initContainer ut Add changes to minio client code to handle generic S3 compatable remote stores, namely GCS. * CSPL-1301: Trigger app install for modified app pkgs (#503) * Trigger app install for modified app pkgs While an app package modification does trigger rewritteing the configmap. There are cases where this rewrite will result in the same data section as the previous version of the configmap (for example, when a single app is installed initially then modified.) When this is applied no change is detected and the Pod does not reset or install the new app. By adding a label to the configmap metadata, when an app package change is detected, we can increment this label. This will not affect the data in the configmap since the label is in the metadata section, however the label change will force a new ResourceVersion of the configmap and restart the pod, triggering the modified app install. [UPDATE] Instead of using a label in the ConfigMap metadata, reset the data in the app listing ConfigMap to nothing prior to setting it, forcing a new resourceVerison. * CSPL-1302: Bias-language removal Phase 1 [Comments & Docs] (#497) * Bias-language removal Phase 1 * CSPL-1316 : Avoid app framework flow from re-entrancy (#506) * Automated pre release workflow (#508) * CSPL-1230 Remove need for Secret keys in IAM env (#505) * CSPL-1230 Remove need for Secret keys in IAM env The operator code can be changed to allow no secretRef to be a valid volume config. If no secretRef is configured for a SmartStore or AppFramework volume, then assume the credentials are available on the env itself, possibly through a tool such as kube2iam. * Moved action file to right location (#510) * Update the URL on relese RULE (#512) * CSPL-1339: Fix make_bundle.sh to check for version change (#517) * Splunk Operator 1.0.3 release (#511) * Cspl 1335 official release workflow (#523) * Automated Release Workflow * Added automated workflow to merge develop to master * Fix typo in kubectl create secret command (#524) * Update README.md (#526) * Fix image push jobs from master branch (#530) * Fixing merge issues * CSPL-1298 (#521) * CSPL-1307: Bias-language removal Phase 1 (#504) Implement Github Actions workflow for Bias Language * Update spunk-operator image tag in release directory (#535) * CSPL-1327 App framework: App installation isn't triggered if 1 appsource is empty. (#519) * CSPL-1327 App install isn't triggered if 1 appsource is empty An empty `appSource` is not necessarily an error. Treat it as a no-op and throw a log message. * Adding fix for CSPL-1316 for MC * Update test case to accomodate Standalone Updating Phase * Merge issues * Fix Automation test case * Fix automation test * CSPL-1305 - Remove bias language from Paths & URLs (#509) * Bias-Lang Removal P1 - Handling Exceptions Consolidate Paths & URLs from external sources Include exceptions to GithubActions workflow * Fix URLs

* Merge master1.0.1 (#370) 1.0.1 Release * release/1.0.2 (#473) * release/1.0.2 - Addressing doc references to release 1.0.2 * - doc change * - Fixing the olm-catalog CRDs for the versions 1.0.0-RC, 1.0.0, and 1.0.1 (#475) * Fix role yaml * CSPL:1217 Added new cases for app version downgrade (#452) Co-authored-by: Sirish Mohan <68884189+smohan-splunk@users.noreply.github.com> * move m4 test to integration to clear smoke run (#487) * Feature circleci migration (#490) * Migration of Unit Test and Smoke Test pipelines from CircleCi to Github Actions (#441) * Update cron schedule * Fix a bug where standalone with replicas>1 won't come up (#489) * Added int test workflow and nightly workflows. (#493) * Modify monitoring console selection name to avoid eks cluster creation failure (#494) * Update the name of int test step (#496) * CSPL-1219 (#470) * [CSPL-1283] Fix AWS & minio S3 client code to support App framework on GCS (#498) * Fix minio S3 client code & incorrect minio initContainer ut Add changes to minio client code to handle generic S3 compatable remote stores, namely GCS. * CSPL-1301: Trigger app install for modified app pkgs (#503) * Trigger app install for modified app pkgs While an app package modification does trigger rewritteing the configmap. There are cases where this rewrite will result in the same data section as the previous version of the configmap (for example, when a single app is installed initially then modified.) When this is applied no change is detected and the Pod does not reset or install the new app. By adding a label to the configmap metadata, when an app package change is detected, we can increment this label. This will not affect the data in the configmap since the label is in the metadata section, however the label change will force a new ResourceVersion of the configmap and restart the pod, triggering the modified app install. [UPDATE] Instead of using a label in the ConfigMap metadata, reset the data in the app listing ConfigMap to nothing prior to setting it, forcing a new resourceVerison. * CSPL-1302: Bias-language removal Phase 1 [Comments & Docs] (#497) * Bias-language removal Phase 1 * CSPL-1316 : Avoid app framework flow from re-entrancy (#506) * Automated pre release workflow (#508) * CSPL-1230 Remove need for Secret keys in IAM env (#505) * CSPL-1230 Remove need for Secret keys in IAM env The operator code can be changed to allow no secretRef to be a valid volume config. If no secretRef is configured for a SmartStore or AppFramework volume, then assume the credentials are available on the env itself, possibly through a tool such as kube2iam. * Moved action file to right location (#510) * Update the URL on relese RULE (#512) * CSPL-1339: Fix make_bundle.sh to check for version change (#517) * Splunk Operator 1.0.3 release (#511) * Cspl 1335 official release workflow (#523) * Automated Release Workflow * Added automated workflow to merge develop to master * Fix typo in kubectl create secret command (#524) * Update README.md (#526) * Fix image push jobs from master branch (#530) * CSPL-1298 (#521) * CSPL-1307: Bias-language removal Phase 1 (#504) Implement Github Actions workflow for Bias Language * Update spunk-operator image tag in release directory (#535) * CSPL-1327 App framework: App installation isn't triggered if 1 appsource is empty. (#519) * CSPL-1327 App install isn't triggered if 1 appsource is empty An empty `appSource` is not necessarily an error. Treat it as a no-op and throw a log message. * CSPL-1305 - Remove bias language from Paths & URLs (#509) * Bias-Lang Removal P1 - Handling Exceptions Consolidate Paths & URLs from external sources Include exceptions to GithubActions workflow * CSPL-1305 - Resolved Nightly builds regression (#547) Fixing Bias Language URLs to use correct tokens * CSPL-1277: Check for status of bundle push on CM (#536) * CSPL-1310 Merge MC feature branch to develop (#548) * CSPL-1306 - Remove bias language from Functions & Local variables (#543) * Bias-Lang P1 - Removal from Functions & Vars Renamed Functions and variables Renamed non-CRD files Added local script for bias language * Generating CRDs after changes * Removing CRD changes for olm legacy versions * Added new image push to int test (#566) * CSPL:1387 Modify app upload logic to selectively upload apps within test case (#557) Co-authored-by: Sirish Mohan <68884189+smohan-splunk@users.noreply.github.com> * CSPL-1379 IDXC fails to scale up when MC CR is deployed with pre-existing IDXC (#562) * IDXC fails to scale up when MC CR is deployed in the namespace with pre existing IDXC * Review comments * CSPL-1438: Add missing mc test develop (#565) * Add Missing MC test in custom resource test * Add missing MC Test in License Manger * Add missing MC test in secret test cases * Cleanup for MC Missing test * CSPL-1412/CSPL-1413 Break down app framework tests (#567) * CSPL-1412 * Update appframework_test.go * CSPL-1447-updated-version-for-find-replace-action (#576) * Updated go version to 1.17.3 (#581) * Cspl 1456 fix nightly test failures by changing testenv length and cleanup env (#579) * CSPL-1443 increase length of random string in appframework test to avoid reuse of same namespace * CSPL-1443 change cleanup logic to delete env every time unless DEBUG is set to true * disabling failing smoke tests in S1 and C3

jambrosiano requested review from pdhanoya-splunk and tpereirasplunk August 20, 2021 01:45

CSPL-1219

aa4cba5

jambrosiano force-pushed the CSPL-1219 branch from 5f37546 to aa4cba5 Compare August 20, 2021 21:38

Merge branch 'develop' into CSPL-1219

5cfd11b

jambrosiano requested review from jryb, sgontla, kashok-splunk, mgaldino-splunk, gaurav-splunk, vebken-splunk, kumarajeet, smohan-splunk and akondur August 23, 2021 18:35

Merge branch 'develop' into CSPL-1219

0860080

jambrosiano force-pushed the CSPL-1219 branch from 2e7d364 to b71f3f2 Compare August 26, 2021 01:30

Merge branch 'develop' into CSPL-1219

a43415a

jambrosiano force-pushed the CSPL-1219 branch from b71f3f2 to a43415a Compare August 27, 2021 02:10

Merge branch 'develop' into CSPL-1219

1828aea

Merge branch 'develop' into CSPL-1219

5b01ec3

tpereirasplunk approved these changes Aug 31, 2021

View reviewed changes

jambrosiano added 2 commits September 1, 2021 12:10

Merge branch 'develop' into CSPL-1219

83789e9

Merge branch 'develop' into CSPL-1219

42f9c84

pdhanoya-splunk reviewed Sep 2, 2021

View reviewed changes

pdhanoya-splunk approved these changes Sep 2, 2021

View reviewed changes

Merge branch 'develop' into CSPL-1219

0339a82

sgontla approved these changes Sep 2, 2021

View reviewed changes

smohan-splunk approved these changes Sep 2, 2021

View reviewed changes

smohan-splunk merged commit 2efa9f2 into develop Sep 2, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

CSPL-1219: Adding app framework boundary tests #470

CSPL-1219: Adding app framework boundary tests #470

Uh oh!

jambrosiano commented Aug 20, 2021 •

edited

Loading

Uh oh!

sgontla commented Aug 31, 2021 •

edited

Loading

Uh oh!

jambrosiano commented Aug 31, 2021 •

edited

Loading

Uh oh!

pdhanoya-splunk Sep 2, 2021

Uh oh!

jambrosiano Sep 2, 2021

Uh oh!

pdhanoya-splunk Sep 2, 2021

Uh oh!

jambrosiano Sep 2, 2021

Uh oh!

pdhanoya-splunk Sep 2, 2021

Uh oh!

smohan-splunk Sep 2, 2021

Uh oh!

smohan-splunk Sep 2, 2021

Uh oh!

Uh oh!

CSPL-1219: Adding app framework boundary tests #470

CSPL-1219: Adding app framework boundary tests #470

Uh oh!

Conversation

jambrosiano commented Aug 20, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sgontla commented Aug 31, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jambrosiano commented Aug 31, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pdhanoya-splunk Sep 2, 2021

Choose a reason for hiding this comment

Uh oh!

jambrosiano Sep 2, 2021

Choose a reason for hiding this comment

Uh oh!

pdhanoya-splunk Sep 2, 2021

Choose a reason for hiding this comment

Uh oh!

jambrosiano Sep 2, 2021

Choose a reason for hiding this comment

Uh oh!

pdhanoya-splunk Sep 2, 2021

Choose a reason for hiding this comment

Uh oh!

smohan-splunk Sep 2, 2021

Choose a reason for hiding this comment

Uh oh!

smohan-splunk Sep 2, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jambrosiano commented Aug 20, 2021 •

edited

Loading

sgontla commented Aug 31, 2021 •

edited

Loading

jambrosiano commented Aug 31, 2021 •

edited

Loading