CSPL-1305 - Remove bias language from Paths & URLs #509

mgaldino-splunk · 2021-09-13T23:55:50Z

This PR focus on exceptional cases of Paths & URLs where the bias language comes from external sources, outside the Operator codebase which we do not have the control to change.

Approach: We centralized all bias Paths & URLs into the splcommon package which will be handled as an exception. Then we utilize the constants created as global variables across the codebase.

Benefits: Have a central point with constants that are used across the codebase. Also, in the future when those Paths & URLs change to non-bias language the efforts to adapt the operator will be minimal.

Estimated progress:
Before PR:
"terms_found": true,
"total_lines_matched": 1806,
"total_words_matched": 2417,
"total_files_matched": 132

After PR:
"terms_found": true,
"total_lines_matched": 1029,
"total_words_matched": 1205,
"total_files_matched": 78

Consolidate Paths & URLs from external sources Include exceptions to GithubActions workflow

mgaldino-splunk · 2021-10-05T22:12:58Z

test/smoke/cluster_master_sites_response.go

 package smoke

 // ClusterMasterSitesResponse is a representation of the sites managed by a Splunk cluster-manager
-// Endpoint: /services/cluster/master/sites


Some endpoints have been migrated to use manager as
here. But not all.

Our general approach was to migrate all comments while using constants for literals (non-comments).

mgaldino-splunk · 2021-10-05T22:16:27Z

pkg/splunk/common/urls_testingcode.go

+	//*****************
+
+	//TestGetMonitoringConsoleStatefulSet
+	TestGetMonitoringConsoleStatefulSet = `{"kind":"StatefulSet","apiVersion":"apps/v1","metadata":{"name":"splunk-test-monitoring-console","namespace":"test","creationTimestamp":null},"spec":{"replicas":1,"selector":{"matchLabels":{"app.kubernetes.io/component":"monitoring-console","app.kubernetes.io/instance":"splunk-test-monitoring-console","app.kubernetes.io/managed-by":"splunk-operator","app.kubernetes.io/name":"monitoring-console","app.kubernetes.io/part-of":"splunk-test-monitoring-console"}},"template":{"metadata":{"creationTimestamp":null,"labels":{"app.kubernetes.io/component":"monitoring-console","app.kubernetes.io/instance":"splunk-test-monitoring-console","app.kubernetes.io/managed-by":"splunk-operator","app.kubernetes.io/name":"monitoring-console","app.kubernetes.io/part-of":"splunk-test-monitoring-console"},"annotations":{"monitoringConsoleConfigRev":"","traffic.sidecar.istio.io/excludeOutboundPorts":"8089,8191,9997","traffic.sidecar.istio.io/includeInboundPorts":"8000,8088"}},"spec":{"volumes":[{"name":"mnt-splunk-etc","emptyDir":{}},{"name":"mnt-splunk-var","emptyDir":{}},{"name":"mnt-splunk-secrets","secret":{"secretName":"splunk-test-secret","defaultMode":420}}],"containers":[{"name":"splunk","image":"splunk/splunk","ports":[{"name":"http-splunkweb","containerPort":8000,"protocol":"TCP"},{"name":"http-hec","containerPort":8088,"protocol":"TCP"},{"name":"https-splunkd","containerPort":8089,"protocol":"TCP"},{"name":"tcp-s2s","containerPort":9997,"protocol":"TCP"}],"envFrom":[{"configMapRef":{"name":"splunk-test-monitoring-console"}}],"env":[{"name":"SPLUNK_HOME","value":"/opt/splunk"},{"name":"SPLUNK_START_ARGS","value":"--accept-license"},{"name":"SPLUNK_DEFAULTS_URL","value":"/mnt/splunk-secrets/default.yml"},{"name":"SPLUNK_HOME_OWNERSHIP_ENFORCEMENT","value":"false"},{"name":"SPLUNK_ROLE","value":"splunk_monitor"},{"name":"SPLUNK_DECLARATIVE_ADMIN_PASSWORD","value":"true"},{"name":"SPLUNK_CLUSTER_MASTER_URL","value":"splunk-stack1-cluster-master-service"}],"resources":{"limits":{"cpu":"4","memory":"8Gi"},"requests":{"cpu":"100m","memory":"512Mi"}},"volumeMounts":[{"name":"mnt-splunk-etc","mountPath":"/opt/splunk/etc"},{"name":"mnt-splunk-var","mountPath":"/opt/splunk/var"},{"name":"mnt-splunk-secrets","mountPath":"/mnt/splunk-secrets"}],"livenessProbe":{"exec":{"command":["/sbin/checkstate.sh"]},"initialDelaySeconds":300,"timeoutSeconds":30,"periodSeconds":30},"readinessProbe":{"exec":{"command":["/bin/grep","started","/opt/container_artifact/splunk-container.state"]},"initialDelaySeconds":10,"timeoutSeconds":5,"periodSeconds":5},"imagePullPolicy":"IfNotPresent"}],"securityContext":{"runAsUser":41812,"fsGroup":41812},"affinity":{"podAntiAffinity":{"preferredDuringSchedulingIgnoredDuringExecution":[{"weight":100,"podAffinityTerm":{"labelSelector":{"matchExpressions":[{"key":"app.kubernetes.io/instance","operator":"In","values":["splunk-test-monitoring-console"]}]},"topologyKey":"kubernetes.io/hostname"}}]}},"schedulerName":"default-scheduler"}},"serviceName":"splunk-test-monitoring-console-headless","podManagementPolicy":"Parallel","updateStrategy":{"type":"OnDelete"}},"status":{"replicas":0}}`


@pdhanoyasplunk & @jambrosiano I was puzzled to find a way to optimize the Body responses but did not find a solution without requiring a lot of effort/refactoring. Please feel free to add any suggestions.

mgaldino-splunk · 2021-10-05T22:33:55Z

.biased_lang_exclude

+pkg/splunk/common/urls.go
+pkg/splunk/common/urls_testingcode.go
+pkg/splunk/common/paths.go
+deploy/olm-catalog/


This line adds the Legacy CRDs as exceptions for the Bias Language process.

pkg/splunk/common/urls.go

pkg/splunk/common/util_test.go

test/c3/appframework/appframework_test.go

test/testenv/appframework_utils.go

pkg/splunk/common/urls.go

pkg/splunk/enterprise/clustermaster_test.go

pkg/splunk/enterprise/indexercluster_test.go

pkg/splunk/common/urls_testingcode.go

pkg/splunk/common/urls.go

pkg/splunk/enterprise/util_test.go

pkg/splunk/common/paths.go

pkg/splunk/common/urls.go

* move m4 test to integration to clear smoke run (#487) * Feature circleci migration (#490) * Migration of Unit Test and Smoke Test pipelines from CircleCi to Github Actions (#441) * Update cron schedule * Fix a bug where standalone with replicas>1 won't come up (#489) * Added int test workflow and nightly workflows. (#493) * Modify monitoring console selection name to avoid eks cluster creation failure (#494) * Update the name of int test step (#496) * CSPL-1219 (#470) * [CSPL-1283] Fix AWS & minio S3 client code to support App framework on GCS (#498) * Fix minio S3 client code & incorrect minio initContainer ut Add changes to minio client code to handle generic S3 compatable remote stores, namely GCS. * CSPL-1301: Trigger app install for modified app pkgs (#503) * Trigger app install for modified app pkgs While an app package modification does trigger rewritteing the configmap. There are cases where this rewrite will result in the same data section as the previous version of the configmap (for example, when a single app is installed initially then modified.) When this is applied no change is detected and the Pod does not reset or install the new app. By adding a label to the configmap metadata, when an app package change is detected, we can increment this label. This will not affect the data in the configmap since the label is in the metadata section, however the label change will force a new ResourceVersion of the configmap and restart the pod, triggering the modified app install. [UPDATE] Instead of using a label in the ConfigMap metadata, reset the data in the app listing ConfigMap to nothing prior to setting it, forcing a new resourceVerison. * CSPL-1302: Bias-language removal Phase 1 [Comments & Docs] (#497) * Bias-language removal Phase 1 * CSPL-1316 : Avoid app framework flow from re-entrancy (#506) * Automated pre release workflow (#508) * CSPL-1230 Remove need for Secret keys in IAM env (#505) * CSPL-1230 Remove need for Secret keys in IAM env The operator code can be changed to allow no secretRef to be a valid volume config. If no secretRef is configured for a SmartStore or AppFramework volume, then assume the credentials are available on the env itself, possibly through a tool such as kube2iam. * Moved action file to right location (#510) * Update the URL on relese RULE (#512) * CSPL-1339: Fix make_bundle.sh to check for version change (#517) * Splunk Operator 1.0.3 release (#511) * Cspl 1335 official release workflow (#523) * Automated Release Workflow * Added automated workflow to merge develop to master * Fix typo in kubectl create secret command (#524) * Update README.md (#526) * Fix image push jobs from master branch (#530) * Fixing merge issues * CSPL-1298 (#521) * CSPL-1307: Bias-language removal Phase 1 (#504) Implement Github Actions workflow for Bias Language * Update spunk-operator image tag in release directory (#535) * CSPL-1327 App framework: App installation isn't triggered if 1 appsource is empty. (#519) * CSPL-1327 App install isn't triggered if 1 appsource is empty An empty `appSource` is not necessarily an error. Treat it as a no-op and throw a log message. * Adding fix for CSPL-1316 for MC * Update test case to accomodate Standalone Updating Phase * Merge issues * Fix Automation test case * Fix automation test * CSPL-1305 - Remove bias language from Paths & URLs (#509) * Bias-Lang Removal P1 - Handling Exceptions Consolidate Paths & URLs from external sources Include exceptions to GithubActions workflow * Fix URLs

* Merge master1.0.1 (#370) 1.0.1 Release * release/1.0.2 (#473) * release/1.0.2 - Addressing doc references to release 1.0.2 * - doc change * - Fixing the olm-catalog CRDs for the versions 1.0.0-RC, 1.0.0, and 1.0.1 (#475) * Fix role yaml * CSPL:1217 Added new cases for app version downgrade (#452) Co-authored-by: Sirish Mohan <68884189+smohan-splunk@users.noreply.github.com> * move m4 test to integration to clear smoke run (#487) * Feature circleci migration (#490) * Migration of Unit Test and Smoke Test pipelines from CircleCi to Github Actions (#441) * Update cron schedule * Fix a bug where standalone with replicas>1 won't come up (#489) * Added int test workflow and nightly workflows. (#493) * Modify monitoring console selection name to avoid eks cluster creation failure (#494) * Update the name of int test step (#496) * CSPL-1219 (#470) * [CSPL-1283] Fix AWS & minio S3 client code to support App framework on GCS (#498) * Fix minio S3 client code & incorrect minio initContainer ut Add changes to minio client code to handle generic S3 compatable remote stores, namely GCS. * CSPL-1301: Trigger app install for modified app pkgs (#503) * Trigger app install for modified app pkgs While an app package modification does trigger rewritteing the configmap. There are cases where this rewrite will result in the same data section as the previous version of the configmap (for example, when a single app is installed initially then modified.) When this is applied no change is detected and the Pod does not reset or install the new app. By adding a label to the configmap metadata, when an app package change is detected, we can increment this label. This will not affect the data in the configmap since the label is in the metadata section, however the label change will force a new ResourceVersion of the configmap and restart the pod, triggering the modified app install. [UPDATE] Instead of using a label in the ConfigMap metadata, reset the data in the app listing ConfigMap to nothing prior to setting it, forcing a new resourceVerison. * CSPL-1302: Bias-language removal Phase 1 [Comments & Docs] (#497) * Bias-language removal Phase 1 * CSPL-1316 : Avoid app framework flow from re-entrancy (#506) * Automated pre release workflow (#508) * CSPL-1230 Remove need for Secret keys in IAM env (#505) * CSPL-1230 Remove need for Secret keys in IAM env The operator code can be changed to allow no secretRef to be a valid volume config. If no secretRef is configured for a SmartStore or AppFramework volume, then assume the credentials are available on the env itself, possibly through a tool such as kube2iam. * Moved action file to right location (#510) * Update the URL on relese RULE (#512) * CSPL-1339: Fix make_bundle.sh to check for version change (#517) * Splunk Operator 1.0.3 release (#511) * Cspl 1335 official release workflow (#523) * Automated Release Workflow * Added automated workflow to merge develop to master * Fix typo in kubectl create secret command (#524) * Update README.md (#526) * Fix image push jobs from master branch (#530) * CSPL-1298 (#521) * CSPL-1307: Bias-language removal Phase 1 (#504) Implement Github Actions workflow for Bias Language * Update spunk-operator image tag in release directory (#535) * CSPL-1327 App framework: App installation isn't triggered if 1 appsource is empty. (#519) * CSPL-1327 App install isn't triggered if 1 appsource is empty An empty `appSource` is not necessarily an error. Treat it as a no-op and throw a log message. * CSPL-1305 - Remove bias language from Paths & URLs (#509) * Bias-Lang Removal P1 - Handling Exceptions Consolidate Paths & URLs from external sources Include exceptions to GithubActions workflow * CSPL-1305 - Resolved Nightly builds regression (#547) Fixing Bias Language URLs to use correct tokens * CSPL-1277: Check for status of bundle push on CM (#536) * CSPL-1310 Merge MC feature branch to develop (#548) * CSPL-1306 - Remove bias language from Functions & Local variables (#543) * Bias-Lang P1 - Removal from Functions & Vars Renamed Functions and variables Renamed non-CRD files Added local script for bias language * Generating CRDs after changes * Removing CRD changes for olm legacy versions * Added new image push to int test (#566) * CSPL:1387 Modify app upload logic to selectively upload apps within test case (#557) Co-authored-by: Sirish Mohan <68884189+smohan-splunk@users.noreply.github.com> * CSPL-1379 IDXC fails to scale up when MC CR is deployed with pre-existing IDXC (#562) * IDXC fails to scale up when MC CR is deployed in the namespace with pre existing IDXC * Review comments * CSPL-1438: Add missing mc test develop (#565) * Add Missing MC test in custom resource test * Add missing MC Test in License Manger * Add missing MC test in secret test cases * Cleanup for MC Missing test * CSPL-1412/CSPL-1413 Break down app framework tests (#567) * CSPL-1412 * Update appframework_test.go * CSPL-1447-updated-version-for-find-replace-action (#576) * Updated go version to 1.17.3 (#581) * Cspl 1456 fix nightly test failures by changing testenv length and cleanup env (#579) * CSPL-1443 increase length of random string in appframework test to avoid reuse of same namespace * CSPL-1443 change cleanup logic to delete env every time unless DEBUG is set to true * disabling failing smoke tests in S1 and C3

mgaldino-splunk marked this pull request as draft September 13, 2021 23:56

mgaldino-splunk force-pushed the bias-language-exceptions branch 5 times, most recently from 386d95c to f9f2e35 Compare September 14, 2021 19:25

mgaldino-splunk changed the title ~~Bias language exceptions~~ Draft: Bias language exceptions Sep 15, 2021

mgaldino-splunk force-pushed the bias-language-exceptions branch 4 times, most recently from 49155b5 to b21c52d Compare September 17, 2021 15:57

mgaldino-splunk force-pushed the bias-language-exceptions branch 7 times, most recently from 4d9277f to 57967b1 Compare October 4, 2021 21:56

mgaldino-splunk changed the title ~~Draft: Bias language exceptions~~ Draft: CSPL-1305 - Remove bias language from hardcoded paths & URLs Oct 5, 2021

mgaldino-splunk force-pushed the bias-language-exceptions branch 2 times, most recently from 99b50ad to feda87f Compare October 5, 2021 21:51

Bias-Lang Removal P1 - Handling Exceptions

8f6d1bf

Consolidate Paths & URLs from external sources Include exceptions to GithubActions workflow

mgaldino-splunk force-pushed the bias-language-exceptions branch from feda87f to 8f6d1bf Compare October 5, 2021 21:52

mgaldino-splunk requested review from kashok-splunk, smohan-splunk, jryb, kumarajeet, gaurav-splunk, vebken-splunk and sgontla October 5, 2021 21:57

mgaldino-splunk self-assigned this Oct 5, 2021

mgaldino-splunk marked this pull request as ready for review October 5, 2021 22:01

mgaldino-splunk changed the title ~~Draft: CSPL-1305 - Remove bias language from Paths & URLs~~ CSPL-1305 - Remove bias language from Paths & URLs Oct 5, 2021

mgaldino-splunk commented Oct 5, 2021

View reviewed changes

Merge branch 'develop' into bias-language-exceptions

5f36aa6

mgaldino-splunk requested a review from tpereirasplunk October 8, 2021 17:11

smohan-splunk approved these changes Oct 8, 2021

View reviewed changes

Addressing review comments I

be917ef

mgaldino-splunk force-pushed the bias-language-exceptions branch from 2850617 to be917ef Compare October 11, 2021 16:48

smohan-splunk approved these changes Oct 11, 2021

View reviewed changes

Merge branch 'develop' into bias-language-exceptions

5a2492d

sgontla reviewed Oct 11, 2021

View reviewed changes

gaurav-splunk reviewed Oct 11, 2021

View reviewed changes

pkg/splunk/common/urls.go Outdated Show resolved Hide resolved

pkg/splunk/enterprise/util_test.go Outdated Show resolved Hide resolved

pkg/splunk/enterprise/util_test.go Outdated Show resolved Hide resolved

mgaldino-splunk force-pushed the bias-language-exceptions branch 5 times, most recently from 7d090f1 to 6040689 Compare October 13, 2021 20:36

gaurav-splunk approved these changes Oct 13, 2021

View reviewed changes

sgontla reviewed Oct 13, 2021

View reviewed changes

pkg/splunk/common/paths.go Outdated Show resolved Hide resolved

pkg/splunk/common/urls.go Outdated Show resolved Hide resolved

mgaldino-splunk force-pushed the bias-language-exceptions branch 2 times, most recently from ea78fb8 to eaa7ec4 Compare October 13, 2021 23:51

Addressing review comments II

1c35d97

mgaldino-splunk force-pushed the bias-language-exceptions branch from eaa7ec4 to 1c35d97 Compare October 13, 2021 23:54

sgontla approved these changes Oct 13, 2021

View reviewed changes

smohan-splunk merged commit f3414d0 into develop Oct 14, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

CSPL-1305 - Remove bias language from Paths & URLs #509

CSPL-1305 - Remove bias language from Paths & URLs #509

Uh oh!

mgaldino-splunk commented Sep 13, 2021 •

edited

Loading

Uh oh!

mgaldino-splunk Oct 5, 2021

Uh oh!

mgaldino-splunk Oct 5, 2021

Uh oh!

mgaldino-splunk Oct 5, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

CSPL-1305 - Remove bias language from Paths & URLs #509

CSPL-1305 - Remove bias language from Paths & URLs #509

Uh oh!

Conversation

mgaldino-splunk commented Sep 13, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mgaldino-splunk Oct 5, 2021

Choose a reason for hiding this comment

Uh oh!

mgaldino-splunk Oct 5, 2021

Choose a reason for hiding this comment

Uh oh!

mgaldino-splunk Oct 5, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mgaldino-splunk commented Sep 13, 2021 •

edited

Loading