[CSPL-1283] Fix AWS & minio S3 client code to support App framework on GCS #498
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add changes to minio client code to handle generic S3 compatable remote stores, namely GCS.
sgontla
reviewed
Sep 3, 2021
sgontla
approved these changes
Sep 3, 2021
gaurav-splunk
approved these changes
Sep 3, 2021
smohan-splunk
approved these changes
Sep 7, 2021
smohan-splunk
changed the title
smohan-splunk
changed the title
smohan-splunk
pushed a commit
that referenced
this pull request
Oct 19, 2021
* move m4 test to integration to clear smoke run (#487) * Feature circleci migration (#490) * Migration of Unit Test and Smoke Test pipelines from CircleCi to Github Actions (#441) * Update cron schedule * Fix a bug where standalone with replicas>1 won't come up (#489) * Added int test workflow and nightly workflows. (#493) * Modify monitoring console selection name to avoid eks cluster creation failure (#494) * Update the name of int test step (#496) * CSPL-1219 (#470) * [CSPL-1283] Fix AWS & minio S3 client code to support App framework on GCS (#498) * Fix minio S3 client code & incorrect minio initContainer ut Add changes to minio client code to handle generic S3 compatable remote stores, namely GCS. * CSPL-1301: Trigger app install for modified app pkgs (#503) * Trigger app install for modified app pkgs While an app package modification does trigger rewritteing the configmap. There are cases where this rewrite will result in the same data section as the previous version of the configmap (for example, when a single app is installed initially then modified.) When this is applied no change is detected and the Pod does not reset or install the new app. By adding a label to the configmap metadata, when an app package change is detected, we can increment this label. This will not affect the data in the configmap since the label is in the metadata section, however the label change will force a new ResourceVersion of the configmap and restart the pod, triggering the modified app install. [UPDATE] Instead of using a label in the ConfigMap metadata, reset the data in the app listing ConfigMap to nothing prior to setting it, forcing a new resourceVerison. * CSPL-1302: Bias-language removal Phase 1 [Comments & Docs] (#497) * Bias-language removal Phase 1 * CSPL-1316 : Avoid app framework flow from re-entrancy (#506) * Automated pre release workflow (#508) * CSPL-1230 Remove need for Secret keys in IAM env (#505) * CSPL-1230 Remove need for Secret keys in IAM env The operator code can be changed to allow no secretRef to be a valid volume config. If no secretRef is configured for a SmartStore or AppFramework volume, then assume the credentials are available on the env itself, possibly through a tool such as kube2iam. * Moved action file to right location (#510) * Update the URL on relese RULE (#512) * CSPL-1339: Fix make_bundle.sh to check for version change (#517) * Splunk Operator 1.0.3 release (#511) * Cspl 1335 official release workflow (#523) * Automated Release Workflow * Added automated workflow to merge develop to master * Fix typo in kubectl create secret command (#524) * Update README.md (#526) * Fix image push jobs from master branch (#530) * Fixing merge issues * CSPL-1298 (#521) * CSPL-1307: Bias-language removal Phase 1 (#504) Implement Github Actions workflow for Bias Language * Update spunk-operator image tag in release directory (#535) * CSPL-1327 App framework: App installation isn't triggered if 1 appsource is empty. (#519) * CSPL-1327 App install isn't triggered if 1 appsource is empty An empty `appSource` is not necessarily an error. Treat it as a no-op and throw a log message. * Adding fix for CSPL-1316 for MC * Update test case to accomodate Standalone Updating Phase * Merge issues * Fix Automation test case * Fix automation test * CSPL-1305 - Remove bias language from Paths & URLs (#509) * Bias-Lang Removal P1 - Handling Exceptions Consolidate Paths & URLs from external sources Include exceptions to GithubActions workflow * Fix URLs
smohan-splunk
pushed a commit
that referenced
this pull request
Nov 10, 2021
* Merge master1.0.1 (#370) 1.0.1 Release * release/1.0.2 (#473) * release/1.0.2 - Addressing doc references to release 1.0.2 * - doc change * - Fixing the olm-catalog CRDs for the versions 1.0.0-RC, 1.0.0, and 1.0.1 (#475) * Fix role yaml * CSPL:1217 Added new cases for app version downgrade (#452) Co-authored-by: Sirish Mohan <68884189+smohan-splunk@users.noreply.github.com> * move m4 test to integration to clear smoke run (#487) * Feature circleci migration (#490) * Migration of Unit Test and Smoke Test pipelines from CircleCi to Github Actions (#441) * Update cron schedule * Fix a bug where standalone with replicas>1 won't come up (#489) * Added int test workflow and nightly workflows. (#493) * Modify monitoring console selection name to avoid eks cluster creation failure (#494) * Update the name of int test step (#496) * CSPL-1219 (#470) * [CSPL-1283] Fix AWS & minio S3 client code to support App framework on GCS (#498) * Fix minio S3 client code & incorrect minio initContainer ut Add changes to minio client code to handle generic S3 compatable remote stores, namely GCS. * CSPL-1301: Trigger app install for modified app pkgs (#503) * Trigger app install for modified app pkgs While an app package modification does trigger rewritteing the configmap. There are cases where this rewrite will result in the same data section as the previous version of the configmap (for example, when a single app is installed initially then modified.) When this is applied no change is detected and the Pod does not reset or install the new app. By adding a label to the configmap metadata, when an app package change is detected, we can increment this label. This will not affect the data in the configmap since the label is in the metadata section, however the label change will force a new ResourceVersion of the configmap and restart the pod, triggering the modified app install. [UPDATE] Instead of using a label in the ConfigMap metadata, reset the data in the app listing ConfigMap to nothing prior to setting it, forcing a new resourceVerison. * CSPL-1302: Bias-language removal Phase 1 [Comments & Docs] (#497) * Bias-language removal Phase 1 * CSPL-1316 : Avoid app framework flow from re-entrancy (#506) * Automated pre release workflow (#508) * CSPL-1230 Remove need for Secret keys in IAM env (#505) * CSPL-1230 Remove need for Secret keys in IAM env The operator code can be changed to allow no secretRef to be a valid volume config. If no secretRef is configured for a SmartStore or AppFramework volume, then assume the credentials are available on the env itself, possibly through a tool such as kube2iam. * Moved action file to right location (#510) * Update the URL on relese RULE (#512) * CSPL-1339: Fix make_bundle.sh to check for version change (#517) * Splunk Operator 1.0.3 release (#511) * Cspl 1335 official release workflow (#523) * Automated Release Workflow * Added automated workflow to merge develop to master * Fix typo in kubectl create secret command (#524) * Update README.md (#526) * Fix image push jobs from master branch (#530) * CSPL-1298 (#521) * CSPL-1307: Bias-language removal Phase 1 (#504) Implement Github Actions workflow for Bias Language * Update spunk-operator image tag in release directory (#535) * CSPL-1327 App framework: App installation isn't triggered if 1 appsource is empty. (#519) * CSPL-1327 App install isn't triggered if 1 appsource is empty An empty `appSource` is not necessarily an error. Treat it as a no-op and throw a log message. * CSPL-1305 - Remove bias language from Paths & URLs (#509) * Bias-Lang Removal P1 - Handling Exceptions Consolidate Paths & URLs from external sources Include exceptions to GithubActions workflow * CSPL-1305 - Resolved Nightly builds regression (#547) Fixing Bias Language URLs to use correct tokens * CSPL-1277: Check for status of bundle push on CM (#536) * CSPL-1310 Merge MC feature branch to develop (#548) * CSPL-1306 - Remove bias language from Functions & Local variables (#543) * Bias-Lang P1 - Removal from Functions & Vars Renamed Functions and variables Renamed non-CRD files Added local script for bias language * Generating CRDs after changes * Removing CRD changes for olm legacy versions * Added new image push to int test (#566) * CSPL:1387 Modify app upload logic to selectively upload apps within test case (#557) Co-authored-by: Sirish Mohan <68884189+smohan-splunk@users.noreply.github.com> * CSPL-1379 IDXC fails to scale up when MC CR is deployed with pre-existing IDXC (#562) * IDXC fails to scale up when MC CR is deployed in the namespace with pre existing IDXC * Review comments * CSPL-1438: Add missing mc test develop (#565) * Add Missing MC test in custom resource test * Add missing MC Test in License Manger * Add missing MC test in secret test cases * Cleanup for MC Missing test * CSPL-1412/CSPL-1413 Break down app framework tests (#567) * CSPL-1412 * Update appframework_test.go * CSPL-1447-updated-version-for-find-replace-action (#576) * Updated go version to 1.17.3 (#581) * Cspl 1456 fix nightly test failures by changing testenv length and cleanup env (#579) * CSPL-1443 increase length of random string in appframework test to avoid reuse of same namespace * CSPL-1443 change cleanup logic to delete env every time unless DEBUG is set to true * disabling failing smoke tests in S1 and C3
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem
When attempting to access a bucket on GCS (which is S3 compatable) the aws and minio clients both fail. The aws client fails due to a missing region in the endpoint formatting. The minio client init container fails to download anything and the app-listing configmap is malformed:
Solution
Fix the init container command so that there are no "//" in the path.
Fix the List API so that it doesn't use object pointers and get stuck on the last app returned from minio list API.
Test
Create a bucket on GCS with a secret/access key. Create a GCS secret object and standalone referring to that remote store with
minioas the provider:Make sure apps get downloaded and installed onto the standalone: