CSPL-1230 Remove need for Secret keys in IAM env #505

jryb · 2021-09-10T21:59:39Z

Problem

The current SmartStore and AppFramework volume configs require a secretRef containing the remote store secret and access keys. Some environments block the use of keys in favor of IAM roles. IAM roles can be inherited from the K8 Node itself or using 3rd party tools such as kube2iam. These volumes need to be allowed to get their credentials without the need of a secret.

Solution

kube2iam inherits the IAM roles from the node that runs the pod. Through annotations in the CRs, different roles can be assumed by the operator and Splunk instance pods (assuming the background tasks of IAM policy/trust between these roles is configured). The operator code can be changed to allow no secretRef to be a valid volume config. If no secretRef is configured for a SmartStore or AppFramework volume, then assume the credentials are available on the env itself, possibly through a tool such as kube2iam.

Since the operator and Splunk instance pods need to access the appRepo volumes, these annotations/credentials will need to be configured on both the operator and Splunk instance CRs.

Testing

Positive Testing

Setup kube2iam (out of scope of this PR).

operator.yaml:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: splunk-operator
spec:
  replicas: 1
  selector:
    matchLabels:
      name: splunk-operator
  template:
    metadata:
      labels:
        name: splunk-operator
      annotations:
        iam.amazonaws.com/role: 'arn:aws:iam::8675309:role/my-cool-role'
    spec:
      serviceAccountName: splunk-operator
      containers:
      - name: splunk-operator
        image: myimage/splunk-operator:test
        imagePullPolicy: Always
        env:
        - name: WATCH_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: OPERATOR_NAME
          value: "splunk-operator"
        - name: RELATED_IMAGE_SPLUNK_ENTERPRISE
          value: "docker.io/splunk/splunk:8.2.1-a2"

standalone.yml:

apiVersion: enterprise.splunk.com/v2
kind: Standalone
metadata:
  name: kube2iam-test
  finalizers:
  - enterprise.splunk.com/delete-pvc
  annotations:
    iam.amazonaws.com/role: 'arn:aws:iam::8675309:role/my-cool-role'
spec:
  replicas: 1
  appRepo:
    appsRepoPollIntervalSeconds: 60
    volumes:
      - name: volume_app_repo1
        path: bkt/apps
        endpoint: https://generic-endpoint
        storageType: s3
        provider: aws
    appSources:
      - name: common
        location: common/
        volumeName: volume_app_repo1
        scope: local
      - name: admin-apps
        location: standalones/
        volumeName: volume_app_repo1
        scope: local

Operator logs:

{"level":"info","ts":1631308822.6883898,"logger":"splunk.reconcile.Reconcile","msg":"Reconciling custom resource","Group":"enterprise.splunk.com","Version":"v2","Kind":"Standalone","Namespace":"default","Name":"kube2iam-test"}
{"level":"info","ts":1631308822.694542,"logger":"splunk.enterprise.ValidateAppFrameworkSpec","msg":"configCheck","scope":true}
{"level":"info","ts":1631308822.6945612,"logger":"splunk.enterprise.validateRemoteVolumeSpec","msg":"No valid SecretRef for volume.","volumeName":"volume_app_repo1"}
{"level":"info","ts":1631308822.6945689,"logger":"splunk.enterprise.ValidateAppFrameworkSpec","msg":"App framework configuration is valid"}
{"level":"info","ts":1631308822.6945956,"logger":"splunk.enterprise.HasAppRepoCheckTimerExpired","msg":"App repo polling interval timer has expired","LastAppInfoCheckTime":"0","current epoch time":"1631308822"}
{"level":"info","ts":1631308822.694604,"logger":"splunk.enterprise.initAndCheckAppInfoStatus","msg":"Checking status of apps on remote storage...","name":"kube2iam-test","namespace":"default"}
{"level":"info","ts":1631308822.6946113,"logger":"splunk.enterprise.GetAppListFromS3Bucket","msg":"Getting the list of apps from remote storage...","name":"kube2iam-test","namespace":"default"}
{"level":"info","ts":1631308822.6946204,"logger":"splunk.enterprise.GetRemoteStorageClient","msg":"No secrectRef provided.  Attempt to access remote storage client without access/secret keys","name":"kube2iam-test","namespace":"default"}
{"level":"info","ts":1631308822.6946275,"logger":"splunk.enterprise.GetRemoteStorageClient","msg":"Creating the client","name":"kube2iam-test","namespace":"default","volume":"volume_app_repo1","bucket":"appframework-demo-apps","bucket path":"apps/common/"}
{"level":"info","ts":1631308822.6947663,"logger":"splunk.client.InitAWSClientSession","msg":"AWS Client Session initialization successful.","region":"us-west-2","TLS Version":"TLS 1.2"}
{"level":"info","ts":1631308822.694777,"logger":"splunk.client.GetAppsList","msg":"Getting Apps list","AWS S3 Bucket":"appframework-demo-apps"}
{"level":"info","ts":1631308823.0418956,"logger":"splunk.enterprise.GetRemoteStorageClient","msg":"No secrectRef provided.  Attempt to access remote storage client without access/secret keys","name":"kube2iam-test","namespace":"default"}
{"level":"info","ts":1631308823.0419178,"logger":"splunk.enterprise.GetRemoteStorageClient","msg":"Creating the client","name":"kube2iam-test","namespace":"default","volume":"volume_app_repo1","bucket":"appframework-demo-apps","bucket path":"apps/standalones/"}
{"level":"info","ts":1631308823.0420961,"logger":"splunk.client.InitAWSClientSession","msg":"AWS Client Session initialization successful.","region":"us-west-2","TLS Version":"TLS 1.2"}
{"level":"info","ts":1631308823.0421097,"logger":"splunk.client.GetAppsList","msg":"Getting Apps list","AWS S3 Bucket":"appframework-demo-apps"}
{"level":"info","ts":1631308823.4298623,"logger":"splunk.enterprise.initAndCheckAppInfoStatus","msg":"Apps List retrieved from remote storage","name":"kube2iam-test","namespace":"default","App Source":"common","Content":[{"Etag":"\"a548ee98a59af6a15d0222b118cba154\"","Key":"apps/common/demo_cfg_for_all.spl","LastModified":"2021-07-13T06:30:21Z","Size":1248,"StorageClass":"STANDARD"},{"Etag":"\"045082d2e38ce0323f043c640550b663\"","Key":"apps/common/demo_limits_search.spl","LastModified":"2021-07-13T06:30:21Z","Size":1152,"StorageClass":"STANDARD"}]}
{"level":"info","ts":1631308823.4299347,"logger":"splunk.enterprise.initAndCheckAppInfoStatus","msg":"Apps List retrieved from remote storage","name":"kube2iam-test","namespace":"default","App Source":"admin-apps","Content":[{"Etag":"\"3091594731e8b9c2d550067556f9bf63\"","Key":"apps/standalones/demo_my_dashboards.spl","LastModified":"2021-09-10T17:03:53Z","Size":1804,"StorageClass":"STANDARD"}]}
{"level":"info","ts":1631308823.4299843,"logger":"splunk.enterprise.handleAppRepoChanges","msg":"received App listing","kind":"Standalone","name":"kube2iam-test","namespace":"default","for App sources":2}
{"level":"info","ts":1631308823.4299984,"logger":"splunk.enterprise.AddOrUpdateAppSrcDeploymentInfoList","msg":"New App found:","Called with length: ":2,"appName":"demo_cfg_for_all.spl"}
{"level":"info","ts":1631308823.4300036,"logger":"splunk.enterprise.AddOrUpdateAppSrcDeploymentInfoList","msg":"New App found:","Called with length: ":2,"appName":"demo_limits_search.spl"}
{"level":"info","ts":1631308823.4300103,"logger":"splunk.enterprise.AddOrUpdateAppSrcDeploymentInfoList","msg":"New App found:","Called with length: ":1,"appName":"demo_my_dashboards.spl"}
{"level":"info","ts":1631308823.4428954,"logger":"splunk.reconcile.CreateResource","msg":"Created resource","name":"splunk-kube2iam-test-standalone-app-list","namespace":"default"}
{"level":"info","ts":1631308823.4429157,"logger":"splunk.enterprise.SetLastAppInfoCheckTime","msg":"Setting the LastAppInfoCheckTime to current time","current epoch time":1631308823}

Splunk pod validation:

splunk-default-monitoring-console-0   0/1     Running   0          12s
splunk-kube2iam-test-standalone-0     1/1     Running   0          83s
splunk-operator-6c45c9f664-chdl4      1/1     Running   0          6m9s

$ kex-sc splunk-kube2iam-test-standalone-0
[splunk@splunk-kube2iam-test-standalone-0 splunk]$ ls -Rl /init-apps/
/init-apps/:
total 0
drwxr-sr-x 2 splunk splunk 36 Sep 10 21:20 admin-apps
drwxr-sr-x 2 splunk splunk 64 Sep 10 21:20 common

/init-apps/admin-apps:
total 4
-rw-r--r-- 1 splunk splunk 1804 Sep 10 17:03 demo_my_dashboards.spl

/init-apps/common:
total 8
-rw-r--r-- 1 splunk splunk 1248 Jul 13 06:30 demo_cfg_for_all.spl
-rw-r--r-- 1 splunk splunk 1152 Jul 13 06:30 demo_limits_search.spl
[splunk@splunk-kube2iam-test-standalone-0 splunk]$ ls etc/apps/
SplunkForwarder       introspection_generator_addon  splunk-dashboard-studio	  splunk_metrics_workspace
SplunkLightForwarder  journald_input		     splunk_archiver		  splunk_monitoring_console
alert_logevent	      launcher			     splunk_enterprise_on_docker  splunk_rapid_diag
alert_webhook	      learned			     splunk_essentials_8_2	  splunk_secure_gateway
appsbrowser	      legacy			     splunk_gdi			  user-prefs
demo_cfg_for_all      python_upgrade_readiness_app   splunk_httpinput
demo_limits_search    sample_app		     splunk_instrumentation
demo_my_dashboards    search			     splunk_internal_metrics
[splunk@splunk-kube2iam-test-standalone-0 splunk]$ export SPLUNK_ADMIN_PW=$(cat /mnt/splunk-secrets/password); bin/splunk display app -auth admin:$SPLUNK_ADMIN_PW

alert_logevent                 CONFIGURED          ENABLED             INVISIBLE           
alert_webhook                  CONFIGURED          ENABLED             INVISIBLE           
appsbrowser                    CONFIGURED          ENABLED             INVISIBLE           
demo_cfg_for_all               CONFIGURED          ENABLED             INVISIBLE           
demo_limits_search             CONFIGURED          ENABLED             INVISIBLE           
demo_my_dashboards             CONFIGURED          ENABLED             VISIBLE             
introspection_generator_addon  CONFIGURED          ENABLED             INVISIBLE           
journald_input                 UNCONFIGURED        ENABLED             INVISIBLE           
launcher                       CONFIGURED          ENABLED             VISIBLE             
learned                        UNCONFIGURED        ENABLED             INVISIBLE           
legacy                         UNCONFIGURED        DISABLED            INVISIBLE           
python_upgrade_readiness_app   UNCONFIGURED        ENABLED             VISIBLE             
sample_app                     UNCONFIGURED        DISABLED            INVISIBLE           
search                         CONFIGURED          ENABLED             VISIBLE             
splunk-dashboard-studio        CONFIGURED          ENABLED             VISIBLE             
splunk_archiver                CONFIGURED          ENABLED             INVISIBLE           
splunk_enterprise_on_docker    CONFIGURED          ENABLED             INVISIBLE           
splunk_essentials_8_2          CONFIGURED          ENABLED             VISIBLE             
splunk_gdi                     UNCONFIGURED        ENABLED             INVISIBLE           
splunk_httpinput               UNCONFIGURED        ENABLED             INVISIBLE           
splunk_instrumentation         UNCONFIGURED        ENABLED             VISIBLE             
splunk_internal_metrics        UNCONFIGURED        ENABLED             INVISIBLE           
splunk_metrics_workspace       UNCONFIGURED        ENABLED             VISIBLE             
splunk_monitoring_console      UNCONFIGURED        ENABLED             VISIBLE             
splunk_rapid_diag              UNCONFIGURED        ENABLED             VISIBLE             
splunk_secure_gateway          UNCONFIGURED        ENABLED             VISIBLE             
SplunkForwarder                UNCONFIGURED        DISABLED            INVISIBLE           
SplunkLightForwarder           UNCONFIGURED        DISABLED            INVISIBLE           
[splunk@splunk-kube2iam-test-standalone-0 splunk]$ 
[splunk@splunk-kube2iam-test-standalone-0 splunk]$ exit
exit
jryb:AppFramework$ k get configmap/splunk-kube2iam-test-standalone-app-list -o yaml
apiVersion: v1
data:
  app-list-local.yaml: |-
    splunk:
      app_paths_install:
        default:
          - "/init-apps/admin-apps/demo_my_dashboards.spl"
          - "/init-apps/common/demo_cfg_for_all.spl"
          - "/init-apps/common/demo_limits_search.spl"
kind: ConfigMap
metadata:
  creationTimestamp: "2021-09-10T21:20:23Z"
  name: splunk-kube2iam-test-standalone-app-list
  namespace: default
  ownerReferences:
  - apiVersion: enterprise.splunk.com/v2
    controller: true
    kind: Standalone
    name: kube2iam-test
    uid: 19aaf46e-2b87-40f2-911f-510a1730a172
  resourceVersion: "68879123"
  selfLink: /api/v1/namespaces/default/configmaps/splunk-kube2iam-test-standalone-app-list
  uid: f715e360-2f0f-42b0-b2c7-8d9814ae8eb0

Validate S2

Using the following standalone CR validate that SmartStore works using without a secretref.

apiVersion: enterprise.splunk.com/v2
kind: Standalone
metadata:
  name: kube2iam-test
  finalizers:
  - enterprise.splunk.com/delete-pvc
  annotations:
    iam.amazonaws.com/role: 'arn:aws:iam::8675309:role/my-cool-role'
spec:
  replicas: 1
  smartstore:
    defaults:
      volumeName: s2s3_vol
    indexes:
      - name: testindex2
      - name: testindex3
      - name: testindex1
        volumeName: s2s3_vol
        remotePath: $_index_name
    volumes:
      - name: s2s3_vol
        path: s2-test/kube2iam-test
        endpoint: https://generic-endpoint
        storageType: s3
        provider: aws

Check S3 and they buckets are being uploaded. SmartStore is verified to work without a SecretRef.

Negative testing

Incorrect pod annotation results in init container failure:

metadata:
  annotations:
    iam.amazonaws.com/role: 'arn:aws:iam::8675309:role/not-my-cool-role'

$ klog splunk-kube2iam-test-standalone-0   common-init-0-local
fatal error: Unable to locate credentials

$ kdes pod/splunk-kube2iam-test-standalone-0 
Name:         splunk-kube2iam-test-standalone-0
Namespace:    default
Priority:     0
Node:         ip-192-168-12-134.us-west-2.compute.internal/192.168.12.134
Start Time:   Fri, 10 Sep 2021 14:17:38 -0700
Labels:       app.kubernetes.io/component=standalone
              app.kubernetes.io/instance=splunk-kube2iam-test-standalone
              app.kubernetes.io/managed-by=splunk-operator
              app.kubernetes.io/name=standalone
              app.kubernetes.io/part-of=splunk-kube2iam-test-standalone
              controller-revision-hash=splunk-kube2iam-test-standalone-66b4dd9496
              statefulset.kubernetes.io/pod-name=splunk-kube2iam-test-standalone-0
Annotations:  appListingRev: 68878489
              iam.amazonaws.com/role: arn:aws:iam::8675309:role/not-my-cool-role
              kubernetes.io/psp: eks.privileged
              traffic.sidecar.istio.io/excludeOutboundPorts: 8089,8191,9997
              traffic.sidecar.istio.io/includeInboundPorts: 8000,8088
Status:       Pending
IP:           192.168.11.87
IPs:
  IP:           192.168.11.87
Controlled By:  StatefulSet/splunk-kube2iam-test-standalone
Init Containers:
  common-init-0-local:
    Container ID:  docker://b577362d39cd7a0d8d2095806fe618122dfabb4897bf4e9b69a4ff4806748c5c
    Image:         amazon/aws-cli
    Image ID:      docker-pullable://amazon/aws-cli@sha256:eaf6b4fef81106e9eb1e3ae125070a22e76d671fea7be1d87790998ce4d9f2c2
    Port:          <none>
    Host Port:     <none>
    Args:
      --endpoint-url=https://s3-us-west-2.amazonaws.com
      s3
      sync
      s3://appframework-demo-apps/apps/common/
      /init-apps/common/
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Fri, 10 Sep 2021 14:18:12 -0700
      Finished:     Fri, 10 Sep 2021 14:18:13 -0700
    Ready:          False
    Restart Count:  2
    Environment:    <none>
    Mounts:
      /init-apps/ from init-apps (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-qbtm6 (ro)

Incorrect IAM role in operator results in operator not connecting to list appRepo and configmap not being created (though instance pod is able to download app packages).

spec:
  template:
    metadata:
      annotations:
        iam.amazonaws.com/role: 'arn:aws:iam::8675309:role/not-my-cool-role'
    spec:
      serviceAccountName: splunk-operator
      containers:
      - name: splunk-operator

{"level":"info","ts":1631309417.179098,"logger":"splunk.enterprise.validateRemoteVolumeSpec","msg":"No valid SecretRef for volume.","volumeName":"volume_app_repo1"}
{"level":"info","ts":1631309417.179105,"logger":"splunk.enterprise.ValidateAppFrameworkSpec","msg":"App framework configuration is valid"}
{"level":"info","ts":1631309417.1791325,"logger":"splunk.enterprise.HasAppRepoCheckTimerExpired","msg":"App repo polling interval timer has expired","LastAppInfoCheckTime":"0","current epoch time":"1631309417"}
{"level":"info","ts":1631309417.1791406,"logger":"splunk.enterprise.initAndCheckAppInfoStatus","msg":"Checking status of apps on remote storage...","name":"kube2iam-test","namespace":"default"}
{"level":"info","ts":1631309417.1791482,"logger":"splunk.enterprise.GetAppListFromS3Bucket","msg":"Getting the list of apps from remote storage...","name":"kube2iam-test","namespace":"default"}
{"level":"info","ts":1631309417.179165,"logger":"splunk.enterprise.GetRemoteStorageClient","msg":"No secrectRef provided.  Attempt to access remote storage client without access/secret keys","name":"kube2iam-test","namespace":"default"}
{"level":"info","ts":1631309417.1791806,"logger":"splunk.enterprise.GetRemoteStorageClient","msg":"Creating the client","name":"kube2iam-test","namespace":"default","volume":"volume_app_repo1","bucket":"appframework-demo-apps","bucket path":"apps/common/"}
{"level":"info","ts":1631309417.1794202,"logger":"splunk.client.InitAWSClientSession","msg":"AWS Client Session initialization successful.","region":"us-west-2","TLS Version":"TLS 1.2"}
{"level":"info","ts":1631309417.1794322,"logger":"splunk.client.GetAppsList","msg":"Getting Apps list","AWS S3 Bucket":"appframework-demo-apps"}
{"level":"error","ts":1631309417.826377,"logger":"splunk.client.GetAppsList","msg":"Unable to list items in bucket","AWS S3 Bucket":"appframework-demo-apps","error":"NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\tsplunk-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/splunk/splunk-operator/pkg/splunk/client.(*AWSS3Client).GetAppsList\n\tsplunk-operator/pkg/splunk/client/awss3client.go:184\ngithub.com/splunk/splunk-operator/pkg/splunk/enterprise.(*S3ClientManager).GetAppsList\n\tsplunk-operator/pkg/splunk/enterprise/util.go:512\ngithub.com/splunk/splunk-operator/pkg/splunk/enterprise.GetAppListFromS3Bucket\n\tsplunk-operator/pkg/splunk/enterprise/util.go:553\ngithub.com/splunk/splunk-operator/pkg/splunk/enterprise.initAndCheckAppInfoStatus\n\tsplunk-operator/pkg/splunk/enterprise/util.go:984\ngithub.com/splunk/splunk-operator/pkg/splunk/enterprise.ApplyStandalone\n\tsplunk-operator/pkg/splunk/enterprise/standalone.go:76\ngithub.com/splunk/splunk-operator/pkg/controller.StandaloneController.Reconcile\n\tsplunk-operator/pkg/controller/add_standalone.go:59\ngithub.com/splunk/splunk-operator/pkg/splunk/controller.splunkReconciler.Reconcile\n\tsplunk-operator/pkg/splunk/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsplunk-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:256\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsplunk-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\tsplunk-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\tsplunk-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\tsplunk-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\tsplunk-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\tsplunk-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90"}
{"level":"error","ts":1631309417.8265083,"logger":"splunk.enterprise.GetAppListFromS3Bucket","msg":"Unable to get apps list","name":"kube2iam-test","namespace":"default","appSource":"common","error":"NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\tsplunk-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/splunk/splunk-operator/pkg/splunk/enterprise.GetAppListFromS3Bucket\n\tsplunk-operator/pkg/splunk/enterprise/util.go:556\ngithub.com/splunk/splunk-operator/pkg/splunk/enterprise.initAndCheckAppInfoStatus\n\tsplunk-operator/pkg/splunk/enterprise/util.go:984\ngithub.com/splunk/splunk-operator/pkg/splunk/enterprise.ApplyStandalone\n\tsplunk-operator/pkg/splunk/enterprise/standalone.go:76\ngithub.com/splunk/splunk-operator/pkg/controller.StandaloneController.Reconcile\n\tsplunk-operator/pkg/controller/add_standalone.go:59\ngithub.com/splunk/splunk-operator/pkg/splunk/controller.splunkReconciler.Reconcile\n\tsplunk-operator/pkg/splunk/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsplunk-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:256\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsplunk-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\tsplunk-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\tsplunk-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\tsplunk-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\tsplunk-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\tsplunk-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90"}
{"level":"info","ts":1631309417.826565,"logger":"splunk.enterprise.GetRemoteStorageClient","msg":"No secrectRef provided.  Attempt to access remote storage client without access/secret keys","name":"kube2iam-test","namespace":"default"}
{"level":"info","ts":1631309417.826578,"logger":"splunk.enterprise.GetRemoteStorageClient","msg":"Creating the client","name":"kube2iam-test","namespace":"default","volume":"volume_app_repo1","bucket":"appframework-demo-apps","bucket path":"apps/standalones/"}
{"level":"info","ts":1631309417.8267417,"logger":"splunk.client.InitAWSClientSession","msg":"AWS Client Session initialization successful.","region":"us-west-2","TLS Version":"TLS 1.2"}
{"level":"info","ts":1631309417.826758,"logger":"splunk.client.GetAppsList","msg":"Getting Apps list","AWS S3 Bucket":"appframework-demo-apps"}
{"level":"error","ts":1631309418.447108,"logger":"splunk.client.GetAppsList","msg":"Unable to list items in bucket","AWS S3 Bucket":"appframework-demo-apps","error":"NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\tsplunk-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/splunk/splunk-operator/pkg/splunk/client.(*AWSS3Client).GetAppsList\n\tsplunk-operator/pkg/splunk/client/awss3client.go:184\ngithub.com/splunk/splunk-operator/pkg/splunk/enterprise.(*S3ClientManager).GetAppsList\n\tsplunk-operator/pkg/splunk/enterprise/util.go:512\ngithub.com/splunk/splunk-operator/pkg/splunk/enterprise.GetAppListFromS3Bucket\n\tsplunk-operator/pkg/splunk/enterprise/util.go:553\ngithub.com/splunk/splunk-operator/pkg/splunk/enterprise.initAndCheckAppInfoStatus\n\tsplunk-operator/pkg/splunk/enterprise/util.go:984\ngithub.com/splunk/splunk-operator/pkg/splunk/enterprise.ApplyStandalone\n\tsplunk-operator/pkg/splunk/enterprise/standalone.go:76\ngithub.com/splunk/splunk-operator/pkg/controller.StandaloneController.Reconcile\n\tsplunk-operator/pkg/controller/add_standalone.go:59\ngithub.com/splunk/splunk-operator/pkg/splunk/controller.splunkReconciler.Reconcile\n\tsplunk-operator/pkg/splunk/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsplunk-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:256\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsplunk-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\tsplunk-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\tsplunk-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\tsplunk-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\tsplunk-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\tsplunk-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90"}
{"level":"error","ts":1631309418.4472423,"logger":"splunk.enterprise.GetAppListFromS3Bucket","msg":"Unable to get apps list","name":"kube2iam-test","namespace":"default","appSource":"admin-apps","error":"NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\tsplunk-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/splunk/splunk-operator/pkg/splunk/enterprise.GetAppListFromS3Bucket\n\tsplunk-operator/pkg/splunk/enterprise/util.go:556\ngithub.com/splunk/splunk-operator/pkg/splunk/enterprise.initAndCheckAppInfoStatus\n\tsplunk-operator/pkg/splunk/enterprise/util.go:984\ngithub.com/splunk/splunk-operator/pkg/splunk/enterprise.ApplyStandalone\n\tsplunk-operator/pkg/splunk/enterprise/standalone.go:76\ngithub.com/splunk/splunk-operator/pkg/controller.StandaloneController.Reconcile\n\tsplunk-operator/pkg/controller/add_standalone.go:59\ngithub.com/splunk/splunk-operator/pkg/splunk/controller.splunkReconciler.Reconcile\n\tsplunk-operator/pkg/splunk/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsplunk-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:256\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsplunk-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\tsplunk-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\tsplunk-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\tsplunk-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\tsplunk-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\tsplunk-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90"}

$ k get configmap/splunk-kube2iam-test-standalone-app-list -o yaml
Error from server (NotFound): configmaps "splunk-kube2iam-test-standalone-app-list" not found
$ kex-sc splunk-kube2iam-test-standalone-0
[splunk@splunk-kube2iam-test-standalone-0 splunk]$ ls -Rl /init-apps/
/init-apps/:
total 0
drwxr-sr-x 2 splunk splunk 36 Sep 10 21:31 admin-apps
drwxr-sr-x 2 splunk splunk 64 Sep 10 21:31 common

/init-apps/admin-apps:
total 4
-rw-r--r-- 1 splunk splunk 1804 Sep 10 17:03 demo_my_dashboards.spl

/init-apps/common:
total 8
-rw-r--r-- 1 splunk splunk 1248 Jul 13 06:30 demo_cfg_for_all.spl
-rw-r--r-- 1 splunk splunk 1152 Jul 13 06:30 demo_limits_search.spl
[splunk@splunk-kube2iam-test-standalone-0 splunk]$ export SPLUNK_ADMIN_PW=$(cat /mnt/splunk-secrets/password); bin/splunk display app -auth admin:$SPLUNK_ADMIN_PW

alert_logevent                 CONFIGURED          ENABLED             INVISIBLE           
alert_webhook                  CONFIGURED          ENABLED             INVISIBLE           
appsbrowser                    CONFIGURED          ENABLED             INVISIBLE           
introspection_generator_addon  CONFIGURED          ENABLED             INVISIBLE           
journald_input                 UNCONFIGURED        ENABLED             INVISIBLE           
launcher                       CONFIGURED          ENABLED             VISIBLE             
learned                        UNCONFIGURED        ENABLED             INVISIBLE           
legacy                         UNCONFIGURED        DISABLED            INVISIBLE           
python_upgrade_readiness_app   UNCONFIGURED        ENABLED             VISIBLE             
sample_app                     UNCONFIGURED        DISABLED            INVISIBLE           
search                         CONFIGURED          ENABLED             VISIBLE             
splunk-dashboard-studio        CONFIGURED          ENABLED             VISIBLE             
splunk_archiver                CONFIGURED          ENABLED             INVISIBLE           
splunk_enterprise_on_docker    CONFIGURED          ENABLED             INVISIBLE           
splunk_essentials_8_2          CONFIGURED          ENABLED             VISIBLE             
splunk_gdi                     UNCONFIGURED        ENABLED             INVISIBLE           
splunk_httpinput               UNCONFIGURED        ENABLED             INVISIBLE           
splunk_instrumentation         UNCONFIGURED        ENABLED             VISIBLE             
splunk_internal_metrics        UNCONFIGURED        ENABLED             INVISIBLE           
splunk_metrics_workspace       UNCONFIGURED        ENABLED             VISIBLE             
splunk_monitoring_console      UNCONFIGURED        ENABLED             VISIBLE             
splunk_rapid_diag              UNCONFIGURED        ENABLED             VISIBLE             
splunk_secure_gateway          UNCONFIGURED        ENABLED             VISIBLE             
SplunkForwarder                UNCONFIGURED        DISABLED            INVISIBLE           
SplunkLightForwarder           UNCONFIGURED        DISABLED            INVISIBLE           
[splunk@splunk-kube2iam-test-standalone-0 splunk]$ 
[splunk@splunk-kube2iam-test-standalone-0 splunk]$ ls etc/apps/
SplunkForwarder		       launcher			     splunk_archiver		  splunk_metrics_workspace
SplunkLightForwarder	       learned			     splunk_enterprise_on_docker  splunk_monitoring_console
alert_logevent		       legacy			     splunk_essentials_8_2	  splunk_rapid_diag
alert_webhook		       python_upgrade_readiness_app  splunk_gdi			  splunk_secure_gateway
appsbrowser		       sample_app		     splunk_httpinput		  user-prefs
introspection_generator_addon  search			     splunk_instrumentation
journald_input		       splunk-dashboard-studio	     splunk_internal_metrics
[splunk@splunk-kube2iam-test-standalone-0 splunk]$

The operator code can be changed to allow no secretRef to be a valid volume config. If no secretRef is configured for a SmartStore or AppFramework volume, then assume the credentials are available on the env itself, possibly through a tool such as kube2iam.

Missing secretRef in volume is a valid config.

pkg/splunk/client/awss3client.go

pkg/splunk/client/minioclient.go

pkg/splunk/enterprise/configuration_test.go

pkg/splunk/enterprise/util.go

- Cleanup code for readability - Tweak minio S3Client to work for IAM

pkg/splunk/enterprise/configuration.go

smohan-splunk · 2021-09-13T18:41:40Z

Can we please update the docs Appframework.md & Smartstore.md to make it clear that post this change we don't mandate the volume credentials via secretref

jryb · 2021-09-13T18:53:39Z

Can we please update the docs Appframework.md & Smartstore.md to make it clear that post this change we don't mandate the volume credentials via secretref

I'm currently looking through the docs to make these changes. Do we want to update the docs as part of this PR or create a separate docs PR?

smohan-splunk · 2021-09-13T18:55:45Z

Can we please update the docs Appframework.md & Smartstore.md to make it clear that post this change we don't mandate the volume credentials via secretref

I'm currently looking through the docs to make these changes. Do we want to update the docs as part of this PR or create a separate docs PR?

Let us include in this PR if possible.

For Smartstore and app framework, role based credentials are allowed making the secretRef for static credentials optional.

…splunk/splunk-operator into bugfix/CSPL-1230_allow_kube2iam

docs/AppFramework.md

…splunk/splunk-operator into bugfix/CSPL-1230_allow_kube2iam

* move m4 test to integration to clear smoke run (#487) * Feature circleci migration (#490) * Migration of Unit Test and Smoke Test pipelines from CircleCi to Github Actions (#441) * Update cron schedule * Fix a bug where standalone with replicas>1 won't come up (#489) * Added int test workflow and nightly workflows. (#493) * Modify monitoring console selection name to avoid eks cluster creation failure (#494) * Update the name of int test step (#496) * CSPL-1219 (#470) * [CSPL-1283] Fix AWS & minio S3 client code to support App framework on GCS (#498) * Fix minio S3 client code & incorrect minio initContainer ut Add changes to minio client code to handle generic S3 compatable remote stores, namely GCS. * CSPL-1301: Trigger app install for modified app pkgs (#503) * Trigger app install for modified app pkgs While an app package modification does trigger rewritteing the configmap. There are cases where this rewrite will result in the same data section as the previous version of the configmap (for example, when a single app is installed initially then modified.) When this is applied no change is detected and the Pod does not reset or install the new app. By adding a label to the configmap metadata, when an app package change is detected, we can increment this label. This will not affect the data in the configmap since the label is in the metadata section, however the label change will force a new ResourceVersion of the configmap and restart the pod, triggering the modified app install. [UPDATE] Instead of using a label in the ConfigMap metadata, reset the data in the app listing ConfigMap to nothing prior to setting it, forcing a new resourceVerison. * CSPL-1302: Bias-language removal Phase 1 [Comments & Docs] (#497) * Bias-language removal Phase 1 * CSPL-1316 : Avoid app framework flow from re-entrancy (#506) * Automated pre release workflow (#508) * CSPL-1230 Remove need for Secret keys in IAM env (#505) * CSPL-1230 Remove need for Secret keys in IAM env The operator code can be changed to allow no secretRef to be a valid volume config. If no secretRef is configured for a SmartStore or AppFramework volume, then assume the credentials are available on the env itself, possibly through a tool such as kube2iam. * Moved action file to right location (#510) * Update the URL on relese RULE (#512) * CSPL-1339: Fix make_bundle.sh to check for version change (#517) * Splunk Operator 1.0.3 release (#511) * Cspl 1335 official release workflow (#523) * Automated Release Workflow * Added automated workflow to merge develop to master * Fix typo in kubectl create secret command (#524) * Update README.md (#526) * Fix image push jobs from master branch (#530) * Fixing merge issues * CSPL-1298 (#521) * CSPL-1307: Bias-language removal Phase 1 (#504) Implement Github Actions workflow for Bias Language * Update spunk-operator image tag in release directory (#535) * CSPL-1327 App framework: App installation isn't triggered if 1 appsource is empty. (#519) * CSPL-1327 App install isn't triggered if 1 appsource is empty An empty `appSource` is not necessarily an error. Treat it as a no-op and throw a log message. * Adding fix for CSPL-1316 for MC * Update test case to accomodate Standalone Updating Phase * Merge issues * Fix Automation test case * Fix automation test * CSPL-1305 - Remove bias language from Paths & URLs (#509) * Bias-Lang Removal P1 - Handling Exceptions Consolidate Paths & URLs from external sources Include exceptions to GithubActions workflow * Fix URLs

* Merge master1.0.1 (#370) 1.0.1 Release * release/1.0.2 (#473) * release/1.0.2 - Addressing doc references to release 1.0.2 * - doc change * - Fixing the olm-catalog CRDs for the versions 1.0.0-RC, 1.0.0, and 1.0.1 (#475) * Fix role yaml * CSPL:1217 Added new cases for app version downgrade (#452) Co-authored-by: Sirish Mohan <68884189+smohan-splunk@users.noreply.github.com> * move m4 test to integration to clear smoke run (#487) * Feature circleci migration (#490) * Migration of Unit Test and Smoke Test pipelines from CircleCi to Github Actions (#441) * Update cron schedule * Fix a bug where standalone with replicas>1 won't come up (#489) * Added int test workflow and nightly workflows. (#493) * Modify monitoring console selection name to avoid eks cluster creation failure (#494) * Update the name of int test step (#496) * CSPL-1219 (#470) * [CSPL-1283] Fix AWS & minio S3 client code to support App framework on GCS (#498) * Fix minio S3 client code & incorrect minio initContainer ut Add changes to minio client code to handle generic S3 compatable remote stores, namely GCS. * CSPL-1301: Trigger app install for modified app pkgs (#503) * Trigger app install for modified app pkgs While an app package modification does trigger rewritteing the configmap. There are cases where this rewrite will result in the same data section as the previous version of the configmap (for example, when a single app is installed initially then modified.) When this is applied no change is detected and the Pod does not reset or install the new app. By adding a label to the configmap metadata, when an app package change is detected, we can increment this label. This will not affect the data in the configmap since the label is in the metadata section, however the label change will force a new ResourceVersion of the configmap and restart the pod, triggering the modified app install. [UPDATE] Instead of using a label in the ConfigMap metadata, reset the data in the app listing ConfigMap to nothing prior to setting it, forcing a new resourceVerison. * CSPL-1302: Bias-language removal Phase 1 [Comments & Docs] (#497) * Bias-language removal Phase 1 * CSPL-1316 : Avoid app framework flow from re-entrancy (#506) * Automated pre release workflow (#508) * CSPL-1230 Remove need for Secret keys in IAM env (#505) * CSPL-1230 Remove need for Secret keys in IAM env The operator code can be changed to allow no secretRef to be a valid volume config. If no secretRef is configured for a SmartStore or AppFramework volume, then assume the credentials are available on the env itself, possibly through a tool such as kube2iam. * Moved action file to right location (#510) * Update the URL on relese RULE (#512) * CSPL-1339: Fix make_bundle.sh to check for version change (#517) * Splunk Operator 1.0.3 release (#511) * Cspl 1335 official release workflow (#523) * Automated Release Workflow * Added automated workflow to merge develop to master * Fix typo in kubectl create secret command (#524) * Update README.md (#526) * Fix image push jobs from master branch (#530) * CSPL-1298 (#521) * CSPL-1307: Bias-language removal Phase 1 (#504) Implement Github Actions workflow for Bias Language * Update spunk-operator image tag in release directory (#535) * CSPL-1327 App framework: App installation isn't triggered if 1 appsource is empty. (#519) * CSPL-1327 App install isn't triggered if 1 appsource is empty An empty `appSource` is not necessarily an error. Treat it as a no-op and throw a log message. * CSPL-1305 - Remove bias language from Paths & URLs (#509) * Bias-Lang Removal P1 - Handling Exceptions Consolidate Paths & URLs from external sources Include exceptions to GithubActions workflow * CSPL-1305 - Resolved Nightly builds regression (#547) Fixing Bias Language URLs to use correct tokens * CSPL-1277: Check for status of bundle push on CM (#536) * CSPL-1310 Merge MC feature branch to develop (#548) * CSPL-1306 - Remove bias language from Functions & Local variables (#543) * Bias-Lang P1 - Removal from Functions & Vars Renamed Functions and variables Renamed non-CRD files Added local script for bias language * Generating CRDs after changes * Removing CRD changes for olm legacy versions * Added new image push to int test (#566) * CSPL:1387 Modify app upload logic to selectively upload apps within test case (#557) Co-authored-by: Sirish Mohan <68884189+smohan-splunk@users.noreply.github.com> * CSPL-1379 IDXC fails to scale up when MC CR is deployed with pre-existing IDXC (#562) * IDXC fails to scale up when MC CR is deployed in the namespace with pre existing IDXC * Review comments * CSPL-1438: Add missing mc test develop (#565) * Add Missing MC test in custom resource test * Add missing MC Test in License Manger * Add missing MC test in secret test cases * Cleanup for MC Missing test * CSPL-1412/CSPL-1413 Break down app framework tests (#567) * CSPL-1412 * Update appframework_test.go * CSPL-1447-updated-version-for-find-replace-action (#576) * Updated go version to 1.17.3 (#581) * Cspl 1456 fix nightly test failures by changing testenv length and cleanup env (#579) * CSPL-1443 increase length of random string in appframework test to avoid reuse of same namespace * CSPL-1443 change cleanup logic to delete env every time unless DEBUG is set to true * disabling failing smoke tests in S1 and C3

jryb added 2 commits September 10, 2021 14:48

Fix ut error.

3103b7d

Missing secretRef in volume is a valid config.

jryb requested review from gaurav-splunk, smohan-splunk, sgontla, vebken-splunk, kashok-splunk and pdhanoya-splunk September 10, 2021 22:07

gaurav-splunk reviewed Sep 10, 2021

View reviewed changes

pkg/splunk/client/awss3client.go Show resolved Hide resolved

pkg/splunk/client/minioclient.go Outdated Show resolved Hide resolved

pkg/splunk/enterprise/configuration_test.go Outdated Show resolved Hide resolved

pkg/splunk/enterprise/util.go Show resolved Hide resolved

Address PR comments

59f103b

- Cleanup code for readability - Tweak minio S3Client to work for IAM

gaurav-splunk approved these changes Sep 12, 2021

View reviewed changes

sgontla reviewed Sep 13, 2021

View reviewed changes

pkg/splunk/enterprise/configuration.go Show resolved Hide resolved

pkg/splunk/enterprise/configuration.go Show resolved Hide resolved

sgontla approved these changes Sep 13, 2021

View reviewed changes

kashok-splunk approved these changes Sep 13, 2021

View reviewed changes

jryb and others added 2 commits September 13, 2021 11:22

Fix scopedLog name.

2623581

Merge branch 'develop' into bugfix/CSPL-1230_allow_kube2iam

f2a8696

jryb added 2 commits September 13, 2021 13:10

Modify docs to make sercretRef optional

a1bb2c8

For Smartstore and app framework, role based credentials are allowed making the secretRef for static credentials optional.

Merge branch 'bugfix/CSPL-1230_allow_kube2iam' of https://github.com/…

3e8a7fe

…splunk/splunk-operator into bugfix/CSPL-1230_allow_kube2iam

jryb requested a review from eddikay September 13, 2021 20:17

eddikay reviewed Sep 13, 2021

View reviewed changes

docs/AppFramework.md Outdated Show resolved Hide resolved

docs/AppFramework.md Outdated Show resolved Hide resolved

Fix wording for AppFramework docs.

9a899e5

smohan-splunk approved these changes Sep 14, 2021

View reviewed changes

docs/AppFramework.md Outdated Show resolved Hide resolved

jryb and others added 5 commits September 14, 2021 09:55

Use IAM role instead in docs.

0358b6e

Merge branch 'develop' into bugfix/CSPL-1230_allow_kube2iam

731ffbe

Fix typo

1a29e09

Merge branch 'bugfix/CSPL-1230_allow_kube2iam' of https://github.com/…

2d52ba8

…splunk/splunk-operator into bugfix/CSPL-1230_allow_kube2iam

Merge branch 'develop' into bugfix/CSPL-1230_allow_kube2iam

6537c33

smohan-splunk merged commit f803e2d into develop Sep 14, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

CSPL-1230 Remove need for Secret keys in IAM env #505

CSPL-1230 Remove need for Secret keys in IAM env #505

Uh oh!

jryb commented Sep 10, 2021 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

smohan-splunk commented Sep 13, 2021

Uh oh!

jryb commented Sep 13, 2021

Uh oh!

smohan-splunk commented Sep 13, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

CSPL-1230 Remove need for Secret keys in IAM env #505

CSPL-1230 Remove need for Secret keys in IAM env #505

Uh oh!

Conversation

jryb commented Sep 10, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Solution

Testing

Positive Testing

Validate S2

Negative testing

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

smohan-splunk commented Sep 13, 2021

Uh oh!

jryb commented Sep 13, 2021

Uh oh!

smohan-splunk commented Sep 13, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jryb commented Sep 10, 2021 •

edited

Loading