CSPL-821 Implement Init Container Image #350

jryb · 2021-05-24T19:16:08Z

Problem

At startup the Splunk pod needs the application packages it needs to install local to the pod to be installed via Ansible. These packages are define in the CRD via CSPL-1023 (#329).

Solution

To get the packages local to the Splunk instance init containers are used based on the app repository type to download the images to a shared volume. This volume will be shared between all init containers and the Splunk container. App packages will be placed in this volume based on the name define in their appSources. An init container is created per appSource to connect to the appRepo volume and download all packages in the appSource this shared volume. Once finished, all app package will be available for installation prior to the Splunk instance's instantiation.

Additional changes made:

Added minio as a provider client to the S3 client API to handle S3 setups that use Minio as their object store (like my local test setup).
Added init container code to the S3 client API to specify the image and commands needed by each S3 client to download the app packages as part of an init container.
Moved the getRegion() function to inside the aws S3 client code as it is AWS specific.
Added Endpoint to the S3 client code (particularly for the Minio client)

Unit Test

Verified app package download for both the minio and aws providers using multiple volume and multiple appSources.

Minio

CR

apiVersion: enterprise.splunk.com/v1
kind: Standalone
metadata:
  name: s-af
  finalizers:
  - enterprise.splunk.com/delete-pvc
spec:
  replicas: 1
  appRepo:
    appsRepoPollIntervalSeconds: 60
    volumes:
      - name: volume_app_repo1
        path: s7-apps/
        endpoint: http://10.244.0.6:9000
        secretRef: afsecret
        storageType: s3
        provider: minio
      - name: volume_app_repo2
        path: s7-apps-test/
        endpoint: http://10.244.0.6:9000
        secretRef: afsecret
        storageType: s3
        provider: minio
    appSources:
      - name: s7
        location: af-test/
        volumeName: volume_app_repo1
        scope: local
      - name: admin-apps
        location: admin/
        volumeName: volume_app_repo1
        scope: local
      - name: default-test
        location: default/
        volumeName: volume_app_repo2
        scope: local

Validate

k get pods

NAME                                  READY   STATUS    RESTARTS   AGE    IP             NODE                 NOMINATED NODE   READINESS GATES
minio-64b7c649f9-xqwm9                1/1     Running   0          3d1h   10.244.0.6     kind-control-plane   <none>           <none>
splunk-default-monitoring-console-0   0/1     Running   0          33s    10.244.0.183   kind-control-plane   <none>           <none>
splunk-operator-6596bf74d9-g8qr7      1/1     Running   0          64m    10.244.0.173   kind-control-plane   <none>           <none>
splunk-s-af-standalone-0              1/1     Running   0          110s   10.244.0.180   kind-control-plane   <none>           <none>
...
jryb:yamls$  kex-sc splunk-s-af-standalone-0  splunk
[splunk@splunk-s-af-standalone-0 splunk]$ ls -Rl /init-apps/
/init-apps/:
total 12
drwxr-sr-x 2 splunk splunk 4096 May 24 19:07 admin-apps
drwxr-sr-x 3 splunk splunk 4096 May 24 19:07 default-test
drwxr-sr-x 2 splunk splunk 4096 May 24 19:07 s7

/init-apps/admin-apps:
total 4
-rw-r--r-- 1 splunk splunk 1154 May 21 22:53 application4.tgz

/init-apps/default-test:
total 12
-rw-r--r-- 1 splunk splunk 1167 May 21 22:54 application3.tgz

/init-apps/s7:
total 8
-rw-r--r-- 1 splunk splunk 1154 May 21 17:54 application5.tgz

AWS

CR

apiVersion: enterprise.splunk.com/v1
kind: Standalone
metadata:
  name: s-af-aws
  finalizers:
  - enterprise.splunk.com/delete-pvc
spec:
  replicas: 1
  appRepo:
    appsRepoPollIntervalSeconds: 60
    volumes:
      - name: volume_app_repo1
        path: test-bucket/apps
        endpoint: https://s3-us-west-2.amazonaws.com
        secretRef: af-aws-secret
        storageType: s3
        provider: aws
    appSources:
      - name: af
        location: af-test/
        volumeName: volume_app_repo1
        scope: local
      - name: admin
        location: admin-apps/
        volumeName: volume_app_repo1
        scope: local
      - name: default
        location: default-test/
        volumeName: volume_app_repo1
        scope: local

Validate

k get pods

NAME                                  READY   STATUS    RESTARTS   AGE    IP             NODE                 NOMINATED NODE   READINESS GATES
splunk-default-monitoring-console-0   1/1     Running   0          47m    10.244.0.177   kind-control-plane   <none>           <none>
splunk-operator-6596bf74d9-g8qr7      1/1     Running   0          60m    10.244.0.173   kind-control-plane   <none>           <none>
splunk-s-af-aws-standalone-0          1/1     Running   0          48m    10.244.0.176   kind-control-plane   <none>           <none>
...
jryb:yamls$  kex-sc splunk-s-af-aws-standalone-0  splunk
[splunk@splunk-s-af-aws-standalone-0 splunk]$ ls -Rl /init-apps/
/init-apps/:
total 12
drwxr-sr-x 2 splunk splunk 4096 May 24 18:16 admin
drwxr-sr-x 2 splunk splunk 4096 May 24 18:16 af
drwxr-sr-x 2 splunk splunk 4096 May 24 18:16 default

/init-apps/admin:
total 4
-rw-r--r-- 1 splunk splunk 1155 May 24 17:21 application1.tgz

/init-apps/af:
total 4
-rw-r--r-- 1 splunk splunk 1154 May 24 17:21 application5.tgz

/init-apps/default:
total 4
-rw-r--r-- 1 splunk splunk 1154 May 24 17:21 application4.tgz

Mixed Minio and AWS

CR

apiVersion: enterprise.splunk.com/v1
kind: Standalone
metadata:
  name: s-af-mixed
  finalizers:
  - enterprise.splunk.com/delete-pvc
spec:
  replicas: 1
  appRepo:
    appsRepoPollIntervalSeconds: 60
    volumes:
      - name: volume_app_repo_minio
        path: s7-apps/
        endpoint: http://10.244.0.8:9000
        secretRef: afsecret
        storageType: s3
        provider: minio
      - name: volume_app_repo_aws
        path: test-bucket/apps
        endpoint: https://random-endpoint
        secretRef: af-aws-secret
        storageType: s3
        provider: aws
    appSources:
      - name: af
        location: af-test/
        volumeName: volume_app_repo_aws
        scope: local
      - name: s7
        location: af-test/
        volumeName: volume_app_repo_minio
        scope: local
      - name: admin-apps
        location: admin-apps/
        volumeName: volume_app_repo_minio
        scope: local

Validate

jryb:AppFramework$ kex-sc splunk-s-af-mixed-standalone-0 splunk
[splunk@splunk-s-af-mixed-standalone-0 splunk]$ ls -Rl /init-apps/
/init-apps/:
total 12
drwxr-sr-x 2 splunk splunk 4096 May 26 18:57 admin-apps
drwxr-sr-x 2 splunk splunk 4096 May 26 18:57 af
drwxr-sr-x 2 splunk splunk 4096 May 26 18:57 s7

/init-apps/admin-apps:
total 4
-rw-r--r-- 1 splunk splunk 1148 May 26 18:50 application2.tgz

/init-apps/af:
total 4
-rw-r--r-- 1 splunk splunk 1154 May 24 17:21 application5.tgz

/init-apps/s7:
total 4
-rw-r--r-- 1 splunk splunk 1167 May 26 18:50 application3.tgz
[splunk@splunk-s-af-mixed-standalone-0 splunk]$

pkg/apis/enterprise/v1/licensemaster_types.go

pkg/splunk/client/minioclient.go

pkg/splunk/enterprise/util.go

sgontla · 2021-05-25T18:47:52Z

pkg/splunk/client/awss3client.go

+
+// GetInitContainerCmd returns the init container command on a per app source basis to be used by the initContainer
+func (awsclient *AWSS3Client) GetInitContainerCmd(endpoint string, bucket string, path string, appSrcName string, appMnt string) []string {
+	return ([]string{fmt.Sprintf("--endpoint-url=%s", endpoint), "s3", "sync", fmt.Sprintf("s3://%s/%s", bucket, path), fmt.Sprintf("%s/%s", appMnt, appSrcName) /*, "--include \"*.tgz\"", "--include \"*.spl\""*/})


Info: Just for my understanding, does s3 sync utility uses the keys from the environment?

Yes. It pulls the creds from the env or command line (if specified, which they aren't here)

pkg/splunk/client/awss3client.go

pkg/splunk/client/minioclient.go

pkg/splunk/client/s3client.go

gaurav-splunk · 2021-05-26T14:21:56Z

pkg/splunk/enterprise/util.go

+				Image:           s3Client.GetInitContainerImage(),
+				ImagePullPolicy: "IfNotPresent",
+				Name:            initContainerName,
+				Args:            s3Client.GetInitContainerCmd(appS3Endpoint, appBkt, appPath, appName, appBktMnt),


nit: Any particular reason for using these local variables or we could have directly used the member variables? :)

Purely for readability and portability. I can switch it if you like or just let the compiler handle it.

pkg/splunk/client/minioclient.go

pkg/splunk/enterprise/clustermaster.go

pkg/splunk/enterprise/util.go

smohan-splunk · 2021-05-27T16:25:34Z

@jryb In the Unit Test examples, I see the old apiVersion apiVersion: enterprise.splunk.com/v1beta1 still being used. If not already done so, can we test with v1 too please?

jryb · 2021-05-27T17:15:30Z

@jryb In the Unit Test examples, I see the old apiVersion apiVersion: enterprise.splunk.com/v1beta1 still being used. If not already done so, can we test with v1 too please?

I've retest with the v1 apis as well and it passed.

pkg/splunk/enterprise/clustermaster.go

Add minio client to S3Client Add an init container image and command for each S3Client Add initContainer to Pod Spec on pre appSource Add init-apps volume to all init container and Splunk container

sgontla reviewed May 25, 2021

View reviewed changes

gaurav-splunk reviewed May 26, 2021

View reviewed changes

sgontla approved these changes May 26, 2021

View reviewed changes

jryb force-pushed the feature/CSPL-821-app_initContainer branch from 6055b68 to ac2e364 Compare May 26, 2021 22:34

gaurav-splunk reviewed May 26, 2021

View reviewed changes

pkg/splunk/client/minioclient.go Show resolved Hide resolved

smohan-splunk approved these changes May 27, 2021

View reviewed changes

pkg/splunk/enterprise/clustermaster.go Show resolved Hide resolved

pkg/splunk/enterprise/util.go Show resolved Hide resolved

smohan-splunk approved these changes May 27, 2021

View reviewed changes

pkg/splunk/enterprise/clustermaster.go Show resolved Hide resolved

gaurav-splunk approved these changes May 28, 2021

View reviewed changes

CSPL-821 Implement Init Container Image

733e37d

Add minio client to S3Client Add an init container image and command for each S3Client Add initContainer to Pod Spec on pre appSource Add init-apps volume to all init container and Splunk container

jryb force-pushed the feature/CSPL-821-app_initContainer branch from 2f53cc4 to 733e37d Compare May 28, 2021 16:54

smohan-splunk merged commit 21972eb into feature-appframework May 28, 2021

CSPL-821 Implement Init Container Image #350

CSPL-821 Implement Init Container Image #350

Uh oh!

Conversation

jryb commented May 24, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Solution

Unit Test

Minio

CR

Validate

AWS

CR

Validate

Mixed Minio and AWS

CR

Validate

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sgontla May 25, 2021

Choose a reason for hiding this comment

Uh oh!

jryb May 26, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gaurav-splunk May 26, 2021

Choose a reason for hiding this comment

Uh oh!

jryb May 26, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

smohan-splunk commented May 27, 2021

Uh oh!

jryb commented May 27, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jryb commented May 24, 2021 •

edited

Loading

jryb commented May 27, 2021 •

edited

Loading