CSPL-663 Optimize deployment of Splunk apps on SHC

[jryb]

Problem

As part of #126 it mentions an optimization for installing apps on a SHC since there is not separate CRD for the deployer.

On SearchHeadClusters CRDs, the defaultsUrl property of the CRD is shared by the deployer and the search-heads. When the defaultsUrl is updated with new applications, this would trigger a rolling rebuild of the search-head pods and deployer. However the deployer would trigger an apply shcluster-bundle, potentially triggering a rolling restart of the searchhead cluster in parallel. Rebuilding the SH pods is not necessary and can lengthen the downtime of the the SHC.

This problem does not exist on the IndexerCluster CRD since the applications are handled by the cluster-master which has a dedicated CRD. So the defaultsUrl from indexers in the cluster can be different. If a separate CRD is implement for the SHC Deployer, this would be the same case for SHC as well. However until that CRD is created, this problem exists.

Solution

Add a defaultsUrlApps parameter to the CRDs for Standalone, IndexerClusters, and SearchHeadClusters. This parameter would be used specifically for app installs and used on Standalones, SH Deployer, and IDC CM only. Individual SHs and indexers in clusters would ignore this parameter. This would allow the applications for a clustered env to be changed without rebuilding the SH member pods. The SH Deployer would then push out the bundle and install the apps on all SH members.

This could have been implement specific to SH Deployer (since the SHC is the only deployment with this issue), however keeping it more generic allows for future optimizations, potentially installing apps without the need pod rebuilds.

Testing

Create a SHC using a defaults.yml in defaultsUrlApps:

shc.yml:

apiVersion: enterprise.splunk.com/v1beta1
kind: SearchHeadCluster
metadata:
  name: shc1
  finalizers:
  - enterprise.splunk.com/delete-pvc
spec:
  clusterMasterRef:
    name: cm1
  volumes:
    - name: apps
      configMap:
        name: splunk-apps
  defaultsUrlApps: /mnt/apps/defaults_apps.yml

NAME                                  READY   STATUS    RESTARTS   AGE
splunk-cm1-cluster-master-0           1/1     Running   0          23m
splunk-default-monitoring-console-0   1/1     Running   0          17m
splunk-lm1-license-master-0           1/1     Running   0          23m
splunk-operator-7d868fb689-mqtrb      1/1     Running   0          27m
splunk-shc1-deployer-0                1/1     Running   0          22m
splunk-shc1-search-head-0             1/1     Running   0          22m
splunk-shc1-search-head-1             1/1     Running   0          22m
splunk-shc1-search-head-2             1/1     Running   0          22m

Then adjust the defaultsUrlApps parameter and verify that only the shc deployer gets rebuilt and that the correct apps from the respective defaults_apps.yml are installed.

  defaultsUrlApps: /mnt/apps/defaults_apps2.yml

Apply changes and validate only deployer is rebuilt:

NAME                                  READY   STATUS    RESTARTS   AGE
splunk-cm1-cluster-master-0           1/1     Running   0          36m
splunk-default-monitoring-console-0   1/1     Running   0          31m
splunk-lm1-license-master-0           1/1     Running   0          36m
splunk-operator-7d868fb689-mqtrb      1/1     Running   0          41m
splunk-shc1-deployer-0                1/1     Running   0          100s
splunk-shc1-search-head-0             1/1     Running   0          36m
splunk-shc1-search-head-1             1/1     Running   0          36m
splunk-shc1-search-head-2             1/1     Running   0          36m

[smohan-splunk]

So if I have understood this right, to install apps on Standalones, CM & Deployer, setup defaultsUrlApps

• Am I right in saying that on LM, we still need to use defaultsUrl? For consistency purposes, should LM also be included?
• If both defaultsUrlApps & defaultsUrl are configured with apps_locaton, wonder how Ansible would treat this? I am guessing it would be much more efforts to expect operator to be smart enough to parse & detect this duplicate, & strip off one of the duplicates. Completely dropping apps_location from defaultsUrl would also need parsing logic.
• We should document the new argument & best practices in Examples.md

[jryb]

Am I right in saying that on LM, we still need to use defaultsUrl? For consistency purposes, should LM also be included?

So currently yes, but I can add LM to this list as well. Or probably better, just exclude Indexer and SearchHead types since they will always be in the cluster.

If both defaultsUrlApps & defaultsUrl are configured with apps_locaton, wonder how Ansible would treat this? I am guessing it would be much more efforts to expect operator to be smart enough to parse & detect this duplicate, & strip off one of the duplicates. Completely dropping apps_location from defaultsUrl would also need parsing logic.

I will test this and verifiy that ansible behaves correctly.

We should document the new argument & best practices in Examples.md

We can add this to the documentation. Though if we plan on making the Deployer its own CRD, we may not need this parameter. So it maybe a temporary fix and might not need documenting if it's going away soon. However we can pull the docs once that occurs. I'll make the change.

[jryb]

Validated in testing the use of multiple defaults.yml files works correctly, including with overlapping apps.

[smohan-splunk]

Nitpick: Typo

[jryb]

Fixed

[smohan-splunk]

Nitpick: Just so that both statements are consistent

• For SHC, these applications will only be installed on the SHC Deployer and pushed to the members via SH Bundle push.

[jryb]

fixed

[smohan-splunk]

Nitpick: For standalone & License Master, the apps will be installed as normal.

[jryb]

fixed

[smohan-splunk]

Installing Applications via defaultsUrl, while continues to work, is not going to be a recommended practice anymore, right? Especially to avoid what this PR is fixing (the double pod resets).

[jryb]

Yes while technically you can still have apps in the defaultsUrl this would not be preferred but would be left in for backward compatibility. The new block would be:

For application installation the preferred method will be through the defaultsUrlApps
while other ansible defaults can be still be installed via defaultsUrl. For backwards
compatibility applications could be installed via defaultsUrl though this is not
recommended. Both options can be used in conjunction:

[kashok-splunk]

Look good :)

[jryb]

Squashing commits