CSPL-1206: Introducing the App Source scope clusterWithPreConfig

[sgontla]

Support for new App Source scope clusterWithPreConfig

[sgontla]

With the changes listed here in this PR, first-time ES app installation works fine. However, when the Deployer Pod resets for any reason(say, for example, a new app added or existing app modified), then it is getting into crashLoopBackoff.

When the new deployer Pod is coming up, Ansible playbooks validate all the installed apps. In case if the ES app is available, it runs the post set up playbooks, which is failing with the following error

TASK [splunk_deployer : Run ESS post-install setup] ****************************
fatal: [localhost]: FAILED! => {
    "changed": false,
    "cmd": [
        "/opt/splunk/bin/splunk",
        "search",
        "| essinstall --ssl_enablement ignore --deployment_type shc_deployer",
        "-auth",
        "admin:2YM3giXfE2XS0BaYWpcHwu5V"
    ],
    "delta": "0:00:00.583384",
    "end": "2021-07-27 16:55:38.110817",
    "failed_when_result": true,
    "rc": 1,
    "start": "2021-07-27 16:55:37.527433"
}

STDERR:

Unknown search command 'essinstall'.


MSG:

non-zero return code

Had a discussion with Jeff and Julien, and we all believe the command eesinstall is not recognized anymore, as the app was already installed, configured, and deployed to the SHC members. (Ideally, this should be the same issue even with the traditional way of deploying the ES app), and is outside the scope of the Operator.

To work around this behavior on the Ansible side added the following fix, and it works fine, for all the Pod reset scenarios. This is just to make sure that the ES installation works in the big picture of the Operator framework, and not the final fix at this time, as we may miss the genuine error cases.

sgontla-MBP-8CAF6:splunk-ansible sgontla$ git diff roles/splunk_common/tasks/premium_apps/configure_ess.yml
diff --git a/roles/splunk_common/tasks/premium_apps/configure_ess.yml b/roles/splunk_common/tasks/premium_apps/configure_ess.yml
index 5bddbac..d978c46 100644
--- a/roles/splunk_common/tasks/premium_apps/configure_ess.yml
+++ b/roles/splunk_common/tasks/premium_apps/configure_ess.yml
@@ -19,6 +19,6 @@
   become_user: "{{ splunk.user }}"
   register: essinstall
   changed_when: essinstall.rc == 0
-  failed_when: essinstall.rc != 0
+  failed_when: essinstall.rc != 0 and 'Unknown search command' not in essinstall.stderr
   notify:
     - Restart the splunkd service
sgontla-MBP-8CAF6:splunk-ansible sgontla$