Monitoring console broken in complex installations

[ryanfaircloth]

Because the deployment for MC is auto-generated the admin can not pass critical information into the template such as licensemasterref

Looking at the pod deployment it looks like this would not work correctly if there was more than one indexer cluster in the deployment in two ways the admin can not control which indexer cluster is used for output and it can not define multiple indexer clusters for search. The ask is as follows
Define a new CRD for Monitoring Console that also uses licenseref and indexerref
on each defined CR i.e cluster master add a mcref field to specify which monitoring console should monitor this deployment.
Ensure the ref syntax support namespace/object pattern such that MC can monitor across namepsaces

[romain-bellanger]

Hello,
@rfaircloth-splunk I think this is related to the problem which you describe as the solution is probably the same, but please let me know if you prefer this to be documented in a separate ticket.

Problem description: "unusable filesystem"

This issue has other impacts: no volumes are defined on monitoring console pods, files created or change in etc and var are written directly in the container's layer. I think the impact depends on the used Docker storage driver, but on our side the containers root FS is of type overlay, which doesn't seem to be supported by Splunk:

oc rsh splunk-splunk-dev-monitoring-console-0 df -hT var
Filesystem     Type     Size  Used Avail Use% Mounted on
overlay        overlay   60G  8.4G   52G  14% /

At startup Splunk fails to start and reports unusable filesystem:

STDERR:

homePath='/opt/splunk/var/lib/splunk/audit/db' of index=_audit on unusable filesystem.
Validating databases (splunkd validatedb) failed with code '1'.  If you cannot resolve the issue(s) above after consulting documentation, please file a case online at http://www.splunk.com/page/submit_issue

As described in this Splunk answers ticket, patching the splunk-launch.conf before Splunk starts allows the pod to start successfully.

echo 'echo "OPTIMISTIC_ABOUT_FILE_LOCKING = 1" >>etc/splunk-launch.conf' | oc rsh splunk-splunk-dev-monitoring-console-0

However that's not persistent so monitoring console fails again at every restart of the pod. It's not possible to use the ansible parameter splunk.launch to make this setting persistent as the SPLUNK_DEFAULTS_URL variable of the monitoring console can be inherited from the spec of any custom resource in the current solution.

Proposed solutions

As a short term solution, to solve this specific problem, the StatefulSet of the monitoring console could be modified to mount emptyDir volumes to /opt/splunk/etc and /opt/splunk/var.

But in general the current solution to define the specs of the monitoring console is not satisfactory. The reconciliation of any Splunk custom resource triggers an update of the StatefulSet of the monitoring console based on its own specs. So when the pod is deleted, it can come back with the specs (resource quotas, affinity, ansible config) of a cluster master, searchhead cluster, indexer cluster... depending on the last operation performed on other custom resources.

I also think that a new CRD should be created for the monitoring console, allowing to define its resource quotas, affinity and storage class and size.

[romain-bellanger]

Issue with multisite setup

Another impact on multi-site cluster with usage of defaultsUrl: if the creation of the monitoring console is triggered by the setup of a cluster master, it inherits its ansible configuration which contains the property multisite_master: localhost (as described in the documentation). Ansible then tries to connect the monitoring console to itself as a cluster-master. Other solutions exist, but this would also be solved by a dedicated CRD.

Support app load

Monitoring console should also support Splunk app load (through ansible splunk.apps_location) which is not the case today, for instance to allow configuring LDAP authentication.

[kashok-splunk]

Hi @ryanfaircloth, @romain-bellanger this issue should be resolved after Splunk Operator 1.1.0 release, which contains separate CRD for MC. Please let us know if there still exists any other issues.