Multi-tenanted external ElasticSearch

[chrismccracken]

In order to efficiently use resources with an external ElasticSearch cluster, we need the ability for multiple dotCMS clusters to share an ElasticSearch cluster with private per-cluster namespacing. This should be hidden to the dotCMS Administrator, ie they should not have visibility that the Elastic cluster is shared and access should be isolated so that a given dotCMS cluster can only access and manage its own indexes.

A suggested pattern for enabling this would be to intercept all index API calls and inject a prefix into the index name, and stripping that prefix out when displaying in the dotCMS application (transparent to the app admins and developers). This prefix could use the existed unique CLUSTERID, which can also be set as a configuration parameter.

Example architecture:

[chrismccracken]

Also re: https://github.com/dotCMS/enterprise-2.x/pull/314

[chrismccracken]

Also need to take into consideration any non-obvious tools that access ElasticSearch:

• Site Search
• Viewtools (Site Search and ES tool)

Anything that grants access to an ES index should control visibility to only the ES namespace for it's own cluster

[wezell]

We should use the shorty value of the cluster id to prefix the index names.
This needs to be fully hidden from dotCMS users - meaning, they should not be able to call elasticsearch with those prefixes being applied.
We should pay attention to importing and exporting index snapshots as well - it is ok if the clusterId is prefixed when downloading or creating a snapshot, as long as we insure that it is always prefixed on any attempts to import a snapshot.

[freddyucv]

For QA: Bug in Site Search.
Steps:

1. Have two dotCMS running in different port with different DB
2. Go to "Dev Tools/Site Search" in Node 1, and . create a new "Site Search Index"
   3._ Go to "Dev Tools/Site Search" in Node 1 and you can see the Serach Index created in the first node.

[freddyucv]

For QA: Bug in Index tab in Maintanence
Steps:

1. Have two dotCMS running in different port with different DB
2. Go to "Maintenence" and Index tab in Node 1, yo here can see the Node 2's Index.
3. Go to "Maintenence" and Index tab in Node 2, yo here can see the Node 3's Index.

[wezell]

Let's read the links in this convo:
https://stackoverflow.com/questions/41868056/multi-tenancy-in-elastic-search

[wezell]

Note to Doc:

All indexes will now be prefixed with the value of the dotCMS Cluster Id. This value will be autogenerated, though it can be set in the dotmarketing-config.properties
as DOTCMS_CLUSTER_ID=xxxxx
It can also be specified at runtime as an environmental property export DOTCMS_CLUSTER_ID=xxxxx . or a system.property, e.g. -DDOTCMS_CLUSTER_ID=xxxxx

This change doesn't affect UI, however in a ES server, indexes and aliases are stored with the cluster id prefix. Now, indexes and aliases are stored using this format: "cluster"_{cluster_id}.{index_or_alias}

[nollymar]

PRs to merge into issue-15942
Core: #17746
EE: https://github.com/dotCMS/enterprise-2.x/pull/696

[stale]

This issue has been automatically marked as stale because it has not had activity within the past 90 days. It will be closed in 30 days no further activity occurs. Thank you.

[dsilvam]

Tests are covered in this issue: #18046

[dsilvam]

Passed both Internal and Full QA. Tested here: #18046