Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DOCS][ESQL][8.14] Add API key based security model info for ESQL CCS #109155

Merged
merged 9 commits into from
Jun 3, 2024
98 changes: 84 additions & 14 deletions docs/reference/esql/esql-across-clusters.asciidoc
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
[[esql-cross-clusters]]
=== Using {esql} across clusters

++++
<titleabbrev>Using {esql} across clusters</titleabbrev>
++++
Expand All @@ -11,6 +10,8 @@ preview::["{ccs-cap} for {esql} is in technical preview and may be changed or re

With {esql}, you can execute a single query across multiple clusters.

[discrete]
[[esql-ccs-prerequisites]]
==== Prerequisites

include::{es-ref-dir}/search/search-your-data/search-across-clusters.asciidoc[tag=ccs-prereqs]
Expand All @@ -19,6 +20,70 @@ include::{es-ref-dir}/search/search-your-data/search-across-clusters.asciidoc[ta

include::{es-ref-dir}/search/search-your-data/search-across-clusters.asciidoc[tag=ccs-proxy-mode]

[discrete]
[[esql-ccs-security-model]]
==== Security model

{es} supports two security models for cross-cluster search (CCS):

* <<esql-ccs-security-model-certificate, TLS certificate authentication>>
* <<esql-ccs-security-model-api-key, API key authentication>>

[discrete]
[[esql-ccs-security-model-certificate]]
===== TLS certificate authentication

TLS certificate authentication makes sense in single administrator scenarios where you have full control over both clusters.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
TLS certificate authentication makes sense in single administrator scenarios where you have full control over both clusters.
TLS certificate authentication secures remote clusters with mutual TLS and is available under the basic license. This may be the preferred model where a single administrator has full control over both clusters.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we talk about licenses explicitly in docs given variance between self-managed and cloud?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know our general stance, but it seems like an important detail since ES|QL is not gated by an enterprise license. IIUC in cloud user have access to all features, but are billed based on usage.

Copy link
Contributor Author

@leemthompo leemthompo May 31, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We use generic messaging about licensing as a rule, and the main pages for these security models have prerequisites sections that say stuff like "The local and remote clusters must have an appropriate license. For more information, refer to https://www.elastic.co/subscriptions". So I will refrain from mentioning licensing in this doc.

Importantly, you must ensure that named roles and their privileges are identical in both clusters.
leemthompo marked this conversation as resolved.
Show resolved Hide resolved
Refer to <<remote-clusters-cert, TLS certificate authentication>> for prerequisites and detailed setup instructions.

[discrete]
[[esql-ccs-security-model-api-key]]
===== API key authentication
leemthompo marked this conversation as resolved.
Show resolved Hide resolved

[NOTE]
====
`ENRICH` is *not supported* in this version when using {esql} with the API key based security model.
====

The following information pertains to using {esql} across clusters with the <<remote-clusters-api-key, *API key based security model*>>.
API key based cross-cluster search (CCS) enables more granular control over allowed actions between clusters.
leemthompo marked this conversation as resolved.
Show resolved Hide resolved
As a result, defining roles and privileges is more complex than with the certificate based security model.
leemthompo marked this conversation as resolved.
Show resolved Hide resolved
For example, it makes sense in scenarios when you have different administrators for different clusters.
leemthompo marked this conversation as resolved.
Show resolved Hide resolved

You will need to create an API key using the <<security-api-create-cross-cluster-api-key,Create cross-cluster API key>> API or using the {kibana-ref}/api-keys.html[Kibana API keys UI].
leemthompo marked this conversation as resolved.
Show resolved Hide resolved

The following example API call creates a role that can query remote indices using {esql} with the API key based security model.
leemthompo marked this conversation as resolved.
Show resolved Hide resolved

[source,console]
----
POST /_security/role/remote1
{
"cluster": ["cross_cluster_search"], <2>
leemthompo marked this conversation as resolved.
Show resolved Hide resolved
"indices": [
{
"names" : [""], <3>
"privileges": ["read"]
}
],
"remote_indices": [ <4>
{
"names": [ "logs-*" ],
leemthompo marked this conversation as resolved.
Show resolved Hide resolved
"privileges": [ "read","read_cross_cluster" ], <1>
"clusters" : ["my_remote_cluster"] <5>
}
]
}
----

<1> The `read_cross_cluster` privilege is always required when using {esql} across clusters with the API key based security model.
<2> The `cross_cluster_search` cluster privilege is also required for the _local_ cluster.
<3> Typically, users will have permissions to read both local and remote indices. However, for cases where the role is intended to ONLY search the remote cluster, the `read` permission is still required for the local cluster. To provide read access to the local cluster, but disallow reading any indices in the local cluster, the `names` field may be an empty string.
<4> The indices allowed read access to the remote cluster. The configured <<security-api-create-cross-cluster-api-key,cross-cluster API key>> must also allow this index to be read.
<5> The remote clusters to which these privileges apply. This remote cluster must be configured with a <<security-api-create-cross-cluster-api-key,cross-cluster API key>> and connected to the remote cluster before the remote index can be queried.
Verify connection using the <<cluster-remote-info, Remote cluster info>> API.


[discrete]
[[ccq-remote-cluster-setup]]
==== Remote cluster setup
Expand Down Expand Up @@ -71,13 +136,18 @@ FROM *:my-index-000001
Enrich in {esql} across clusters operates similarly to <<esql-enrich,local enrich>>.
If the enrich policy and its enrich indices are consistent across all clusters, simply
write the enrich command as you would without remote clusters. In this default mode,
{esql} can execute the enrich command on either the querying cluster or the fulfilling
{esql} can execute the enrich command on either the local cluster or the remote
clusters, aiming to minimize computation or inter-cluster data transfer. Ensuring that
the policy exists with consistent data on both the querying cluster and the fulfilling
the policy exists with consistent data on both the local cluster and the remote
clusters is critical for ES|QL to produce a consistent query result.

[NOTE]
====
Enrich across clusters is *not supported* in this version when using {esql} with the <<remote-clusters-api-key, *API key based security model*>>.
====

In the following example, the enrich with `hosts` policy can be executed on
either the querying cluster or the remote cluster `cluster_one`.
either the local cluster or the remote cluster `cluster_one`.

[source,esql]
----
Expand All @@ -87,8 +157,8 @@ FROM my-index-000001,cluster_one:my-index-000001
----

Enrich with an {esql} query against remote clusters only can also happen on
the querying cluster. This means the below query requires the `hosts` enrich
policy to exist on the querying cluster as well.
the local cluster. This means the below query requires the `hosts` enrich
policy to exist on the local cluster as well.

[source,esql]
----
Expand All @@ -99,10 +169,10 @@ FROM cluster_one:my-index-000001,cluster_two:my-index-000001

[discrete]
[[esql-enrich-coordinator]]
==== Enrich with coordinator mode
===== Enrich with coordinator mode

{esql} provides the enrich `_coordinator` mode to force {esql} to execute the enrich
command on the querying cluster. This mode should be used when the enrich policy is
command on the local cluster. This mode should be used when the enrich policy is
not available on the remote clusters or maintaining consistency of enrich indices
across clusters is challenging.

Expand All @@ -118,21 +188,21 @@ FROM my-index-000001,cluster_one:my-index-000001
[IMPORTANT]
====
Enrich with the `_coordinator` mode usually increases inter-cluster data transfer and
workload on the querying cluster.
workload on the local cluster.
====

[discrete]
[[esql-enrich-remote]]
==== Enrich with remote mode
===== Enrich with remote mode

{esql} also provides the enrich `_remote` mode to force {esql} to execute the enrich
command independently on each fulfilling cluster where the target indices reside.
command independently on each remote cluster where the target indices reside.
This mode is useful for managing different enrich data on each cluster, such as detailed
information of hosts for each region where the target (main) indices contain
log events from these hosts.

In the below example, the `hosts` enrich policy is required to exist on all
fulfilling clusters: the `querying` cluster (as local indices are included),
remote clusters: the `querying` cluster (as local indices are included),
the remote cluster `cluster_one`, and `cluster_two`.

[source,esql]
Expand All @@ -157,12 +227,12 @@ FROM my-index-000001,cluster_one:my-index-000001,cluster_two:my-index-000001

[discrete]
[[esql-multi-enrich]]
==== Multiple enrich commands
===== Multiple enrich commands

You can include multiple enrich commands in the same query with different
modes. {esql} will attempt to execute them accordingly. For example, this
query performs two enriches, first with the `hosts` policy on any cluster
and then with the `vendors` policy on the querying cluster.
and then with the `vendors` policy on the local cluster.

[source,esql]
----
Expand Down