Introduce authorization for enrich in ESQL

[dnhatn]

This change introduces a new privilege monitor_enrich. Users are required to have this privilege in order to use the enrich functionality in ESQL. Additionally, it eliminates the need to use the enrich_origin when executing enrich lookups. The enrich_origin will only be used when resolving enrich policies to prevent warnings when accessing system indices directly.

Closes #98482

[elasticsearchmachine]

Pinging @elastic/es-ql (Team:QL)

[elasticsearchmachine]

Pinging @elastic/elasticsearch-esql (:Query Languages/ES|QL)

[elasticsearchmachine]

Pinging @elastic/es-security (Team:Security)

[ywangd]

@dnhatn I am no longer on the security team. So I'll ping @jakelandis for a formal review.

Briefly looking through the changes, I think the general approach makes sense to me. We might need to tweak exactly what privilege is required for resolving the enrich policy. Right now it is monitor which feels a bit off to me. Existing manage_enrich cluster privilege does not seem to fit either. We might need add a new and more specific read level cluster privilege for this purpose. That said, I'll defer to Jake for concrete suggestions. Thanks!

[dnhatn]

@ywangd No problem. Thank you for taking a look. I've requested @jakelandis to take a look at the PR.

[jakelandis]

Do I understand this correctly ?:

Before: any user could use the enrich key word which allows the "indices:data/read/esql/lookup" by using the ClientHelper.ENRICH_ORIGIN (which in turn resulted in executing as the (InternalUsers.XPACK_USER user)

With this PR: we no longer user the ClientHelper.ENRICH_ORIGIN which means the user must be explicitly granted permissions. Specifically the user now need index:read (or explicitly named: indices:data/read/esql/lookup) for the enrich policy name and cluster:monitor (or explicitly named: cluster:monitor/esql/resolve_enrich) to order use the use the 'enrich' keyword. The new cluster:monitor is needed since the enrich indices are system (or at least hidden) indices and the concrete index names are implementation details. So this is needed map the friendly name of the enrich policy to the underlying index name.

Assuming I understand the change correctly (I may not!) ... it feels odd that we are applying index level privileges on the enrich policy name. I worry that it violates our expectations w.r.t. applying privileges for indices/aliases/data streams. I am also kinda surprised this actually works since for index level permissions since we iterate through the known indices, datastreams, and aliases, compare them against what is in the role and then rewrite the indices that are requests omiting the ones you don't have access to. I am failing to see how that works since that security logic does not take the enrich resolution. However, I also am not familiar with how ESQL works and feels like I am missing a key piece of information to understand how this works (is that what the messageRecieved override is doing ?...then how is the comparison against the friendly name working ?). A better understanding here might help with these technical concerns. However, there is still a non-technical concern w.r.t. to placing non-index abstractions (alias, datastream, index name) in the indices name of the role. For example, I can not query "test-enrich" with the normal _search API and am concerned this could lead to confusion for what is 'indices.names'... how does this work with Kibana UIs ? how do users know to put the policy name in there ? and the eventual questions for why does it not also work with enrich via ingest node pipelines or policy execution.

I also have concerns w.r.t. normal users that just want to do some queries needing the cluster:monitor privilege. That cluster privilege is intended to be a (lower) admin level set of privileges. There probably isn't too much of pure security concern with giving normal users that access in the scope that it does not leak credentials or the like... but doing so IMO it violates least privilege. It doesn't seem right that to use enrich queries I also have to give users permission to see things like autoscaling stats or ml jobs or ccr stats, etc. Being overly restrictive can have the opposite effect in that some admins will just give wider permissions (lowering security) to users that don't need it just to accomplish the goal without fully understanding the implications. Also, I don't believe that any normal users in serverless will ever have this permission.

Ideally we would have a permission model for enrich that is applicable to both ESQL, ingest nodes, and policy execution that does not require users to know to how to configure indices permissions for things that are not index abstractions (as defined by being able to _search on those names). This probably looks like a new field in the role 'enrich.names' where the enrich names are treated like abstractions such as date math or aliases that get resolved to the allowed list of concrete index names in the security (and non-security) index resolution. That model would conform to the existing expectations be accessible to ingest node / policy execution workflows and IMO better progresses enrich as concept.

Ultimately you are hitting on our lack of resource specific permissions and there might be some less involved short term solutions. However, before proposing any alternatives I would want to double check my understanding and better understand the requirements. Those conversations probably extend beyond this PR.

[tvernum]

@jakelandis If I understand your comment correctly, I think you might have been mislead by the name of the index in the tests.

test-enrich is a normal index, that is used as part of the testing of the enrichment command.
The enrich index and policy are called songs.

So the ES|QL command is

FROM test-enrich | ENRICH songs ON song_id | stats total_duration = sum(duration) by artist | sort artist

Which (more or less) translates to

1. Read from the test-enrich index
2. Apply the song enrich policy to it, joining on song_id
3. sum duration by artist
4. sort by artist

The security change is in how step 2 is applied.

• Previously every user would be able to perform the enrich step (against any policy).
• After this change, the ability to perform an ES|QL enrich is restricted by a cluster privilege (currently proposed to bemonitor, but I agree that it's not a great fit). No specific index level privilege is required.

The change to the security setup in roles.yml consists of

1. Granting monitor to the existing user (user1) who should be allowed to perform enrich (and previously could because no privilege was required)
2. Creating a read-only user (user4) who should be allowed to run the ES|QL command. (This is helpful because user1 has additional privileges and we'd like to know that the command works without write etc).
3. Creating another read-only user (user5) who does not have monitor and, therefore, should not be allowed to run the ES|QL command.

None of those users have access to anything named songs (neither the source index, nor the enrich policy). This works because the action is cluster:monitor/esql/resolve_enrich, so it is authorized as a cluster monitor action.
It's not really an "action" though - it's all handled as a lightweight transport request handler inside EnrichPolicyResolver.

Interestingly, despite the PR description, the actual enrichment still happens using the enrich origin which explains why it works

• elasticsearch/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/enrich/EnrichPolicyResolver.java

  Lines 114 to 122 in 672258b

  	try (ThreadContext.StoredContext ignored = threadContext.stashWithOrigin(ClientHelper.ENRICH_ORIGIN)) {
  	indexResolver.resolveAsMergedMapping(
  	EnrichPolicy.getBaseName(policyName),
  	IndexResolver.ALL_FIELDS,
  	false,
  	Map.of(),
  	listener.map(indexResult -> new ResolveResponse(new EnrichPolicyResolution(policyName, policy, indexResult)))
  	);
  	}

As @dnhatn has indicated, the ideal outcome would be to have a way to limit which enrich polices a user may use within ES|QL. It looks like we could probably do that with a ConfigurableClusterPrivilege if we wanted. It would require a little bit of rework of this PR (e.g. extracting a EnrichPolicyRequest interface), but I can't see any technical reason why it couldn't be done.

[dnhatn]

@jakelandis @tvernum Thank you so much for looking.

Tim has summarized this perfectly. In the enrich lookup in ESQL, there are two actions:

1. Resolve enrich policy:
   Action (cluster:monitor/esql/resolve_enrich): This action doesn't access its Lucene index but only the enrich policies in the cluster state and its index metadata via the field-caps API. Previously, this action didn't require any access permission, meaning any user could access the enrich policies. This PR now requires users to have a 'monitor' permission to perform this step. We can consider introducing a new privilege for enrich-policy-read. Another question is how to integrate this privilege with the enrich ingest without causing any breaking changes. We will still perform the resolution of enrich policies under the 'enrich_origin' to avoid deprecation warnings when accessing the system indices directly.

2. Lookup enrich data:
   Action (indices:data/read/esql/lookup): This action accesses the Lucene index and performs the actual enrich. This action requires the user to have the index-read permission for the enrich index. This action will no longer execute under the 'enrich_origin' after this PR.

I believe we don't require any extra privilege for ingest-enrich.

[jakelandis]

Do I understand this correctly ?

...no I did not. Thanks Tim and Nhat for the clarifications. One last clarification:

This action (indices:data/read/esql/lookup) requires the user to have the index-read permission for the enrich index.

You mean the index being enriched... right ? The enrich index is defined here as "A special system index tied to a specific enrich policy". Sorry to be pedantic, but confusing the security requirements of two was the source of my original confusion.

Assuming we are all on the same page... The only relevant part of the original comment are :

Ideally we would have a permission model for enrich that is applicable to both ESQL, ingest nodes, and policy execution

While ideal that would be quite involved. An implementation with ConfigurableClusterPrivilege could set us on that path but we would probably want to flesh out the requirements more and consider other options. However, I do think the simpler cluster wide approach is viable and (with a correct understanding of the change) is in line with the existing model even it is not terribly flexible/expressive.

I also have concerns w.r.t. normal users that just want to do some queries needing the cluster:monitor privilege

While it possible to configure a role with the string cluster:monitor/esql/resolve_enrich we generally discourage this (except for tightly controlled internal usecases) since we consider those names implementation details. In practice we would advocate to use the named privileges such as 'monitor' .. as already mentioned that is likely too wide for this usecase. Can we use an ESQL specific named privilege... or change the action name to 'cluster:monitor/xpack/enrich/esql/resolve_enrichand introduce amonitor_enrichwith a pattern ofcluster:monitor/xpack/enrich/*` which would also include the ability to read enrich stats. There is always a friction between grouping raw privileges into named privileges lower vs. least privileges. I would advocate for use an EQSL name if you think there will be more than just 1 in the near future, else I would advocate to group in with a new 'monitor_enrich'.

[dnhatn]

Can we use an ESQL specific named privilege... or change the action name to 'cluster:monitor/xpack/enrich/esql/resolve_enrichand introduce amonitor_enrichwith a pattern ofcluster:monitor/xpack/enrich/*` which would also include the ability to read enrich stats.

@jakelandis Thanks so much for your suggestion. It makes sense to me. I will update this PR to introduce a new privilege.

[dnhatn]

@jakelandis I've introduced the monitor_enrich privilege as you suggested in 216d267. I should have also introduced enrich_user but we already have it for manage_enrich. Would you please take another look? Thank you!

[jakelandis]

LGTM - however, can you update the doc here either as part of this PR or as a followup.

Thanks again for clarifications and apologies for the noise.

[dnhatn]

@jakelandis Thank you for the review + suggestion :).

however, can you update the doc here either as part of this PR or as a followup.

Sure, I will do this in a follow-up.