Cross-project search index expression rewriting #135346

n1v0lg · 2025-09-24T12:25:06Z

This PR implements cross-project search index resolution and ports the resolve index API to use it.

Relates: ES-12690

…v0lg/elasticsearch into record-local-index-resolution-results

x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/SecurityExtension.java

x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/Security.java

slobodanadamovic

Authorization changes LGTM 👍
(left one question and suggestion which should not require another round of review)

… cps-flat-world

n1v0lg · 2025-10-05T19:36:50Z

...security/src/main/java/org/elasticsearch/xpack/security/authz/IndicesAndAliasesResolver.java

-                // we will always store them
-                if (recordResolvedIndexExpressions) {
+                assert indicesRequest.indices() != null : "indices() cannot be null when resolving non-all-index expressions";
+                if (crossProjectModeDecider.resolvesCrossProject(replaceable)


Perhaps this entire block can go into a new class CrossProjectIndicesResolver.

That class could then have the following methods, combining CrossProjectModeDecider, CrossProjectIndexExpressionsRewriter, and CrossProjectIndexResolutionValidator into one coherent class:

class CrossProjectIndicesResolver { boolean crossProjectEnabled() { ... } boolean resolvesCrossProject(Indices.Replaceable request) { ... } ResolvedIndices resolve(Indices.Replaceable request) { ... } ElasticsearchException validate( IndicesOptions indicesOptions, ResolvedIndexExpressions localResolvedExpressions, Map<String, ResolvedIndexExpressions> remoteResolvedExpressions ) { ... } }

I'd rather do that refactor in a follow up though.

I think this is somewhat related to @slobodanadamovic 's comment in other place about having IndicesAndAliasesResolver to be pluggable for CPS. I can see this being a more packaged and self-contained solution. But yeah let's defer it for the time being.

elasticsearchmachine · 2025-10-05T19:38:27Z

Pinging @elastic/es-security (Team:Security)

ywangd

LGTM

Thanks for the iterations!

server/src/main/java/org/elasticsearch/search/crossproject/TargetProjects.java

server/src/main/java/org/elasticsearch/cluster/metadata/IndexAbstractionResolver.java

ywangd · 2025-10-06T01:16:57Z

...security/src/main/java/org/elasticsearch/xpack/security/authz/IndicesAndAliasesResolver.java

        this.remoteClusterResolver = new RemoteClusterResolver(settings, linkedProjectConfigService);
-        this.recordResolvedIndexExpressions = recordResolvedIndexExpressions;
+        this.crossProjectModeDecider = crossProjectModeDecider;
+        this.recordResolvedIndexExpressions = crossProjectModeDecider.crossProjectEnabled();


Do we still need this field separately now that we have crossProjectModeDecider?

Good catch, no sorry we don't need it, just forgot to remove it

ywangd · 2025-10-06T01:20:12Z

...security/src/main/java/org/elasticsearch/xpack/security/authz/IndicesAndAliasesResolver.java

-                // we will always store them
-                if (recordResolvedIndexExpressions) {
+                assert indicesRequest.indices() != null : "indices() cannot be null when resolving non-all-index expressions";
+                if (crossProjectModeDecider.resolvesCrossProject(replaceable)


I think this is somewhat related to @slobodanadamovic 's comment in other place about having IndicesAndAliasesResolver to be pluggable for CPS. I can see this being a more packaged and self-contained solution. But yeah let's defer it for the time being.

ywangd · 2025-10-06T01:25:47Z

...security/src/main/java/org/elasticsearch/xpack/security/authz/IndicesAndAliasesResolver.java

+                    // we need an early return here, instead of relying on the outer none expression logic since the outer handling will
+                    // prematurely throw an IndexNotFound exception if the resolved indices are empty and allow_no_indices is false.
+                    if (resolvedIndicesBuilder.isEmpty()) {
+                        setNoneExpression(replaceable, resolvedIndicesBuilder);
+                    } else {
+                        replaceable.indices(resolvedIndicesBuilder.build().toArray());
+                    }
+                    return resolvedIndicesBuilder.build();


Do we still need this? It seems that we can have

if (indicesOptions.allowNoIndices() || crossProjectModeDecider.crossProjectEnabled())

at line 427 and it should work?

Yes, but crossProjectModeDecider.resolvesCrossProject(replaceable) instead -- otherwise, we don't correctly handle the case where CPS is enabled but we are handling a local request that did not get CPS resolved (e.g., because it does not allowCrossProject).

ywangd

A few more comments while reviewing the serveless PR.

...security/src/main/java/org/elasticsearch/xpack/security/authz/IndicesAndAliasesResolver.java

ywangd · 2025-10-06T01:50:39Z

server/src/main/java/org/elasticsearch/action/admin/indices/resolve/ResolveIndexAction.java

+                    // we still need to call response validation for local results, since qualified expressions like `_origin:index` or
+                    // `<alias-pattern-matching-origin-only>:index` get deferred validation, also


This sentense is not completed. Also, in this case, TargetProjects#linkedProjects should be empty? If that's the case, we might be able to short circuit it in IndicesAndAliasesResolver and avoid going into the application code. Something to look into in future.

Ah it's just very poorly phrase -- ", also" should be ", too" or ", as well" -- let me wordsmith it a bit

If that's the case, we might be able to short circuit it in IndicesAndAliasesResolver and avoid going into the application code.

Yup -- very much something to look into as an optimization once more of the core pieces are in place

I'll log a Jira for it

ywangd · 2025-10-06T01:55:04Z

server/src/main/java/org/elasticsearch/search/crossproject/TargetProjects.java

+import java.util.Set;
+import java.util.stream.Collectors;
+
+public record TargetProjects(@Nullable ProjectRoutingInfo originProject, List<ProjectRoutingInfo> linkedProjects) {


The nullability confused me a bit. It might worth adding a comment to say (IIUC) that null originProject is really a placeholder and only possible when a request is not eligible for CPS expansion after checking all three layers. It does not mean the user has no access to the origin project. In fact, we use a non-null ProjectRoutingInfo for CPS request even when the user has no access to the origin project, the locally accessible indices are resolved separately.

Discussed this sync: originProject can be null if the project was excluded by project routing. I added Javadoc to explain this in more detail.

n1v0lg and others added 30 commits September 15, 2025 12:09

WIP CPS authz service

11e955a

WIP injection

23ed699

Merge branch 'main' into cps-flat-world

ad8a9cb

Index resolutions record resolved expressions

ba24636

Smaller diff

dd1dbe0

Clean up

a363c19

Clean up

103e044

Another clean up round

17c50c7

Use a builder

8a5861c

Remove unused exclude

777a8bc

Nits

0474671

Bring back comment

cbe0bd4

Test ordering

8789cc4

Nits

820303b

Order and TODO

7ba1eec

Debug order

bb79a5c

Sets

aad05d6

Remote also sets

457b45b

Merge branch 'main' into record-local-index-resolution-results

013b1dd

Merge branch 'main' into record-local-index-resolution-results

a00b8cc

Merge branch 'main' into record-local-index-resolution-results

df37ed6

Tests

cb346e0

[CI] Update transport version definitions

665bbec

Tweaks

5ad94e4

Merge branch 'record-local-index-resolution-results' of github.com:n1…

845fd9f

…v0lg/elasticsearch into record-local-index-resolution-results

More

97c6efa

Merge branch 'main' into record-local-index-resolution-results

62bb00f

Toggle storage

add29c3

Test

1d0be73

Rm sysout

c15b0d7

n1v0lg and others added 5 commits October 3, 2025 10:54

Merge

7224d70

[CI] Auto commit changes from spotless

7240d89

Fix response code

395f3a0

Method

39a50a4

Naming nit

02e4693

slobodanadamovic reviewed Oct 3, 2025

View reviewed changes

x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/SecurityExtension.java Outdated Show resolved Hide resolved

slobodanadamovic reviewed Oct 3, 2025

View reviewed changes

x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/Security.java Outdated Show resolved Hide resolved

slobodanadamovic approved these changes Oct 3, 2025

View reviewed changes

n1v0lg and others added 9 commits October 3, 2025 14:34

Early return

1303b06

Feedback

732ace0

Merge branch 'main' into cps-flat-world

0537fa5

Clean up

097a111

Javadoc

e9f2f4e

Tweaks and renames

c7748c2

[CI] Auto commit changes from spotless

69c4a80

Merge branch 'main' into cps-flat-world

802671e

Merge branch 'cps-flat-world' of github.com:n1v0lg/elasticsearch into…

1044829

… cps-flat-world

n1v0lg commented Oct 5, 2025

View reviewed changes

n1v0lg marked this pull request as ready for review October 5, 2025 19:38

elasticsearchmachine added the Team:Security Meta label for security team label Oct 5, 2025

ywangd approved these changes Oct 6, 2025

View reviewed changes

n1v0lg added 3 commits October 6, 2025 10:52

Merge branch 'main' into cps-flat-world

01155f2

Feedback

2bffce3

Merge branch 'main' into cps-flat-world

71866b1

n1v0lg added the auto-merge-without-approval Automatically merge pull request when CI checks pass (NB doesn't wait for reviews!) label Oct 6, 2025

Fix comment

1260c15

elasticsearchmachine merged commit b9f878e into elastic:main Oct 6, 2025
40 checks passed

n1v0lg deleted the cps-flat-world branch October 6, 2025 11:55

		// we still need to call response validation for local results, since qualified expressions like `_origin:index` or
		// `<alias-pattern-matching-origin-only>:index` get deferred validation, also

Cross-project search index expression rewriting #135346

Cross-project search index expression rewriting #135346

Conversation

n1v0lg commented Sep 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

slobodanadamovic left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

n1v0lg Oct 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elasticsearchmachine commented Oct 5, 2025

Uh oh!

ywangd left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

n1v0lg Oct 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ywangd left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

n1v0lg commented Sep 24, 2025 •

edited

Loading

slobodanadamovic left a comment •

edited

Loading

n1v0lg Oct 5, 2025 •

edited

Loading

n1v0lg Oct 6, 2025 •

edited

Loading