Replace transport version utils with a build service #133034

rjernst · 2025-08-16T16:56:38Z

Several transport version build tasks have a need to access the repository-wide transport version resources. Thus far it has been done through several utility methods. This commit moves these utility methods into an encapsulated build service that is shared across the transport version tasks. The advantage is that no paths are used, the build service encapsulates access to the resources and understands internally how to find the correct filesystem path and load it.

closes #132788
closes #132789
closes #132790

Several transport version build tasks have a need to access the repository-wide transport version resources. Thus far it has been done through several utility methods. This commit moves these utility methods into an encapsulated build service that is shared across the transport version tasks. The advantage is that no paths are used, the build service encapsulates access to the resources and understands internally how to find the correct filesystem path and load it.

elasticsearchmachine · 2025-08-16T16:57:04Z

Pinging @elastic/es-core-infra (Team:Core/Infra)

JVerwolf

Much cleaner.

JVerwolf · 2025-08-18T16:22:17Z

.../main/java/org/elasticsearch/gradle/internal/transport/TransportVersionResourcesService.java

+    }
+
+    /**
+     * Return the transport version resources directory for this repository.


"resources directory" means something specific in this context, but also has a more general meaning that might confuse readers. Should we define this term?

I reworded a bit to make clearer these are transport version resources.

.../main/java/org/elasticsearch/gradle/internal/transport/TransportVersionResourcesService.java

...n/java/org/elasticsearch/gradle/internal/transport/GenerateTransportVersionManifestTask.java

…p0=search.vectors/45_knn_search_bit/Vector similarity with filter only} elastic#133037

…p0=search.vectors/45_knn_search_bit/Vector rescoring has no effect for non-quantized vectors and provides same results as non-rescored knn} elastic#133039

…33038) With elastic#132774, the overhead of running queries with DOC partitioning is small. While we might switch the default data partitioning to DOC for all queries in the future, this PR defaults data partitioning to DOC for time-series queries only to minimize any unexpected impact. Relates elastic#132774

…tic#133047) Comparing bitsPerOrd == -1 should be a little bit more optimal compared to maxOrd >= 0 (int vs long comparison).

Async queries in EQL and ES|QL do not create an initial response, and the current logic does not correctly handle expiration updates when the query has already completed. With initial response (no change): First, update the expiration in the async index, then update the task's expiration if the task still exists. Without initial response: First, try to update the task's expiration, then attempt to get the result from the task or async index. If the result is no longer available from the task, update the expiration in the async index before retrieving it (similar to the initial response case). This second step was introduced in this fix. Ideally, we should always create the initial response up front to unify the logic for both async_search and async_query, but this fix is preferred for now as it is more contained. When reviewing the code, I also found a race condition where async-get can return a NOT_FOUND error if the task completes but has not yet stored its result in the async index. This issue would also be resolved by storing an initial response up front. I will open a follow-up issue for it. Closes elastic#130619

Replace int hashmap by a counter in TrackingPostingsInMemoryBytesCodec and use int hashset to keep track of seen fields.

…p0=search.highlight/50_synthetic_source/text multi fvh source order} elastic#133056

) This is no longer needed as offsets of leaf array fields are stored in a doc values offset field that is controlled by field mapper.

- add already-tracked DLS cache stats into `/xpack/usage?filter_path=security.roles.dls`

…ndexing {upgradedNodes=1} elastic#133060

…ndexing {upgradedNodes=0} elastic#133061

Pass down actual index version when creating text field mapper builder when mapping dynamic field. Otherwise, bwc logic always has the wrong index version in mixed cluster scenario and dynamic mapping occurs.

github-actions · 2025-08-18T18:00:02Z

🔍 Preview links for changed docs

mark-vieira · 2025-08-18T18:06:26Z

@rjernst I think perhaps a merge or rebase went wrong here as there are a ton of unrelated changes in this PR.

rjernst · 2025-08-18T20:04:31Z

I think perhaps a merge or rebase went wrong here

Should be good now

mark-vieira · 2025-08-18T21:16:31Z

.../main/java/org/elasticsearch/gradle/internal/transport/TransportVersionResourcesService.java

+    private Set<String> getMainResources() {
+        if (mainResources.get() == null) {
+            synchronized (mainResources) {
+                String output = gitCommand("ls-tree", "--name-only", "-r", "main", ".");


If we're going to use relative paths here, then we should explicitly set the working directory when calling exec. There's no real guarantee that this is the workspace directory.

We set the working directory in gitCommand using -C

mark-vieira · 2025-08-18T21:22:46Z

...c/main/java/org/elasticsearch/gradle/internal/transport/TransportVersionResourcesPlugin.java

+        project.getGradle()
+            .getSharedServices()
+            .registerIfAbsent("transportVersionResources", TransportVersionResourcesService.class, spec -> {
+                Directory transportResources = project.getLayout().getProjectDirectory().dir("src/main/resources/transport");


Presumably this plugin will only ever be applied to a single project, right? Build services are global to the build so all consumers will use the same params. It's not clear if we also use this in serverless and how that will work.

Yes, and this being global is part of how it works. Note that this is also used by ValidateTransportVersionReferencesTask which are in every module/plugin. Essentially it's a way to plumb in a wrapper around the resources that are ready from source on disk instead of having shared methods (and the gradle property) to know where to find these resources.

elasticsearchmachine · 2025-08-19T22:08:10Z

💔 Backport failed

Status	Branch	Result
❌	8.18	Commit could not be cherrypicked due to conflicts
❌	9.0	Commit could not be cherrypicked due to conflicts
❌	9.1	Commit could not be cherrypicked due to conflicts
❌	8.19	Commit could not be cherrypicked due to conflicts

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 133034

Several transport version build tasks have a need to access the repository-wide transport version resources. Thus far it has been done through several utility methods. This commit moves these utility methods into an encapsulated build service that is shared across the transport version tasks. The advantage is that no paths are used, the build service encapsulates access to the resources and understands internally how to find the correct filesystem path and load it.

This commit backports several changes to the new transport version system relates elastic#133034 relates elastic#133185 relates elastic#133186 relates elastic#133189

* Replace transport version utils with a build service (#133034) Several transport version build tasks have a need to access the repository-wide transport version resources. Thus far it has been done through several utility methods. This commit moves these utility methods into an encapsulated build service that is shared across the transport version tasks. The advantage is that no paths are used, the build service encapsulates access to the resources and understands internally how to find the correct filesystem path and load it. * Make transport version ids have descending ordering (#133185) Transport version ids must be in descending order in definition files. However, the compareTo for them was ascending and the validation used reverse ordering. This commit fixes the compareTo so it uses desending ordering and the validation uses natural ordering. * Fix transport version validation task name (#133186) The task class validates resources, but the task name still had the old "definitions" suffix. * Add separate validation for unreferenced transport version definitions (#133189) This commit splits out the validation that is pertinent to unreferenced definitions. It also adds validation that names of named and unreferenced definitions cannot collide. * Use path separator for git path in transport resources (#133228) When looking up files from main in transport resources, the system dependent path is used. However, the beginning slash also needs to be system dependent.

* Backport several transport version fixes This commit backports several changes to the new transport version system relates #133034 relates #133185 relates #133186 relates #133189 * [CI] Auto commit changes from spotless --------- Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>

This commit backports several changes to the new transport version system relates #133034 relates #133185 relates #133186 relates #133189 Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

rjernst requested a review from a team as a code owner August 16, 2025 16:56

rjernst added :Core/Infra/Core Core issues without another label >refactoring auto-backport Automatically create backport pull requests when merged v8.18.6 v9.0.6 v9.1.3 v8.19.3 labels Aug 16, 2025

elasticsearchmachine added v9.2.0 Team:Core/Infra Meta label for core/infra team labels Aug 16, 2025

rjernst requested a review from breskeby August 16, 2025 16:57

[CI] Auto commit changes from spotless

303ba81

JVerwolf approved these changes Aug 18, 2025

View reviewed changes

elasticsearchmachine and others added 15 commits August 18, 2025 10:57

Mute org.elasticsearch.test.rest.yaml.CcsCommonYamlTestSuiteIT test {…

4523585

…p0=search.vectors/45_knn_search_bit/Vector similarity with filter only} elastic#133037

Mute org.elasticsearch.test.rest.yaml.CcsCommonYamlTestSuiteIT test {…

67effbd

…p0=search.vectors/45_knn_search_bit/Vector rescoring has no effect for non-quantized vectors and provides same results as non-rescored knn} elastic#133039

Streamline and optimize check to use decode or decode ordinals. (elas…

3f3a7c1

…tic#133047) Comparing bitsPerOrd == -1 should be a little bit more optimal compared to maxOrd >= 0 (int vs long comparison).

Slightly improve TrackingPostingsInMemoryBytesCodec (elastic#132905)

a10d8e3

Replace int hashmap by a counter in TrackingPostingsInMemoryBytesCodec and use int hashset to keep track of seen fields.

Mute org.elasticsearch.test.rest.yaml.CcsCommonYamlTestSuiteIT test {…

88c3bcd

…p0=search.highlight/50_synthetic_source/text multi fvh source order} elastic#133056

Remove DocumentParserContext#removeLastIgnoredField(...) (elastic#132644

052a93d

) This is no longer needed as offsets of leaf array fields are stored in a doc values offset field that is controlled by field mapper.

Shorten metadata.name of PR benchmark pipeline (elastic#133054)

f371deb

Expose existing DLS cache x-pack usage statistics (elastic#132845)

896e383

- add already-tracked DLS cache stats into `/xpack/usage?filter_path=security.roles.dls`

Mute org.elasticsearch.upgrades.SyntheticSourceRollingUpgradeIT testI…

5a2a931

…ndexing {upgradedNodes=1} elastic#133060

Mute org.elasticsearch.upgrades.SyntheticSourceRollingUpgradeIT testI…

905372d

…ndexing {upgradedNodes=0} elastic#133061

Document index setting around source metadata field. (elastic#127000)

b2bdea8

Fix typos in plugin-management.md (elastic#133062)

a654a33

Text field dynamic field tweak (elastic#129332)

e96297a

Pass down actual index version when creating text field mapper builder when mapping dynamic field. Otherwise, bwc logic always has the wrong index version in mixed cluster scenario and dynamic mapping occurs.

rjernst requested a review from a team as a code owner August 18, 2025 17:58

reword

e6cec60

rjernst added 2 commits August 18, 2025 13:00

Merge branch 'main' into transport/build_service

2743bbc

Merge branch 'main' into transport/build_service

bfd07d4

[CI] Auto commit changes from spotless

2874ff0

mark-vieira reviewed Aug 18, 2025

View reviewed changes

tvernum removed the request for review from a team August 19, 2025 07:31

rjernst added 4 commits August 19, 2025 09:11

Merge branch 'main' into transport/build_service

1d151ea

fix

e5f5101

skip non-existing

c67a164

use correct path

ac9ce1b

rjernst merged commit 7b1edda into elastic:main Aug 19, 2025
35 checks passed

rjernst deleted the transport/build_service branch August 19, 2025 22:07

elasticsearchmachine added the backport pending label Aug 19, 2025

This was referenced Aug 20, 2025

Backport several transport version fixes #133274

Merged

Backport several transport version fixes #133275

Merged

rjernst mentioned this pull request Aug 20, 2025

Backport several transport version fixes #133276

Merged

rjernst mentioned this pull request Aug 20, 2025

Backport several transport version fixes #133277

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Replace transport version utils with a build service #133034

Replace transport version utils with a build service #133034

Uh oh!

rjernst commented Aug 16, 2025 •

edited

Loading

Uh oh!

elasticsearchmachine commented Aug 16, 2025

Uh oh!

JVerwolf left a comment

Uh oh!

JVerwolf Aug 18, 2025

Uh oh!

rjernst Aug 18, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Aug 18, 2025 •

edited

Loading

Uh oh!

mark-vieira commented Aug 18, 2025

Uh oh!

rjernst commented Aug 18, 2025

Uh oh!

mark-vieira Aug 18, 2025

Uh oh!

rjernst Aug 18, 2025

Uh oh!

mark-vieira Aug 18, 2025

Uh oh!

rjernst Aug 19, 2025 •

edited

Loading

Uh oh!

Uh oh!

elasticsearchmachine commented Aug 19, 2025

Uh oh!

Uh oh!

Replace transport version utils with a build service #133034

Replace transport version utils with a build service #133034

Uh oh!

Conversation

rjernst commented Aug 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticsearchmachine commented Aug 16, 2025

Uh oh!

JVerwolf left a comment

Choose a reason for hiding this comment

Uh oh!

JVerwolf Aug 18, 2025

Choose a reason for hiding this comment

Uh oh!

rjernst Aug 18, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Aug 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔍 Preview links for changed docs

Uh oh!

mark-vieira commented Aug 18, 2025

Uh oh!

rjernst commented Aug 18, 2025

Uh oh!

mark-vieira Aug 18, 2025

Choose a reason for hiding this comment

Uh oh!

rjernst Aug 18, 2025

Choose a reason for hiding this comment

Uh oh!

mark-vieira Aug 18, 2025

Choose a reason for hiding this comment

Uh oh!

rjernst Aug 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

elasticsearchmachine commented Aug 19, 2025

💔 Backport failed

Uh oh!

Uh oh!

rjernst commented Aug 16, 2025 •

edited

Loading

github-actions bot commented Aug 18, 2025 •

edited

Loading

rjernst Aug 19, 2025 •

edited

Loading