Skip to content

Conversation

jozala
Copy link
Contributor

@jozala jozala commented Aug 20, 2025

Setting the number of processors available for node to adequately size thread pools.
Related to #130612

Setting the number of processors available for node to adequately size
thread pools.
Related to elastic#130612
@jozala jozala requested a review from a team as a code owner August 20, 2025 09:15
@jozala jozala added >non-issue >test Issues or PRs that are addressing/adding tests :Delivery/Build Build or test infrastructure auto-backport Automatically create backport pull requests when merged v9.2.0 v8.17.11 v8.18.6 v9.0.6 v9.1.3 v8.19.3 labels Aug 20, 2025
@elasticsearchmachine elasticsearchmachine added the Team:Delivery Meta label for Delivery team label Aug 20, 2025
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-delivery (Team:Delivery)

Copy link
Contributor

@breskeby breskeby left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can make this a bit more readable

|| buildParams.getBwcVersions().unreleasedInfo(version) == null
);

NamedDomainObjectContainer<ElasticsearchCluster> testClusters = (NamedDomainObjectContainer<ElasticsearchCluster>) project
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can simplify this a bit:


        String processorCount = shouldConfigureTestClustersWithOneProcessor() ? "1" : "2";
        NamedDomainObjectContainer<ElasticsearchCluster> testClusters = (NamedDomainObjectContainer<ElasticsearchCluster>) project
            .getExtensions()
            .getByName(TestClustersPlugin.EXTENSION_NAME);
        testClusters.configureEach(elasticsearchCluster -> elasticsearchCluster.setting("node.processors", processorCount));

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@jozala jozala requested a review from breskeby August 20, 2025 10:24
@jozala jozala enabled auto-merge (squash) August 20, 2025 10:45
@jozala jozala merged commit 51e7926 into elastic:main Aug 20, 2025
33 of 34 checks passed
jozala added a commit to jozala/elasticsearch that referenced this pull request Aug 20, 2025
Setting the number of processors available for node to adequately size
thread pools.
Related to elastic#130612
jozala added a commit to jozala/elasticsearch that referenced this pull request Aug 20, 2025
Setting the number of processors available for node to adequately size
thread pools.
Related to elastic#130612
@elasticsearchmachine
Copy link
Collaborator

💚 Backport successful

Status Branch Result
8.17
8.18
9.0
9.1
8.19

jozala added a commit to jozala/elasticsearch that referenced this pull request Aug 20, 2025
Setting the number of processors available for node to adequately size
thread pools.
Related to elastic#130612
jozala added a commit to jozala/elasticsearch that referenced this pull request Aug 20, 2025
Setting the number of processors available for node to adequately size
thread pools.
Related to elastic#130612
jozala added a commit to jozala/elasticsearch that referenced this pull request Aug 20, 2025
Setting the number of processors available for node to adequately size
thread pools.
Related to elastic#130612
szybia added a commit to szybia/elasticsearch that referenced this pull request Aug 20, 2025
…improv

* upstream/main: (58 commits)
  Fixing flaky LoggedExec (tests) (elastic#133215)
  CPS search should not use `skip_unavailable` (elastic#132927)
  Don't fail search if bottom doc can't be formatted (elastic#133188)
  Mute org.elasticsearch.xpack.esql.action.RandomizedTimeSeriesIT testGroupByNothing elastic#133225
  Mute org.elasticsearch.index.codec.tsdb.es819.ES819TSDBDocValuesFormatTests testSortedSetDocValuesWithSkipperSmall elastic#133224
  Mute org.elasticsearch.index.codec.tsdb.es819.ES819TSDBDocValuesFormatTests testSortedNumberMergeAwayAllValuesWithSkipper elastic#133223
  Adding support for index.number_of_replicas to data stream settings (elastic#132748)
  Mute org.elasticsearch.index.codec.tsdb.es819.ES819TSDBDocValuesFormatTests testSortedDocValuesSingleUniqueValue elastic#133221
  Fix VectorSimilarityFunctionsIT (elastic#133206)
  Mute org.elasticsearch.xpack.esql.action.RandomizedTimeSeriesIT testGroupBySubset elastic#133220
  Increase the number of FORK branches in ForkGenerator (elastic#132019)
  Mute org.elasticsearch.index.mapper.blockloader.IpFieldBlockLoaderTests testBlockLoader {preference=Params[syntheticSource=true, preference=STORED]} elastic#133218
  Mute org.elasticsearch.index.mapper.blockloader.IpFieldBlockLoaderTests testBlockLoader {preference=Params[syntheticSource=true, preference=DOC_VALUES]} elastic#133217
  Mute org.elasticsearch.index.mapper.blockloader.IpFieldBlockLoaderTests testBlockLoader {preference=Params[syntheticSource=true, preference=NONE]} elastic#133216
  Set default processor allocation for test clusters (elastic#133204)
  Add mapper for exponential histograms (elastic#132493)
  Fix offset handling in Murmur3Hasher (elastic#133193)
  unmute testDoesNotResolveClosedIndex (elastic#133115)
  Fix an AWS SDK v2 release note (elastic#133155)
  Limit the depth of a filter (elastic#133113)
  ...
szybia added a commit to szybia/elasticsearch that referenced this pull request Aug 20, 2025
* upstream/main: (58 commits)
  Fixing flaky LoggedExec (tests) (elastic#133215)
  CPS search should not use `skip_unavailable` (elastic#132927)
  Don't fail search if bottom doc can't be formatted (elastic#133188)
  Mute org.elasticsearch.xpack.esql.action.RandomizedTimeSeriesIT testGroupByNothing elastic#133225
  Mute org.elasticsearch.index.codec.tsdb.es819.ES819TSDBDocValuesFormatTests testSortedSetDocValuesWithSkipperSmall elastic#133224
  Mute org.elasticsearch.index.codec.tsdb.es819.ES819TSDBDocValuesFormatTests testSortedNumberMergeAwayAllValuesWithSkipper elastic#133223
  Adding support for index.number_of_replicas to data stream settings (elastic#132748)
  Mute org.elasticsearch.index.codec.tsdb.es819.ES819TSDBDocValuesFormatTests testSortedDocValuesSingleUniqueValue elastic#133221
  Fix VectorSimilarityFunctionsIT (elastic#133206)
  Mute org.elasticsearch.xpack.esql.action.RandomizedTimeSeriesIT testGroupBySubset elastic#133220
  Increase the number of FORK branches in ForkGenerator (elastic#132019)
  Mute org.elasticsearch.index.mapper.blockloader.IpFieldBlockLoaderTests testBlockLoader {preference=Params[syntheticSource=true, preference=STORED]} elastic#133218
  Mute org.elasticsearch.index.mapper.blockloader.IpFieldBlockLoaderTests testBlockLoader {preference=Params[syntheticSource=true, preference=DOC_VALUES]} elastic#133217
  Mute org.elasticsearch.index.mapper.blockloader.IpFieldBlockLoaderTests testBlockLoader {preference=Params[syntheticSource=true, preference=NONE]} elastic#133216
  Set default processor allocation for test clusters (elastic#133204)
  Add mapper for exponential histograms (elastic#132493)
  Fix offset handling in Murmur3Hasher (elastic#133193)
  unmute testDoesNotResolveClosedIndex (elastic#133115)
  Fix an AWS SDK v2 release note (elastic#133155)
  Limit the depth of a filter (elastic#133113)
  ...
elasticsearchmachine pushed a commit that referenced this pull request Aug 22, 2025
…133213)

* Set default processor allocation for test clusters (#133204)

Setting the number of processors available for node to adequately size
thread pools.
Related to #130612

* Lowering the allocations during update to fit within available processors

---------

Co-authored-by: Jonathan Buttner <jonathan.buttner@elastic.co>
elasticsearchmachine pushed a commit that referenced this pull request Aug 22, 2025
…133210)

* Set default processor allocation for test clusters (#133204)

Setting the number of processors available for node to adequately size
thread pools.
Related to #130612

* Lowering the allocations during update to fit within available processors

---------

Co-authored-by: Jonathan Buttner <jonathan.buttner@elastic.co>
elasticsearchmachine pushed a commit that referenced this pull request Aug 22, 2025
…133209)

* Set default processor allocation for test clusters (#133204)

Setting the number of processors available for node to adequately size
thread pools.
Related to #130612

* Lowering the allocations during update to fit within available processors

---------

Co-authored-by: Jonathan Buttner <jonathan.buttner@elastic.co>
jozala added a commit to jozala/elasticsearch that referenced this pull request Aug 25, 2025
Setting the number of processors available for node to adequately size
thread pools.
Related to elastic#130612
jozala added a commit to jozala/elasticsearch that referenced this pull request Aug 25, 2025
Setting the number of processors available for node to adequately size
thread pools.
Related to elastic#130612
elasticsearchmachine pushed a commit that referenced this pull request Aug 25, 2025
…133211)

* Set default processor allocation for test clusters (#133204)

Setting the number of processors available for node to adequately size
thread pools.
Related to #130612

* Lowering the allocations during update to fit within available processors

---------

Co-authored-by: Jonathan Buttner <jonathan.buttner@elastic.co>
elasticsearchmachine pushed a commit that referenced this pull request Aug 25, 2025
…133212)

* Set default processor allocation for test clusters (#133204)

Setting the number of processors available for node to adequately size
thread pools.
Related to #130612

* Lowering the allocations during update to fit within available processors

---------

Co-authored-by: Jonathan Buttner <jonathan.buttner@elastic.co>
@felixbarny
Copy link
Member

For folks like me who are sometimes doing local benchmarking via ./gradlew run and wonder why the number of threads is suddenly a lot lower, you can override this with -Dtests.es.node.processors=8.

@felixbarny
Copy link
Member

I'm also wondering if we could take an increased heap size (like -Dtests.heap.size=4G) as a signal to not limit the number of processors.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
auto-backport Automatically create backport pull requests when merged :Delivery/Build Build or test infrastructure >non-issue Team:Delivery Meta label for Delivery team >test Issues or PRs that are addressing/adding tests v8.17.11 v8.18.6 v8.19.3 v9.0.6 v9.1.3 v9.2.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants