allocation: add balancer round summary as metrics #136043

schase-es · 2025-10-06T17:26:30Z

This commit adds the BalancerRoundSummary as a collection of APM/open telemetry metrics. These are already logged. The summary collected every ten seconds or so is set as the current state into the telemetry metrics class (AllocationBalancingRoundMetrics). Whenever the telemetry runs, each metric picks up its current view.

Fixes: ES-10343

This commit adds the BalancerRoundSummary as a collection of APM/open telemetry metrics. These are already logged. The summary collected every ten seconds or so is set as the current state into the telemetry metrics class (AllocationBalancingRoundMetrics). Whenever the telemetry runs, each metric picks up its current view.

elasticsearchmachine · 2025-10-08T20:15:52Z

Hi @schase-es, I've created a changelog YAML for you.

nicktindall

Just some comments, I think we should discuss with Dianna what we're aiming for with the node deltas

nicktindall · 2025-10-14T22:44:33Z

server/src/main/java/org/elasticsearch/cluster/ClusterModule.java

        bind(AllocationStatsService.class).toInstance(allocationStatsService);
        bind(TelemetryProvider.class).toInstance(telemetryProvider);
        bind(DesiredBalanceMetrics.class).toInstance(desiredBalanceMetrics);
+        bind(AllocationBalancingRoundMetrics.class).toInstance(balancingRoundMetrics);


Binding like this is only necessary if we use the instance in a cluster annotated @Inject, I'm not sure if we do with the AllocationBalancingRoundMetrics? It probably doesn't matter, but I think in general we don't do it unless we need to.

nicktindall · 2025-10-14T22:46:22Z

.../org/elasticsearch/cluster/routing/allocation/allocator/AllocationBalancingRoundMetrics.java

+        assert summary != null : "balancing round metrics cannot be null";
+
+        nodeNameToWeightChangesRef.set(summary.nodeNameToWeightChanges());
+        if (enableSending) {


Do we need to set the nodeNameToWeightChangesRef if enableSending = false ?

nicktindall · 2025-10-14T23:22:41Z

.../org/elasticsearch/cluster/routing/allocation/allocator/AllocationBalancingRoundMetrics.java

+            long shardCount = nodeWeightChanges.baseWeights().shardCount() + nodeWeightChanges.weightsDiff().shardCountDiff();
+            metrics.add(new LongWithAttributes(shardCount, getNodeAttributes(nodeWeights.getKey())));
+        }
+        return metrics;


I don't think we want the current values of shard count, disk usage and write load, because we already have those in the cluster balance dashboard

Perhaps the deltas are more interesting? I think number of balancing rounds, and shard movements are definitely valuable as you have them published now, but I'm less clear on the value of the specific node shard/weight/disk usage deltas/values that we don't get already from existing metrics.

Maybe the absolute amount of change (e.g. if one node loses X weight and another gains X the sum would be 2X) in those values might be interesting?

If we plotted any of these values for the cluster by simply adding them together, they'd sum to zero I think? because for every node gaining X shards there are other node(s) losing X shards. @DiannaHohensee might have more clarity on the direction here.

I think for serverless as well, disk usage will always be zero after this change?

As discussed I think we should reduce the scope on this PR to just the two metrics we know how to publish and do a second PR when we've discussed how we'd like to publish the write load/shard count/disk usage

.../org/elasticsearch/cluster/routing/allocation/allocator/AllocationBalancingRoundMetrics.java

…histogram

elasticsearchmachine · 2025-10-27T23:39:18Z

Pinging @elastic/es-distributed-coordination (Team:Distributed Coordination)

schase-es · 2025-10-28T01:17:46Z

.../org/elasticsearch/cluster/routing/allocation/allocator/AllocationBalancingRoundMetrics.java

+            shardCountHistogram.record(baseWeights.shardCount() + weightsDiff.shardCountDiff(), getNodeAttributes(nodeName));
+            diskUsageHistogram.record(baseWeights.diskUsageInBytes() + weightsDiff.diskUsageInBytesDiff(), getNodeAttributes(nodeName));
+            writeLoadHistogram.record(baseWeights.writeLoad() + weightsDiff.writeLoadDiff(), getNodeAttributes(nodeName));
+            totalWeightHistogram.record(baseWeights.nodeWeight() + weightsDiff.totalWeightDiff(), getNodeAttributes(nodeName));


These are the metrics I'm wondering about

What's the question you're thinking about? What value to put in the histogram?

DiannaHohensee

I didn't do a thorough review, I'll have to take another look/think later. But publishing some comments as is.

.../org/elasticsearch/cluster/routing/allocation/allocator/AllocationBalancingRoundMetrics.java

DiannaHohensee · 2025-10-30T20:52:23Z

.../org/elasticsearch/cluster/routing/allocation/allocator/AllocationBalancingRoundMetrics.java

+            NUMBER_OF_SHARD_MOVES_METRIC_NAME,
+            "Current number of shard moves",
+            "{shard}"
+        );


Did we discuss having a histogram for shard moves? To avoid the combine effect, merging multiple balancing rounds together 🤔 I can't recall what we decided.

Nick and I talked about it last week again, and we decided to send the absolute difference with each summary. (The comment I made that seemed out of place was to create a link for Nick to see.)

I think we should have a histogram for shard moves as well, those differences are per-node, so we'll see how many shards moved on/off each node, but I think having a histogram of total shard move count would be good too.

We don't have the shard moves per node as a stat right now, so we can't have that :)

The high level conclusion Simon and I were discussing is that generally histograms are going to be more useful for the balancer round summary metrics because our goal is to see per balancing round granularity, as best we can. We don't have a use case for wanting to see how many total shard moves take place over time, but we do want to see roughly how many shard moves happen per balancing round, and the histogram will give us that.

So let's go with a histogram and remove the counter.

I think we should have a histogram for shard moves as well, those differences are per-node, so we'll see how many shards moved on/off each node.

Right now, these metrics aren't available in the balancing round summary. We can certainly try to get them out of the Desired Balancer and get them in through here.

DiannaHohensee · 2025-10-30T20:53:55Z

.../org/elasticsearch/cluster/routing/allocation/allocator/AllocationBalancingRoundMetrics.java

+            shardCountHistogram.record(baseWeights.shardCount() + weightsDiff.shardCountDiff(), getNodeAttributes(nodeName));
+            diskUsageHistogram.record(baseWeights.diskUsageInBytes() + weightsDiff.diskUsageInBytesDiff(), getNodeAttributes(nodeName));
+            writeLoadHistogram.record(baseWeights.writeLoad() + weightsDiff.writeLoadDiff(), getNodeAttributes(nodeName));
+            totalWeightHistogram.record(baseWeights.nodeWeight() + weightsDiff.totalWeightDiff(), getNodeAttributes(nodeName));


What's the question you're thinking about? What value to put in the histogram?

.../org/elasticsearch/cluster/routing/allocation/allocator/AllocationBalancingRoundMetrics.java

nicktindall

Looking good, just added some comments

.../org/elasticsearch/cluster/routing/allocation/allocator/AllocationBalancingRoundMetrics.java

nicktindall · 2025-11-05T06:46:39Z

.../org/elasticsearch/cluster/routing/allocation/allocator/AllocationBalancingRoundMetrics.java

+            NUMBER_OF_SHARD_MOVES_METRIC_NAME,
+            "Current number of shard moves",
+            "{shard}"
+        );


I think we should have a histogram for shard moves as well, those differences are per-node, so we'll see how many shards moved on/off each node, but I think having a histogram of total shard move count would be good too.

DiannaHohensee

Sync'ed with Simon. I think this looks great, just need a histogram for the shardMoves to replace the LongCounter shardMovesCounter.

@nicktindall can I leave the final sign off to you? You're more familiar with this area of the code (APM metrics) than I am right now.

Update: to add an offline discussion, we decided that we'd like the shardMoveCount to also be reported as a histogram. So we'll have both a LongCounter shardMoveCount and LongHistogram shardMoveHistogram.

nicktindall · 2025-11-07T04:47:26Z

.../org/elasticsearch/cluster/routing/allocation/allocator/AllocationBalancingRoundMetrics.java

+    public static final String NUMBER_OF_SHARDS_METRIC_NAME = "es.allocator.balancing_round.shard_count";
+    public static final String DISK_USAGE_BYTES_METRIC_NAME = "es.allocator.balancing_round.disk_usage_bytes";
+    public static final String WRITE_LOAD_METRIC_NAME = "es.allocator.balancing_round.write_load";
+    public static final String TOTAL_WEIGHT_METRIC_NAME = "es.allocator.balancing_round.total_weight";


I thought the metrics registry made you end histogram metric names in .histogram or something?

nicktindall

This looks good to me, with a few nits and a very annoying change request to the labels we're using.

nicktindall · 2025-11-07T04:48:41Z

.../org/elasticsearch/cluster/routing/allocation/allocator/AllocationBalancingRoundMetrics.java

+        this.shardMovesCounter = meterRegistry.registerLongCounter(
+            NUMBER_OF_SHARD_MOVES_METRIC_NAME,
+            "Current number of shard moves",
+            "{shard}"


Nit: I think the convention is to use unit when it's just a count ?

Thanks for pointing this out -- this is one of the conventions that's been hard to understand.

nicktindall · 2025-11-07T04:50:06Z

.../org/elasticsearch/cluster/routing/allocation/allocator/AllocationBalancingRoundMetrics.java

+
+        this.shardMovesHistogram = meterRegistry.registerLongHistogram(
+            NUMBER_OF_SHARD_MOVES_HISTOGRAM_METRIC_NAME,
+            "Histogram of shard moves",


Nit: can we say "number of shard movements executed in a balancing round" to make it clear it's a per-round count

These are great docs suggestions -- thanks!

nicktindall · 2025-11-07T04:51:00Z

.../org/elasticsearch/cluster/routing/allocation/allocator/AllocationBalancingRoundMetrics.java

+        );
+        this.shardMovesCounter = meterRegistry.registerLongCounter(
+            NUMBER_OF_SHARD_MOVES_METRIC_NAME,
+            "Current number of shard moves",


Nit: can we say "total number of shard moves" also above, current is a term often used for gauges

nicktindall · 2025-11-07T04:52:13Z

.../org/elasticsearch/cluster/routing/allocation/allocator/AllocationBalancingRoundMetrics.java

+            "Histogram of shard moves",
+            "unit"
+        );
+        this.shardCountHistogram = meterRegistry.registerLongHistogram(NUMBER_OF_SHARDS_METRIC_NAME, "Current number of shards", "unit");


Nit: perhaps "change in node shard count per balancing round" to make it clear it's node-scoped and a delta

Also similar for the descriptions below.

.../org/elasticsearch/cluster/routing/allocation/allocator/AllocationBalancingRoundMetrics.java

…nges

…tire object

nicktindall

Approved with some minor nits. Nice one.

nicktindall · 2025-11-11T02:54:46Z

.../org/elasticsearch/cluster/routing/allocation/allocator/AllocationBalancingRoundMetrics.java

+            long shardCountDiff = weightsDiff.shardCountDiff();
+            if (shardCountDiff < 0) {
+                shardCountDiff *= -1;
+            }


I'm not sure I follow the reasoning here? can't we just use Math.abs if we know the diff won't be MIN_VALUE?

This is in the forbidden APIs unfortunately -- it fails the math.abs call, but curious does not fail this *= -1 call. It's an annoyance because the whole point of longs is that the max or min value are so, so big that you'll never see it.

I should mention the forbidden APIs in the comment.

Ah that makes sense, yes either mention it in the comment, or there's also org.elasticsearch.core.SuppressForbidden, but on a large method that might be dangerous because I think it's scoped to the whole method.

I moved it into a private method longAbsNegativeSafe -- it matches the other metrics, and it's clearer as the suppress annotation has space for a reason.

nicktindall · 2025-11-11T03:03:33Z

.../main/java/org/elasticsearch/cluster/routing/allocation/allocator/BalancingRoundSummary.java

+            nodeToWeightChanges.forEach((node, nodesWeightChanges) -> nodeNameToWeightChanges.put(node.getName(), nodesWeightChanges));
+
+            return Strings.format(
+                "CombinedBalancingRoundSummary[numberOfBalancingRounds=%d, nodeToWeightChange=%s, " + "numberOfShardMoves=%d]",


Nit: looks like an auto-format mishap (unnecessary concatenation)

nicktindall · 2025-11-11T03:05:43Z

...search/cluster/routing/allocation/allocator/AllocationBalancingRoundSummaryServiceTests.java

+        Map.of(),
+        Set.of(),
+        null
+    );


Nit: there is also org.elasticsearch.cluster.node.DiscoveryNodeUtils for creating DiscoveryNode instances which might make this less verbose?

nicktindall · 2025-11-11T03:07:03Z

.../java/org/elasticsearch/cluster/routing/allocation/allocator/BalancingRoundSummaryTests.java

-        final String NODE_1 = "node1";
-        final String NODE_2 = "node2";
+        final DiscoveryNode NODE_1 = new DiscoveryNode("node1", "node1Id", "eph-node1", "abc", "abc", null, Map.of(), Set.of(), null);
+        final DiscoveryNode NODE_2 = new DiscoveryNode("node2", "node2Id", "eph-node2", "abc", "abc", null, Map.of(), Set.of(), null);


Also here (org.elasticsearch.cluster.node.DiscoveryNodeUtils)

nicktindall · 2025-11-12T00:23:39Z

.../org/elasticsearch/cluster/routing/allocation/allocator/AllocationBalancingRoundMetrics.java

+
+    @SuppressForbidden(reason = "ForbiddenAPIs bans Math.abs on longs, but long.MIN_VALUE is impossible here")
+    private long longAbsNegativeSafe(long input) {
+        return Math.abs(input);


Can we just put an assertion above the call to abs that ensures input isn't Long.MIN_VALUE

elasticsearchmachine added needs:triage Requires assignment of a team area label v9.3.0 labels Oct 6, 2025

[CI] Auto commit changes from spotless

99844ad

schase-es added :Distributed Coordination/Allocation All issues relating to the decision making around placing a shard (both master logic & on the nodes) >enhancement and removed needs:triage Requires assignment of a team area label labels Oct 8, 2025

schase-es marked this pull request as draft October 8, 2025 20:15

schase-es and others added 4 commits October 8, 2025 13:15

Update docs/changelog/136043.yaml

dbce2bd

Added enableSending flag in, and some renames

840b003

Added metrics consolidation, correct diff calculation, and some tests.

5a94657

[CI] Auto commit changes from spotless

c7195fc

nicktindall reviewed Oct 14, 2025

View reviewed changes

.../org/elasticsearch/cluster/routing/allocation/allocator/AllocationBalancingRoundMetrics.java Show resolved Hide resolved

schase-es added 2 commits October 27, 2025 16:12

Merge branch 'main' into ES-10343_balancer-round-apm-export

fec6131

Changing metrics for NodeWeightsChanges and NodeWeightsDiff to use a …

4d1b9db

…histogram

schase-es added the v9.2.0 label Oct 27, 2025

schase-es requested a review from DiannaHohensee October 27, 2025 23:38

schase-es marked this pull request as ready for review October 27, 2025 23:38

elasticsearchmachine added the Team:Distributed Coordination Meta label for Distributed Coordination team label Oct 27, 2025

schase-es commented Oct 28, 2025

View reviewed changes

DiannaHohensee reviewed Oct 30, 2025

View reviewed changes

schase-es added 2 commits November 2, 2025 20:51

Remove extra meter registry

1b149ef

Send absolute value of diff instead of last + diff

41a2c72

nicktindall reviewed Nov 5, 2025

View reviewed changes

DiannaHohensee reviewed Nov 5, 2025

View reviewed changes

schase-es and others added 2 commits November 6, 2025 20:34

Adding shard moves histogram

26f9c14

[CI] Auto commit changes from spotless

40bf700

nicktindall reviewed Nov 7, 2025

View reviewed changes

schase-es and others added 7 commits November 10, 2025 10:27

Fixes to metrics names and summaries

dad8264

Move of DiscoveryNode as key in BalancingRoundSummary.nodeToWeightCha…

28dbf90

…nges

Renaming nodeNameToWeightChanges to nodeToWeightChanges

d89a03c

[CI] Auto commit changes from spotless

ba786de

CombinedBalancingRoundSummary prints DiscoveryNode name instead of en…

85e4919

…tire object

Math.abs on a long is forbidden. Use *= -1 instead

52c1ce7

Style fixes

cb6a196

nicktindall approved these changes Nov 11, 2025

View reviewed changes

schase-es added 3 commits November 11, 2025 15:25

Move of Math.abs on long into suppressed method

d0bb2d3

Use DiscoveryNodeUtils to make a node

a650365

Formatting fixes

ac9ee89

nicktindall reviewed Nov 12, 2025

View reviewed changes

schase-es added 2 commits November 11, 2025 17:06

Adding assertion for Long.MIN_VALUE, better comments and names

adb1741

Merge branch 'main' into ES-10343_balancer-round-apm-export

6eefe31

schase-es added auto-backport Automatically create backport pull requests when merged and removed auto-backport Automatically create backport pull requests when merged v9.2.0 labels Nov 13, 2025

schase-es merged commit 2e9b8f1 into elastic:main Nov 13, 2025
34 checks passed

allocation: add balancer round summary as metrics #136043

allocation: add balancer round summary as metrics #136043

Uh oh!

Conversation

schase-es commented Oct 6, 2025

Uh oh!

elasticsearchmachine commented Oct 8, 2025

Uh oh!

nicktindall left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

elasticsearchmachine commented Oct 27, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

DiannaHohensee left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

nicktindall left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

DiannaHohensee left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nicktindall left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

nicktindall left a comment

Choose a reason for hiding this comment

Uh oh!

DiannaHohensee left a comment •

edited

Loading

schase-es Nov 11, 2025 •

edited

Loading