Data imbalance

[599166320]

Affected Version

24.0.0

Description

When druid is upgraded from 0.22.0 to 24.0.0, a strange phenomenon occurs: it always allocates segments to a specific machine until the capacity of the machine is used up.

I have added this machine to the decommissioningNodes nodes for many times. After the data volume of this machine decreases, I will delete it from the decommissioningNodes nodes list. After deletion, the capacity of this machine will be full again. Such repeated operations cannot balance the data.

[AmatyaAvadhanula]

Hi, were any runtime properties changed with the upgrade?
Could you also please share the coordinator's dynamic config and cost strategy as well?

[599166320]

@AmatyaAvadhanula

1. dynamic config

{
  "millisToWaitBeforeDeleting": 900000,
  "mergeBytesLimit": 524288000,
  "mergeSegmentsLimit": 1000,
  "maxSegmentsToMove": 5,
  "percentOfSegmentsToConsiderPerMove": 100,
  "useBatchedSegmentSampler": true,
  "replicantLifetime": 15,
  "replicationThrottleLimit": 10,
  "balancerComputeThreads": 5,
  "emitBalancingStats": false,
  "killDataSourceWhitelist": [],
  "killPendingSegmentsSkipList": [],
  "maxSegmentsInNodeLoadingQueue": 500,
  "decommissioningNodes": [
   
  ],
  "decommissioningMaxPercentOfMaxSegmentsToMove": 60,
  "pauseCoordination": false,
  "replicateAfterLoadTimeout": false,
  "maxNonPrimaryReplicantsToLoad": 2147483647
}

2. master runtime properties

druid.service=druid/coordinator
druid.plaintextPort=8061

druid.coordinator.startDelay=PT10S
druid.coordinator.period=PT5S
druid.coordinator.balancer.strategy=CachingCostBalancerStrategy

druid.coordinator.asOverlord.enabled=true
druid.coordinator.asOverlord.overlordService=druid/overlord

druid.indexer.queue.startDelay=PT5S

druid.indexer.runner.type=remote
druid.indexer.storage.type=metadata

druid.indexer.logs.kill.enabled=true
druid.indexer.logs.kill.durationToRetain=1
druid.coordinator.kill.on=true
druid.coordinator.kill.durationToRetain=PT1s
druid.coordinator.kill.maxSegments=1000000

3. historical runtime properties

druid.service=druid/historical
druid.plaintextPort=8051

# HTTP server threads
druid.server.http.numThreads=60

# Processing threads and buffers
druid.processing.buffer.sizeBytes=500000000
druid.processing.numMergeBuffers=16
druid.processing.numThreads=62
druid.processing.tmpDir=var/druid/processing

# Segment storage
druid.segmentCache.locations=[{"path":"var/druid/segment-cache","maxSize":"1000g"}]

# Query cache
druid.historical.cache.useCache=true
druid.historical.cache.populateCache=true
druid.cache.type=caffeine
druid.cache.sizeInBytes=2GiB

[AmatyaAvadhanula]

@599166320

1. Could you please check the metric segment/pendingDelete for the problematic historical to confirm if it is an issue with segment assignment or if it is an issue with the historical being unable to drop the data?

2. I believe CachingCostBalancerStrategy needs to be set by using druid.coordinator.balancer.strategy=cachingCost.
   (https://druid.apache.org/docs/latest/configuration/index.html#coordinator-operation)

[599166320]

@AmatyaAvadhanula I think the segment/pendingDelete is normal. The problem may be caused by druid.coordinator.balancer.strategy.

[599166320]

I found that cachingCost and cost may not guarantee the cluster data balance. Maybe I should try to use diskNormalized and random to solve the data balance problem