cluster: publish total available reclaim size to balancer

[dotnwat]

SM publishes reclaimable space up to the local retention level to the cluster balancer. However, it may be the case that a disk is nearly full even at the local retention target, causing the balancer to believe that any space is not available for reclaim. This is problematic for decommission which needs to be able to find some place to create new replicas.

This change swaps out the reclaimable space at local retention level for total reclaimable space. The balancer policy will be expanded further to also take into account the local retention targets and optimize decisions. Changes for that are here #16372.

Related https://github.com/redpanda-data/core-internal/issues/719

Backports Required

• none - not a bug fix
• none - this is a backport
• none - issue does not exist in previous branches
• none - papercut/not impactful enough to backport
• v23.3.x
• v23.2.x
• v23.1.x

Release Notes

Improvements

• Publish total reclaimable space to avoid stuck decommission scenario.

[vbotbuildovich]

new failures in https://buildkite.com/redpanda/redpanda/builds/44451#018d57bd-426f-4946-94f7-cdf8271819d1:

"rptest.tests.audit_log_test.AuditLogTestOauth.test_admin_oauth"

[vbotbuildovich]

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/44451#018d57bd-426c-4bb5-bf55-002e20734efa

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/44491#018d5c18-12a2-48fd-8858-2bf043653c7f

[andrwng]

With this change alone, is the balancer actually balancing any differently? Or does this just change the conditions under which the balancer will balance? I think the latter, but just want to make sure I understand

[ztlpn]

@andrwng the answer is both!

• changes to total size report will affect the triggering conditions ("is this node under disk pressure so that some replicas have to be moved off of it?") as well as individual replica placement decisions ("can I place a replica on this node without causing disk pressure?")
• changes to per-partition size report will affect individual placement decisions too ("if I move this partition replica, what is the amount of disk space it will occupy on the destination node if it is under disk pressure?")

[andrwng]

@andrwng the answer is both!

• changes to total size report will affect the triggering conditions ("is this node under disk pressure so that some replicas have to be moved off of it?") as well as individual replica placement decisions ("can I place a replica on this node without causing disk pressure?")
• changes to per-partition size report will affect individual placement decisions too ("if I move this partition replica, what is the amount of disk space it will occupy on the destination node if it is under disk pressure?")

Got it, thanks for the explanation! That makes sense; I recalled that we don't try to balance space across nodes, but it makes sense that we gate individual moves based on whether there is available/reclaimable space.

[vbotbuildovich]

/backport v23.3.x

[vbotbuildovich]

/backport v23.2.x

[vbotbuildovich]

Failed to create a backport PR to v23.2.x branch. I tried:

git remote add upstream https://github.com/redpanda-data/redpanda.git
git fetch --all
git checkout -b backport-pr-16354-v23.2.x-948 remotes/upstream/v23.2.x
git cherry-pick -x 930db9f0c9e3828656895a80c4d8c0fecaeed029 20e10636780195e0ca1dc8b6d79a3ad998466c05

Workflow run logs.