CORE-1743 rptest: fix flaky metric check by disabling leader balancer

[nvartolomei]

Fixes #16342

Backports Required

• none - not a bug fix
• none - this is a backport
• none - issue does not exist in previous branches
• none - papercut/not impactful enough to backport
• v23.3.x
• v23.2.x

Release Notes

• none

[vbotbuildovich]

new failures in https://buildkite.com/redpanda/redpanda/builds/47735#018ed344-70ed-493a-8653-dec48f325166:

"rptest.tests.partition_move_interruption_test.PartitionMoveInterruption.test_cancelling_partition_move.replication_factor=3.unclean_abort=True.recovery=restart_recovery.compacted=False"

new failures in https://buildkite.com/redpanda/redpanda/builds/47735#018ed34b-3e86-4bdd-8ae4-ddf947ee733f:

"rptest.tests.cloud_storage_timing_stress_test.CloudStorageTimingStressTest.test_cloud_storage_with_partition_moves.cleanup_policy=delete"

[vbotbuildovich]

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/47735#018ed344-70ed-493a-8653-dec48f325166

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/47793#018ee1e8-321a-4ef8-8722-a55bfdd8215a

[dotnwat]

would making the metric check be more flexible be an alternative?

[nvartolomei]

There will be always a chance of a race condition between querying who the leader is and the metric value.

So you have ideas? For this particular test it would probably be easier to just start one replica and not bother about leadership at all

[dotnwat]

i approved the pr, i'm just asking if it's possible/reasonable.

So you have ideas?

if in the face of leadership transfer there is a metrics query, potentially involving aggregation across endpoints, that even with the leadership transfer correctly asserts the property being tested for, then that would be the idea. not that it's easy or possible. just wondering, because less we assume about the system (e.g. balancer enabled/disabled) the more robust the tests are.

[nvartolomei]

/dt