node wise recovery improvements

[bharathv]

Some changes from previous defunct nodes implementation.

• defunct nodes is no longer a thing, will be reimplemented as offline nodes in a future PR.
• node wise recovery and offlining nodes are two separate user workflows. Earlier node wise recovering partitions and marking nodes as defunct were combined into a single command.
• node wise recovery no longer touches partitions with majority, balancer should automatically recover them after unavailability timeout.
• corresponding controller command for node wise recovery is idempotent (can be issued as many times as needed)

API examples:

fetch a list of partitions losing majority - GET http://:9644/v1/partitions/majority_lost?dead_nodes=5,6,7
node wise recovery them - POST http://:9644/v1/partitions/force_recover_from_nodes

example payload.

{'dead_nodes': [5], 'partitions_to_force_recover': [{'ntp': {'ns': 'kafka', 'topic': 'topic-3', 'partition': 0     }, 'topic_revision': 42, 'replicas': [{'node_id': 5, 'core': 0}], 'dead_nodes': [5]}, {'ntp': {'ns': 'kafka', 'topic': 'topic-7', 'partition': 0}, 'topic_revision': 49, 'replicas': [{'node_id': 5, 'core': 1}], 'dead_nodes': [5]}, {'     ntp': {'ns': 'kafka', 'topic': 'topic-5', 'partition': 0}, 'topic_revision': 47, 'replicas': [{'node_id': 5, 'core': 1}], 'dead_nodes': [5]}, {'ntp': {'ns': 'kafka', 'topic': 'topic-1', 'partition': 0}, 'topic_revision': 39, 'replic     as': [{'node_id': 5, 'core': 1}], 'dead_nodes': [5]}, {'ntp': {'ns': 'kafka', 'topic': 'topic-17', 'partition': 1}, 'topic_revision': 59, 'replicas': [{'node_id': 5, 'core': 1}], 'dead_nodes': [5]}]}

Backports Required

• none - not a bug fix
• none - this is a backport
• none - issue does not exist in previous branches
• none - papercut/not impactful enough to backport
• v23.2.x
• v23.1.x
• v22.3.x

Release Notes

Features

• Adds the ability to force recover all partitions from a set of nodes (aka node wise recovery). This bulk recovers all partitions that lost majority because the nodes holding the majority replicas are no longer available.

[bharathv]

/dt

[vbotbuildovich]

new failures in https://buildkite.com/redpanda/redpanda/builds/42572#018c5c79-699d-4e7f-95f6-317755a710a1:

"rptest.tests.cluster_config_test.ClusterConfigAliasTest.test_aliasing_with_upgrade.wipe_cache=False.prop_set=PropertyAliasData.primary_name=.log_retention_ms.aliased_name=.delete_retention_ms.redpanda_version=.23.3.test_values=.1000000.300000.500000"

new failures in https://buildkite.com/redpanda/redpanda/builds/42572#018c5c79-69a4-4294-9b51-0d811376a80f:

"rptest.tests.cluster_config_test.ClusterConfigAliasTest.test_aliasing_with_upgrade.wipe_cache=True.prop_set=PropertyAliasData.primary_name=.log_retention_ms.aliased_name=.delete_retention_ms.redpanda_version=.23.3.test_values=.1000000.300000.500000"

[bharathv]

/dt

[mmaslankaprv]

maybe we can return a validation_err here ?

[bharathv]

this was intentional because if a user makes a get_majority_lost_partitions() they don't expect an error unless something concurrently modified the state while processing the request (hence concurrent_modification_error).. this validation_error is only possible in that situation.

[mmaslankaprv]

just two comments, otherwise looks good

[bharathv]

/dt

[vbotbuildovich]

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/42670#018c6338-c294-4847-8836-7c0c7258f6b6