[fix][broker] Topic policies loading with timeout and retries

[programmerahul]

Fixes [#25294]

Motivation

While loading the system topic policies cache in broker, there is no timeout and retries. Due to this, there are cases observed , where the reader is stuck at some point and so the topic policies never gets loaded. So the topics become unavailable.

Modifications

This pr adds configurable timeout and retries for loading topic policies cache.

Verifying this change

• Make sure that the change passes the CI checks.

(Please pick either of the following options)
This change added tests and can be verified as follows:

• added test to check retries when a stuck reader is present
• added test to check unloading topics after all retries.

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

• Dependencies (add or upgrade a dependency)
• The public API
• The schema
• The default values of configurations
• The threading model
• The binary protocol
• The REST endpoints
• The admin CLI options
• The metrics : Metric gets emitted when topic policies load fails
• Anything that affects deployment

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

Matching PR in forked repository

PR in forked repository: programmerahul#1