Expose recovery, snapshot, and restore rate limits and throttle times in node stats

[kingherc]

Description

The rate limits for recoveries and snapshots can be indirectly computed from Elasticsearch configurations, which makes it hard to ascertain their real values used during runtime because:

• Adjust indices.recovery.max_bytes_per_sec according to external settings #82819 introduced a way to indirectly influence the recovery speed indices.recovery.max_bytes_per_sec by configuring three key bandwidth metrics settings.
• Snapshot recovery rate limit max_restore_bytes_per_sec is already capped by the recovery rate limit.
• Per-node rate limits for snapshots #57023 aims to tie and potentially cap the snapshot speed max_snapshot_bytes_per_sec to the recovery limit (when the node bandwidth metrics settings are configured) apart from the existing snapshot speed configuration.
• Furthermore, some of the aforementioned limits can be configured per node and/or repository and/or at runtime.

To make observability of these rate limits easier, the proposal is to expose the final used rate limit values (considering also any cap, e.g., by the recovery rate limit) in the node stats. Snapshot rate limits will be reported per repository.

Apart from the speed, the proposal is to also expose the throttling times more prominently. Specifically:

• The repos analysis API exposes the snapshot and snapshot restore throttle times. But this is not useful for live usage. We could expose the throttle times in node stats per repository.
• Recovery throttling is already exposed in nodes stats under node_id > indices > recovery > throttle_time. I think no change is needed for recovery throttling stats.

[elasticsearchmachine]

Pinging @elastic/es-distributed (Team:Distributed)