Harden recovery from metastore overload

[fmassot]

Description

A production metastore overload exposed several recovery paths that could keep a cluster degraded long after the initial database pressure had subsided.

See Metastore overload and cluster recovery for the trigger, the fix-by-fix outage mechanics, regression coverage, and remaining limitations.

This change hardens recovery across the control plane, indexing scheduler, ingest router, and metastore server:

• Scope periodic control-plane work, watchers, and delayed shard rebalances to a supervisor generation; clear readiness before teardown and ignore stale callbacks.
• Reschedule the control loop after pre-dispatch TooManyRequests errors instead of respawning the control plane.
• Replay unchanged indexing plans to a new node generation, bound plan-application waits, and preserve draining-node plans safely.
• Track unavailable shard leaders per index source, split control-plane requests by their source-specific exclusion sets, and clear stale routes after NoShardsAvailable responses.
• Share one load-shed semaphore across all generated metastore RPC methods. For PostgreSQL, admit at most max_connections - 1 unary RPCs when possible so readiness and control work retain connection-pool headroom during lock convoys.

The PostgreSQL admission limit is intentionally best-effort rather than a hard reservation: streaming responses and direct pool users do not hold this generic RPC permit for their entire database lifetime. SQLx pool-acquisition timeouts remain conservatively classified as uncertain database errors.

How was this PR tested?

• cargo test -p quickwit-common --lib (132 passed)
• cargo test -p quickwit-ingest --lib -- --skip test_ingester_persist_replicate_grpc (174 passed, 1 ignored)
• cargo test -p quickwit-control-plane --lib (128 passed)
• SSL_CERT_FILE=/etc/ssl/cert.pem cargo test -p quickwit-serve --lib (165 passed)
• cargo clippy -p quickwit-common -p quickwit-ingest -p quickwit-control-plane -p quickwit-serve --lib --tests -- -D warnings
• Nightly rustfmt --check on all eight changed Rust files

Regression coverage includes generation-owned task cancellation, readiness transitions, stale callback rejection, recoverable overload handling, generation-aware plan replay, source-scoped routing recovery, closed-shard feedback sent exactly once across grouped requests, and shared cross-method metastore admission with permit release after cancellation.