Fix object_count metric when grouping is on

[etiennedi]

What's being changed:

• Please see object_count metrics reports nonsense values when prom grouping is active #4396 for a description of what issue is being addressed here
• This PR fixes this by making the object count async and iterate over all collections and shards every 30s.
• With the new .ObjectCountAsync that was introduced lately getting the object count is very cheap – even with 10s of thousand of shards. This is because it only ever hits disk segments which have a pre-computed object count. In the past we would also hit memtables which relied on a very expensive calculation to find out if a write is an update or a new insertion.
• This PR makes the object_count metric async (same as /v1/nodes API). In a sense it's "double async", first the object count itself only reflects already flushed segments, then there is a loop that loops over all collections and all shards at a regular interval
• The eventually consistent (async) behavior should be acceptable for monitoring purposes where everything is async (metric collection, scraping, etc.)
• Note that no behavior is altered if PROMETHEUS_MONITORING_GROUP is not active, in this case, we can just keep relying on the flush event to update the metric per shard.
• This PR also respects lazy shard loading. It adds an allShardsReady flag to each collection. The async monitoring cycle is only started when all shards of all collections are ready
• fixes object_count metrics reports nonsense values when prom grouping is active #4396

Review checklist

• Documentation has been updated, if necessary. Link to changed documentation:
• Chaos pipeline run or not necessary. Link to pipeline:
• All new code is covered by tests where it is reasonable.
• Performance tests have been run or not necessary.

[aliszka]

LGTM

[sonarcloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarCloud