[Feature] Store exposes no shard recovery metrics (sync progress, raft lag) during disaster recovery

[bitflicker64]

Feature Description (功能描述)

Problem

When a Store node is rebuilt after data loss and rejoins the cluster, its shards recover through raft snapshot install plus log catch-up. On a graph of any real size this takes minutes. Under the current implementation, that progress is visible only in log lines: no exported metric reports how many partitions are still syncing, how far along a snapshot install is, or how far the local raft log trails the leader. This makes it hard for an operator to distinguish a healthy long recovery from a stuck one, or to estimate remaining time.

Operational observation from my own fault testing (not a code claim): after deleting a Store pod and its volume in Kubernetes and letting the StatefulSet rebuild it, the only available signals were pod readiness and log lines such as Raft {} begin loadSnapshot / Raft {} end loadSnapshot. Nothing on the Prometheus endpoint moved in a way that reflected recovery progress.

What the Store exports today (verified on master)

The Store node already runs a Spring actuator Prometheus endpoint. hg-store-dist/src/assembly/static/conf/application.yml (lines 22-30) enables the Prometheus meter registry and exposes all web endpoints, so /actuator/prometheus is served on the REST port. Meter families are registered in hg-store-node/src/main/java/org/apache/hugegraph/store/node/metrics/MetricsConfig.java (lines 32-46), which adds the common tag hg=store and initializes five sources:

• StoreMetrics (store/node/metrics/StoreMetrics.java, lines 51-79): hg.up, hg.graphs, and hg.partitions{graph=...} which is only a count of partitions per graph.
• JRaftMetrics (store/node/metrics/JRaftMetrics.java, lines 73-126): jraft.groups plus a generic re-export of whatever sofa-jraft NodeMetrics registers per raft group (latency timers, replicator counters, tagged group=<id>). These are low level jraft internals; nothing in them is partition aware, and nothing expresses "this group is installing a snapshot" or "this group is N entries behind".
• RocksDBMetrics (rocks.stats.* prefix, RocksDBMetricsConst.java line 32), ProcfsMetrics (process_memory.*), GRpcExMetrics (grpc.*).

I read every file in the store/node/metrics package on master: there is no metric family for partition work state, shard sync state, snapshot install progress, raft log lag, leader count per store, or a recovering-partition count.

Where recovery progress already lives in memory

The information needed for such metrics already exists in process; it is just never handed to the meter registry:

• Snapshot install: hg-store-core/src/main/java/org/apache/hugegraph/store/raft/HgStoreStateMachine.java, onSnapshotLoad (lines 216-244) knows the group id and the snapshot's last included index, and hg-store-core/src/main/java/org/apache/hugegraph/store/snapshot/HgSnapshotHandler.java, onSnapshotLoad (lines 165-203) walks the snapshot data directory while loading it into RocksDB. Start, end, and the file set (with sizes on disk) are all known there; today the only output is log lines (lines 177-180).
• Log catch-up: HgStoreStateMachine.onApply advances committedIndex on every applied entry (lines 105, 119-121), surfaced per group by PartitionEngine.getCommittedIndex (hg-store-core/src/main/java/org/apache/hugegraph/store/PartitionEngine.java, lines 718-720). The same value is already packaged per partition into Metapb.RaftStats for the PD heartbeat by hg-store-core/src/main/java/org/apache/hugegraph/store/metric/HgMetricService.java, getRaftMetrics (lines 114-124), so lag against the leader is computable from data the system already collects.
• Partition and shard state: HgMetricService.getStoreMetrics (lines 126-157) attaches each partition's workState and local shard role to the heartbeat, and HeartbeatService.partitionHeartbeat (hg-store-core/src/main/java/org/apache/hugegraph/store/HeartbeatService.java, lines 290-352) reports per-shard SState_Normal / SState_Offline. The proto already defines the exact states a recovery dashboard needs, including SState_Snapshot ("Install snapshots") in hg-pd-grpc/src/main/proto/metapb.proto (lines 251-259) and PartitionState (lines 111-121).
• Leader count: HgStoreEngine.getLeaderPartition (hg-store-core/src/main/java/org/apache/hugegraph/store/HgStoreEngine.java, lines 536-544) already computes the set of groups this node leads. Raft metrics are enabled on every group (PartitionEngine.java, lines 199 and 244).
• There is a JSON view of some of this (/v1/partitions in hg-store-node/src/main/java/org/apache/hugegraph/store/node/controller/PartitionAPI.java, lines 66-113, with leader, term, committed index, work state), but it is an ad hoc REST endpoint, not something Prometheus scrapes or alerts on.

On the PD side, StoreNodeService.heartBeat (hg-pd-core/src/main/java/org/apache/hugegraph/pd/StoreNodeService.java, lines 722-784) persists all of this per store, and TaskScheduleService.patrolStores (hg-pd-core/src/main/java/org/apache/hugegraph/pd/TaskScheduleService.java, lines 121-129 and 202-229) patrols store liveness every 60 seconds. Yet PD's own exporter (hg-pd-service/src/main/java/org/apache/hugegraph/pd/metrics/PDMetrics.java, lines 68-79) only publishes hg.up, hg.graphs, hg.stores, hg.terms, hg.partition.terms{id}, hg.partitions{graph}, and hg.graph.size{graph}. PD knows every store's state and every shard's role and does not export any of it either.

Proposal

Export a small set of recovery-oriented families on the existing actuator endpoint, following the current naming style (dotted micrometer names under the hg. prefix with the hg=store common tag, low-cardinality tags like the existing graph and group):

Store side, registered next to the existing gauges in StoreMetrics / a new RecoveryMetrics source in MetricsConfig:

• hg.partition.state{graph, partition, state} gauge: 1 when the partition's work state equals state (PState_Normal, PState_Warn, PState_Offline, PState_Fault), else 0. Values come straight from Partition.getWorkState(), already read in HgMetricService.getStoreMetrics.
• hg.partitions.recovering gauge: count of local partitions whose work state is not PState_Normal. Gives dashboards a single "still recovering" number per store.
• hg.raft.snapshot.installing{group} gauge: 1 between HgSnapshotHandler.onSnapshotLoad start and finish.
• hg.raft.snapshot.install.bytes{group} and hg.raft.snapshot.install.bytes.total{group} gauges: bytes loaded so far versus total size of the snapshot data directory, both observable inside onSnapshotLoad.
• hg.raft.log.lag{group} gauge: leader committed index minus local applied index, i.e. the distance still to replay after snapshot install. The local side is PartitionEngine.getCommittedIndex; the leader side is already shipped to PD in RaftStats.
• hg.raft.leader.count gauge: size of HgStoreEngine.getLeaderPartition(). During and after recovery this shows leadership draining back to the rebuilt node.

PD side, complementary and cheap because the data is already in StoreNodeService:

• hg.store.state{store, state} gauge: per registered store, 1 for the current StoreState (Up, Offline, Exiting, Tombstone, ...).
• hg.store.leader.count{store} gauge: leaders per store from the shard group table, so balance and recovery are visible even while the recovering store itself cannot serve its endpoint.

All of these are plain gauges over state the process already holds, so the cost is a handful of map lookups per scrape, in line with how StoreMetrics and PDMetrics work today.

Context

While testing distributed deployments on Kubernetes, I observed this during fault tests that kill a Store pod, drop its volume, and verify the cluster heals: grading "healed" today means watching readiness probes and grepping logs, which is exactly the gap described above. For reference, the deployment tooling used for these tests is the Helm chart contributed in PR #3132 (issue #3131); it already scrapes the standard actuator endpoints of Store and PD, so these families would light up recovery dashboards and alerts with no deployment changes at all. I am happy to work on a PR for the store-side gauges if the direction sounds right to the maintainers.