add sharding metrics

[zhangxu19830126]

User description

What type of PR is this?

• API-change
• BUG
• Improvement
• Documentation
• Feature
• Test and CI
• Code Refactoring

Which issue(s) this PR fixes:

issue #16438

What this PR does / why we need it:

add sharding metrics

---

PR Type

Enhancement, Tests

---

Description

• Added Prometheus metrics for various shard operations including AddReplica, DeleteReplica, and DeleteAllReplica.
• Integrated metrics into shard service operations to track replica counts and read operations.
• Created a new Grafana dashboard to visualize sharding metrics including replica counts, operations, and reads.
• Defined new Prometheus counters and gauges for tracking sharding metrics.

---

Changes walkthrough 📝

Relevant files

Enhancement

runtime.go Add metrics counters for shard operations.                              pkg/shardservice/runtime.go Added counters for AddReplica and DeleteReplica operations. Imported new metrics package. +7/-0      service.go Integrate metrics into shard service operations.                  pkg/shardservice/service.go Added metrics for replica count and operations in doTask and doHeartbeat functions. Added a method to count replicas in allocatedCache. +16/-0    service_read.go Add metrics for local and remote reads.                                    pkg/shardservice/service_read.go Added local and remote read counters in the Read function. +8/-0      grafana_dashboard.go Initialize sharding dashboard in Grafana.                                pkg/util/metric/v2/dashboard/grafana_dashboard.go Added initialization of sharding dashboard. +3/-0      grafana_dashboard_sharding.go Create Grafana dashboard for sharding metrics.                      pkg/util/metric/v2/dashboard/grafana_dashboard_sharding.go Created a new sharding dashboard with graphs for replica count, operators, and reads. +80/-0    sharding.go Define Prometheus metrics for sharding.                                    pkg/util/metric/v2/sharding.go Defined Prometheus metrics for replica operations and reads. Added a gauge for replica count. +52/-0

---

💡 PR-Agent usage:
Comment /help on the PR to get a list of all available PR-Agent tools and their descriptions

[codiumai-pr-agent-pro]

PR-Agent was enabled for this repository. To continue using it, please link your git user with your CodiumAI identity here.

PR Review 🔍

⏱️ Estimated effort to review [1-5]	3, because the PR involves multiple changes across various files including metric implementation, service logic adjustments, and dashboard configurations. The changes are moderate in complexity, involving both backend logic and monitoring aspects.
🧪 Relevant tests	No
⚡ Possible issues	Metric Initialization: The metrics are used directly in the service files, but there is no explicit registration shown in the PR for these metrics with Prometheus. This could lead to runtime errors if the metrics are not properly registered.
	Error Handling: In the new metric increment statements, there is no error handling or logging. If any of these operations fail or produce errors, it might be silent at runtime, which could complicate debugging and operational monitoring.
🔒 Security concerns	No

[codiumai-pr-agent-pro]

PR-Agent was enabled for this repository. To continue using it, please link your git user with your CodiumAI identity here.

PR Code Suggestions ✨

Category	Suggestion	Score
Best practice	Register the Prometheus metrics during the package initialization To ensure that the metrics are registered with Prometheus, add a function to register the counters and gauges, and call it during the initialization of the package. pkg/util/metric/v2/sharding.go [1-52] var ( replicaOperatorCounter = prometheus.NewCounterVec( prometheus.CounterOpts{ Namespace: "mo", Subsystem: "sharding", Name: "schedule_replica_total", Help: "Total number of replica schedule.", }, []string{"type"}) AddReplicaOperatorCounter = replicaOperatorCounter.WithLabelValues("add") DeleteReplicaOperatorCounter = replicaOperatorCounter.WithLabelValues("delete") DeleteAllReplicaOperatorCounter = replicaOperatorCounter.WithLabelValues("delete-all") replicaReadCounter = prometheus.NewCounterVec( prometheus.CounterOpts{ Namespace: "mo", Subsystem: "sharding", Name: "replica_read_total", Help: "Total number of replica read.", }, []string{"type"}) ReplicaLocalReadCounter = replicaReadCounter.WithLabelValues("local") ReplicaRemoteReadCounter = replicaReadCounter.WithLabelValues("remote") ) var ( ReplicaCountGauge = prometheus.NewGauge( prometheus.GaugeOpts{ Namespace: "mo", Subsystem: "sharding", Name: "replica_count", Help: "Count of running replica.", }) ) +func init() { + prometheus.MustRegister(replicaOperatorCounter, replicaReadCounter, ReplicaCountGauge) +} + Suggestion importance[1-10]: 9 Why: Registering Prometheus metrics during package initialization is a best practice to ensure that metrics are available as soon as the package is used, enhancing the reliability of metric reporting.	9
Possible issue	Add a default case to handle unexpected operation types in the switch statement To avoid potential issues with uninitialized or nil values in op.Type, it would be prudent to add a default case in the switch statement to handle unexpected operation types. pkg/shardservice/runtime.go [556-561] switch op.Type { case pb.OpType_AddReplica: v2.AddReplicaOperatorCounter.Inc() case pb.OpType_DeleteReplica: v2.DeleteReplicaOperatorCounter.Inc() +default: + c.logger.Warn("unexpected operation type", zap.String("type", string(op.Type))) } Suggestion importance[1-10]: 8 Why: Adding a default case to handle unexpected operation types is crucial for robust error handling and logging, preventing potential runtime errors.	8
	Add a mutex lock to ensure thread-safe access to the s.values map in the replicasCount method To ensure that the replicasCount method is thread-safe, consider using a mutex to protect access to the s.values map, as it could be modified concurrently. pkg/shardservice/service.go [752-758] func (s *allocatedCache) replicasCount() int { + s.mu.Lock() + defer s.mu.Unlock() n := 0 for _, v := range s.values { n += len(v.Replicas) } return n } Suggestion importance[1-10]: 8 Why: Ensuring thread safety in concurrent environments is critical, especially when accessing shared resources like maps. The suggestion correctly identifies a potential race condition.	8
Maintainability	Extract the local and remote replica counting logic into a separate function for better readability To improve readability and maintainability, consider extracting the local and remote replica counting logic into separate functions. pkg/shardservice/service_read.go [81-112] -local := 0 -remote := 0 -for i, shard := range selected.values { - if s.isLocalReplica(shard.Replicas[0]) { - selected.local = append(selected.local, i) - local++ - continue - } - - remote++ - if opts.adjust != nil { - opts.adjust(&shard) - } -} +local, remote := countReplicas(selected.values, s.isLocalReplica) v2.ReplicaLocalReadCounter.Add(float64(local)) v2.ReplicaRemoteReadCounter.Add(float64(remote)) +func countReplicas(values []Shard, isLocal func(Replica) bool) (int, int) { + local, remote := 0, 0 + for i, shard := range values { + if isLocal(shard.Replicas[0]) { + selected.local = append(selected.local, i) + local++ + continue + } + remote++ + if opts.adjust != nil { + opts.adjust(&shard) + } + } + return local, remote +} + Suggestion importance[1-10]: 7 Why: Extracting complex logic into separate functions improves code readability and maintainability, making it easier to manage and understand the codebase.	7