Multi-Factor-Based Balance

[djshow832]

Feature Request

Describe your feature request related problem

It may be inefficient to merely balance based on health and connection counts:

• The workload of each connection may be different: some may be busy while others may be idle. We have seen cases when connection counts are balanced while CPU usage is not.
• When some big queries make a TiDB OOM, all the connections on the TiDB are disconnected. Actually, connections without big queries have a chance to be migrated away before the TiDB fails.
• In multi-AZ deployment, the cross-AZ traffic between TiProxy and TiDB may lead to significant fees. It's more cost-effective for TiProxy to route to the local TiDB preferentially.

Describe the feature you'd like

Balance TiDB instances based on more factors to maximize resource usage and reduce costs.

Tasks

Design

• RFC: Multi-Factor-Based Balance #466

Factor Collection

• Speed up health check #216
• Collect metrics from TiDB #499
• Remove enum BackendStatus #513
• Decouple factor with router #517
• Check SQL port until TLS handshake #577
• Match metric label with backend address #527
• Optimize the time point of querying Prometheus #562

Balance

• Split factors from score-based router #511
• Support error-based balance #543
• Support memory-based balance #540
• Support CPU-based balance #519
• Support label-based balance #538
• Support location-based balance #545
• Limit the speed of rebalance #547
• Record the score history to stabilize balance

Configuration

• Add the config for labels #535
• Add the config for enabling factors #551

Observability

• Add metrics of cross-az traffic and connection migration factors #555
• Add API for querying backend factors
• Add API for querying balance events

Cloud Adaptation

• Query metrics from backends #596
• Define query rules for each factor #598
• Query metrics from metrics reader owner #600
• Support reading from both Prometheus and backends #605
• Elect a metrics reader owner from each zone
• Show the metrics reader owner on Grafana