Storage broker targets two issues:
- Allowing safekeepers and pageservers learn which nodes also hold their timelines, and timeline statuses there.
- Avoiding O(n^2) connections between storage nodes while doing so.
This is used
- By pageservers to determine the most advanced and alive safekeeper to pull WAL from.
- By safekeepers to synchronize on the timeline: advance
remote_consistent_lsn,backup_lsn, choose who offloads WAL to s3.
Technically, it is a simple stateless pub-sub message broker based on tonic (grpc) making multiplexing easy. Since it is stateless, fault tolerance can be provided by k8s; there is no built in replication support, though it is not hard to add.
Currently, the only message is SafekeeperTimelineInfo. Each safekeeper, for
each active timeline, once in a while pushes timeline status to the broker.
Other nodes subscribe and receive this info, using it per above.
Broker serves /metrics on the same port as grpc service.
grpcurl can be used to check which values are currently being pushed:
grpcurl -proto broker/proto/broker.proto -d '{"all":{}}' -plaintext localhost:50051 storage_broker.BrokerService/SubscribeSafekeeperInfo