Feature Request: txthrottler to poll replication lag on PRIMARY only

[timvaillancourt]

Feature Description

Today a tablet with --enable-tx-throttler set will open healthcheck streams to other shard-tablets (in order to monitor replication lag) on every tablet, regardless of the tablet type 😬

This is inefficient because only the PRIMARY tablet receives writes that the throttler could reject with throttler errors

This issue requests that the txthrottler opens healthcheck streams only on PRIMARY tablets by adding a .MakePrimary(bool) method (similar to other tabletserver components) under *stateManager.servePrimary. All setup/shutdown of healthcheck streams will move under .MakePrimary(bool) and on demotion time the streams can be shut down using .MakePrimary(false) under *stateManager.unservePrimary

A potential consequence of this change is a delay in a newly-promoted PRIMARY receiving replication lag state from all replicas, equal to --healthcheck_interval, whereas today a newly-promoted PRIMARY would already have replication lag state. I feel this is an acceptable tradeoff that can be documented.

Use Case(s)

Users of txthrottler, ie: --enable-tx-throttler on vttablet