[VTAdmin API] Cache cluster.GetTablets call for experimental tablet debug vars

[notfelineit]

Right now, if a user keeps the workflow streams UI open, we make a lot of calls to VTAdmin API's experimental debug tablet vars endpoint.

This endpoint first makes a GetTablets request to vtctld, and uses the result to put together tablet URL in order to make an HTTP request to the tablet's API to get the debug vars.

The issue is that all the cluster.GetTablets calls cause great strain on a cluster's vtctld.

We should cache these calls since they rarely change, to avoid hammering vtctld.

Links:

• api server's GetTablet method: https://github.com/vitessio/vitess/blob/main/go/vt/vtadmin/api.go#L1198

[deepthi]

There are a couple of other things to check

• does each API call from within vtadmin create a new grpc client that then connects to vtctld?
• how long are these clients kept around? they should be closed as soon as the response is received.

cc @ajm188

[ajm188]

There are a couple of other things to check

• does each API call from within vtadmin create a new grpc client that then connects to vtctld?

No. VTAdmin has one connection to a vtctld, process-wide, through which all RPCs go. This is managed by the resolver, so reconnects are managed by grpc-go if it's detected connectivity issues.

• how long are these clients kept around? they should be closed as soon as the response is received.

No, we should keep it around, because constant setup/tear down of the underlying connections will strain both the vtctld and vtadmin. That would manifest similarly to what I fixed in #8368

[ajm188]

We should instead I think rework the way the web side makes api requests to get the tablet vars so we are only loading what's being currently viewed (vs what the user could view by switching to another tab for ex) so it's only requesting what it needs for the given context

[notfelineit]

@ajm188 we can do that, and explore caching if that still isn't sufficient.