use prom server registry for load generator & adjust buckets

[longbowlu]

such that NetworkAuthorityClient's metrics can be reported to prom server

and adjust buckets again following https://prometheus.io/docs/practices/histograms/#errors-of-quantile-estimation

[andll]

Thanks for updating buckets @longbowlu!

[velvia]

1. Are we sure we don't need a bucket beyond 90?
2. I think it should be sufficient to use the same buckets as for 1-10, ie: 10 25, 50, 100
3. Technically, if you want to minimize the amount of error, the buckets should be strictly geometric.
   It doesn't really matter where the bucket boundaries are because Prom will linearly interpolate anyways.

The geometric series for 4 buckets between 10 and 100 would be:
10, 18, 32, 56, 100

The reason is that mathematically 10-20 is a much bigger gap (2x) vs between 50-60, thus there will be a much higher rate of relative error in the 10-20 bucket.

[velvia]

Also a global const would be good for LATENCY_SEC_BUCKETS. Maybe we should put some in Mysten-infra, maybe I'll do this.

[lxfind]

We may be interested in different buckets for different metrics.

[andll]

I thought little bit about it and yes, we don't need buckets beyond ~10s even I would say.

However, I also think that it is hard to come up with a decent set of buckets. I do have some PR at work that will try to calculate real pct rather than require to specify buckets initially. It is not going to be super trivial, but I think it should work. In the mean time I thought it would be ok to merge some improvement in bucket list in case we get another deployment to get some better numbers

[velvia]

Note that we already have automatic histograms for all spans. So we shouldn't need to explicitly add histograms where we already have spans - we should instead just add more spans. The span histograms use exponential buckets already.

This is preferred because we will need spans for tracing anyways.

I'll go through and clean this up when I have a chance.