GetValidators e2e benchmarks

[abi87]

Why this should be merged

It shouldn't.This PR mods avalanchego to try and benchmark the average cost of GetValidatorSet calls.

How this works

Just drop GetValidatorsSet caches to hit disk at every request.

How this was tested

1. Deploy branch and make sure prometheus monitoring is in place (see here)
2. Sync node.
3. GetValidators is "naturally" called by proposervm verification (see here) for heights down to 256 below last accepted block height (see here). In order to get measurements for deeper heights, we can use a script to hit deeper heights like warp_measures_script.zip, which hits platform.GetValidatorsAt API.
4. The warp measures script will log the heights requested via `platform.GetValidatorsAt, tagged with a timestamp for when call was issued. We can download an estimate of the time taken to process the request via Prometheus, by issuing the call

curl -g '<Node_IP>:9090/api/v1/query_range?query=rate(avalanche_P_vm_validator_sets_height_diff_sum[30s])&start=<Measures_Start_Time>&end=<Measures_End_Time>&step=30s'

5. Finally with minor post processing, we can get the request_time vs requested_height plot

[abi87]

Below shown the "natural activity" around GetValidatorsSet, as required by proposerVM.
Note that no cache in available to server requests, so diffs are applied for every single request.
First plot shows how deep the diff goes (lastAcceptedBlockHeight - requestedHeight).
Second plots show time required to satisfy the request (in milliseconds).
Not that plots don't show how "populated" a diff is.

[abi87]

Again below the "natural activity" around GetValidatorsSet, as required by proposerVM. Change in metric (Gauge.Set instead of Gauge.Add) allow tracking single requests features (diff depth and duration).
Note that no cache in available to server requests, so diffs are applied for every single request.
First plot shows how deep the diff goes (lastAcceptedBlockHeight - requestedHeight).
Second plots show time required to satisfy the request (in milliseconds).
The depth is less than 256 which is the maximum windows size we allow.

[abi87]

@aaronbuchwald @StephenButtolph the plot of a bunch of GetValidatorsSet calls on a mainnet node, showing diff depths (x-axis) and their measured durations (y-axis, in microseconds).
I got the plot by merging two rounds of measurements on a mainnet node, with a simple script issuing calls to platform.GetValidatorsAt.
A simple linear regressions on the data gives: duration = 161 * depth - 2629us

[aaronbuchwald]

Note: this does still hit one cache here: https://github.com/ava-labs/avalanchego/blob/modded_platformVM_for_validators_set_benchmarks/vms/platformvm/state/state.go#L981, but the size is only 2048, so performing an e2e measurement that goes well past that should make its impact minimal.

Since we're just using this to test, I'll go ahead and close this PR.