Adding counters to the state for metrics

[ruuda]

It would be nice if we could eventually expose some metrics and graph them, perhaps through a little daemon that queries the Solana RPC for the Lido state, and exposes it in Prometheus format.

A few metrics that would be nice to have are

• solido_deposited_lamports_total, sum of of all deposit ever. From this we can compute e.g. amount staked per day.
• solido_withdrawn_lamports_total, sum of all withdrawals ever.
• solido_deposits_total, monotonically increasing counter of the number of deposits since inception, so we can compute e.g. number of deposits per day.
• solido_withdrawals_total, monotonically increasing counter of the number of withdrawals since inception, so we can compute e.g. number of withdrawals per day.
• Together the above metrics also give us the average amount per deposit and withdrawal, but to gain more insight, we could use a histogram. Prometheus histograms include a total sum and a count, so this subsumes the above metrics. The tricky thing is that you have to decide on the histogram buckets up front.
• solido_fee_st_sol_total, the total fees paid since inception, possibly split out by post into treasury, insurance, validation fees, etc.

Aside from this we would want the current pool balance and such, but these we can just query from the Lido state already, we do not need to add counters to the state for this.

A note about why to track withdrawals and deposits separately: without this, if you see the pool balance increase by 10 SOL, you can’t distinguish between a single 10 SOL deposit, and a 1000 SOL withdrawal + a 1010 SOL deposit.

[malikankit]

Is this something @hritique can help with - assuming he has bandwidth beyond the frontend?

[ruuda]

Is this something @hritique can help with

I don’t know, but everything that needs to happen for this would be written in Rust and live either in the on-chain program or in a small daemon that can deserialize the on-chain state and expose a /metrics endpoint over http. I think it would be easiest to write that in Rust as well, because the deserializers and Solana client are available there.

Once we can expose the metrics, we need to scrape them, and then graph them. I am familiar with Prometheus and Grafana, but other options exist. @joe-bowman is this something that the platforms team could help with? I’d be happy to explore this myself too if you can point me in the right direction.

If we also want to show these metrics over time on the Lido page, and not just on an internal dashboard, then we would need a way to expose the historical data over http, and a way to graph it in the frontend. I think @hritique could help with that, but as a first step, I was only proposing this for internal use so we have some insight into what’s going on.

[joe-bowman]

He already have the full prometheus, grafana and alert manager stack set up; provide us something with a /metrics endpoint and we'll get this deployed and hooked up to prometheus.

[glottologist]

@RishiSid this seems related to the discussion we had with Xavier last regarding the analytics.

[ruuda]

I’m going to increase the scope of this issue a bit to be about metrics in general, not just those that we need to embed counters in the state for. One thing that would be nice to track for the maintenance bot:

• spent_lamports_total: Number of Lamports spent on transaction fees since the bot process started running

Then we get a handle on how much it costs to operate the bot, even if the cost is expected to be low.

[ruuda]

I just had a call with @hritique, about the data that should go on the widget, and in addition to the exchange rate, we want the following data on there as well:

• Total amount of SOL deposited since inception
• Total amount of rewards received since inception

The widget needs to get this data from somewhere, and we agreed that it is easiest to get this from the Solido state, if it is available there. This aligns nicely with the plan above about keeping the counters in the state. The maintenance bot can extract the data and make it available to Prometheus, and the widget can extract the data and display it on the website.

I already proposed solido_deposited_lamports_total so that will give us the amount deposited. For the rewards, we could do

• solido_observed_reward_lamports_total, for the raw rewards observed, before the fee cut.
• solido_fee_lamports_total, for the fee cut in lamports. (So if the fee cut is never changed, solido_fee_lamports_total = solido_observed_reward_lamports_total * fee_fraction should hold.)
• The amount of SOL rewards that went to users is then the difference of those two.

[ruuda]

After #213, it should be very easy to add new metrics to the state and expose them:

• Add fields to the Lido struct to track whatever we want to track.
• Format them in maintenance::SolidoState::write_prometheus, which has access to a deserialized Lido struct.

I think it would be nice to track amount of SOL deposited as a histogram. (Maybe with buckets from le=0.001 SOL up to le=1000.0 SOL in 10× increments, that would cost us 8 u64s and one for the count.) This will give us more insight than just a total. Sure, we could also get this from the block explorer, but I don’t expect anybody to automate that.

[ruuda]

I’ll start with this now, as all of the more important issues for the v1 milestone are done.