RocksDB improvements on live node

[jbajic]

Introduction

This is a continuation of RocksDB configuration improvements from #9297

Tasks

Fine-tuning metrics on the live node

• Fix block cache size for the State column feat: Block cache change for State col #9388
• Add changes for FlatState in the RocksDB configuration feat: Change block options for FlatState column #9389
• Test block size on live nodes with/without compaction
  • This will be tested for a longer period of time

Metrics

• Add new metrics for RocksDB metrics similar to what we track in state-perf feat: Add rocksdb performance metrics #9555
• Add metric for the number of ios that RocksDB performs per request => added in Grafana

Based on new metrics

This will be only planned to be started in August since it is difficult to say what exactly will be done.

• if the number of ios per request is bigger than e.g. 1.5 there could be room for improvement
• if compaction causes big spikes ⇒ need for improvement

[walnut-the-cat]

Can we update the issue so we can close it?

[walnut-the-cat]

Plan to wrap up this week and pivot towards building a prototype of a short term solution to address storage performance issue (dedicated cache). Release of the RocksDB performance improvement PR are directly related to ideas we are exploring for stateless validation and whether to merge will be decided based on the final conclusion.

[walnut-the-cat]

RocksDB related work is wrapped up. We are waiting for one PR(#9389) to be reviewed so we can merge, but otherwise, we concluded that most of meaningful works are done.