fix(db): Fix write stalls in RocksDB (again)

[slowli]

What ❔

RocksDB write stalls are still happening, this time for a different reason. Previously, they were caused by too many immutable memtables, this time – by too many level-0 SST files. This PR:

• Tunes RocksDB options some more (the main tuning point is optimizing level-style compaction).
• Increases the number of retries on stall and introduces exponential backoff.
• Introduces a dozen of RocksDB metrics that should help monitoring RocksDB health.

Why ❔

Having write stalls leads to panics and is obviously bad.

Checklist

• PR title corresponds to the body of PR (we generate changelog entries from PRs).
• Tests for the changes have been added / updated.
• Documentation comments have been added / updated.
• Code has been formatted via zk fmt and zk lint.

[codecov]

Codecov Report

Attention: 35 lines in your changes are missing coverage. Please review.

Comparison is base (4a5c54c) 34.40% compared to head (5d2db3b) 33.86%.
Report is 4 commits behind head on main.

❗ Current head 5d2db3b differs from pull request most recent head 5b03f1c. Consider uploading reports for the commit 5b03f1c to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #265      +/-   ##
==========================================
- Coverage   34.40%   33.86%   -0.54%     
==========================================
  Files         519      522       +3     
  Lines       28086    27958     -128     
==========================================
- Hits         9662     9468     -194     
- Misses      18424    18490      +66     

Files	Coverage Δ
core/bin/external_node/src/main.rs	0.00% <ø> (ø)
core/lib/config/src/configs/database.rs	66.66% <100.00%> (+3.03%)	⬆️
core/lib/merkle_tree/src/storage/rocksdb.rs	67.64% <100.00%> (-23.13%)	⬇️
...ore/lib/zksync_core/src/metadata_calculator/mod.rs	88.00% <100.00%> (+0.24%)	⬆️
...lib/zksync_core/src/metadata_calculator/updater.rs	94.79% <100.00%> (+1.19%)	⬆️
core/bin/external_node/src/config/mod.rs	22.22% <0.00%> (-0.86%)	⬇️
...lib/zksync_core/src/metadata_calculator/helpers.rs	87.09% <66.66%> (ø)
core/lib/storage/src/metrics.rs	26.66% <0.00%> (-1.91%)	⬇️
core/lib/storage/src/db.rs	45.81% <41.50%> (-2.71%)	⬇️

... and 75 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[slowli]

On the stage env (if I read RocksDB logs correctly), the latest write stalls were caused by compaction that ran just a bit too long (~2s) to be covered by retries. Since now the interval before the last retry is ~1.9s, it could fix the problem on its own. One thing that we could try is to check whether the writes are stopped immediately after DB initialization, and if they are, wait until writes become unstuck (maybe, with an upper cap like 10s), folding this time into RocksDB initialization. A problem with this is approach – if I read RocksDB logs correctly – is that writes are stopped 0.5s after the DB is initialized, not immediately (not sure why). I've misread the logs; writes are stopped immediately, so implementing this logic makes sense.