feat: Add logs during ledger replay

[GabrielePicco]

Description

Currently, when a validator is restarted, there are no logs available during the ledger replay process. This lack of visibility makes it difficult to track progress or diagnose potential issues.

Proposed Solution

Introduce info-level logs to provide visibility into the ledger replay process. Additionally, implement a progress tracker to indicate the operation's status. Since ledger replay can be time-consuming and no other operations occur during restart, this information would significantly improve monitoring and troubleshooting.

[thlorenz]

Log Level

Putting those logs on info would defeat the purpose of log levels since then I cannot turn them off easily even if I want just very important logs (instead of tracing ledger replay).

Instead I recommend setting the RUST_LOG var as follows:

export RUST_LOG=warn,magicblock=info,magicblock_ledger=trace,magicblock_ledger::blockstore_processor::blockhash_log=off

This will trace it to a decent level to where you can see progress.

If you want excruciating details about each blockhash that is processed you can remove the magicblock_ledger::blockstore_processor::blockhash_log=off part.

Progress Tracker

Via Logs

I could add some sort of update on info or debug if you just want to see progress, i.e. every 100 blocks or so it'd print Processed x/y blocks or similar. Keep in mind though that those would disappear among the more detailed logs should you already trace ledger output.

Via Metrics

We could also add some sort of metric that captures replay progress that you could see in the dashboard, however that would require having grafana setup which might not be ideal.

Via Separate File

Finally (a bit hacky but very useful) we could keep writing progress like x/y into a specific tmp file which you could watch from the terminal.