Log messages rate limit

[SaiProServ]

Issue Summary:

On the 27th of September, 2023, a bug was reported to multiple core team members involving an excessive logging issue that resulted in a server outage. The detailed log message is as follows: ( message have WARN level, however is FATAL )

{"timestamp":"2023-09-27T00:13:01.143965Z","level":"WARN","fields":{"message":"consensus is active, not shutting down","fatal_ann":"fatal error [node/src/reactor/main_reactor/control.rs:199]: attempt to deactivate inactive era era 10894"},"target":"casper_node::reactor::main_reactor"}

This issue led to a rapid accumulation of log files, consuming all available disk space and subsequently causing the server to crash. Recently, I was informed by a team member, Joe, that this issue was addressed in release 1.5.6. While this is commendable, it appears that the broader implications of the problem were not fully addressed.

Further details of the issue can be checked at the below link:
https://github.com/casper-network/casper-node/issues/4536

[marc-casperlabs]

Duplicate of #4536