Fog/Mobilecoind More efficient logging for upgrades.

[jgreat]

Right now fog services that use mobilecoind to sync the ledger generate log entries for each block processed. Since the ledger is approaching 1M+ blocks this creates several million log entries on each application instance when we do upgrades or blue/green style deployments.

This puts additional load and on the infrastructure and adds cost to ship all these logs to our logging provider.

No idea on what the level of effort is for these but, some ideas to reduce volume of logs during startup.

• Reduce level to ERROR during initial sync and reset to LOG_LEVEL when service is healthy.
• Don't log successful archive retrieved for each node, just log once for each block.
• Don't log each node/block when doing roll up blocks just log success on the batch.

[jgreat]

The main offender is Archive block retrieved for Url. We get a message for each block * 10 nodes.

2023-02-15 23:54:55.603855312 UTC INFO Archive block retrieved for Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("bd-mobilecoin-ledger.s3.amazonaws.com")), port: None, path: "/binance.mobilecoin.bdnodes.net/", query: None, fragment: None } 1282651, mc.app: mobilecoind, mc.module: mc_watcher::watcher, mc.src: watcher/src/watcher.rs:191

On start up, if we're back filling the ledger, logging messages by batch would be nice. Maybe 1 message for each 10k blocks processed until we catch up?

[jgreat]

These "error" logs from something deeper GRPC/HTTP stack are also unnecessary, any advice on how to mute these would be awesome.

E0324 19:08:30.384842702      34 hpack_parser.cc:1216]       Error parsing metadata: error=invalid value key=content-type value=text/html

[jcape]

@jgreat gRPC C-library errors need to be set using GRPC_VERBOSITY, see https://github.com/grpc/grpc/blob/master/TROUBLESHOOTING.md for more details. Otherwise, fixed in #3301