[ASI-976] [REPLICA] Fix StateMachineQueue unreliability + expose stateMachineQueue status via health check

[SidSethi]

Note - this is a copy of previous branch #2774 , opened new branch since this was easier than resolving all the merge conflicts while trying to remove the commits I reverted from master previously. Please see previous PR for discussion

Note - I also removed the stateMachineQueue redis lock since it ended up creating more issues. Specifically when a node restarts mid job, the lock is not released. I added code to ensure it is released on each init but for some reason that didn't work. Basically, not worth the hassle for what is in the end redundant logic.

Description

• remove timestamp suffix from bull queue generation so we don't end up with hundreds of diff queues in redis
• remove done() callback consumption in queue processing. this is actually incorrect pattern and should not be combined with async-await processing. (note - don't think this was really causing any errors, but unclear lol)
• track stateMachineQueueLatestJobSuccess and stateMachineQueueLatestJobStart + expose via monitors in health check, and configure health check to error if enforceStateMachineQueueHealth flag passed
• increase prod snapbackModuloBase to 48 (from 24) and decrease snapbackJobInterval to 30min (from 60min)
• add timeout to peerSetManager axios reqs
• Increase stateMachineQueue lockDuration from default 30s to 2x snapbackJobInterval to ensure job does not get prematurely marked as stalled
• set maxStalledCount from default 1 to 0 to ensure stalled jobs are never re-processed
• add stalled job logging to all Snapback queues (note - even though jobs have been getting marked as stalled, that does not seem to have stopped them from processing...its weird. either way, with above config changes, we should not see any more stalling)
• change stateMachineQueue from a setTimeout with manual re-add to a cron on snapbackJobInterval (note - have not confirmed whether this does anything, but lets see)
• track duration for each processStateMachineOperationDecisionTree stage + improve the logging code
• reduce snapback batch_clock_status request timeout from 20s to 10s

Tests

Automated CN tests should all still pass, but that doesn't help much with snapback testing.
For regression testing, confirmed via mad-dog with manualSyncsDisabled = true that content is still being synced correctly
The main validation here is that I've already released this to staging and have confirmed jobs are being correctly processed

How will this change be monitored? Are there sufficient logs?

Will wire up the new stateMachineQueueLatestJobSuccess health check field to a Pingdom alert to ensure jobs are being processed correctly

Instances of batch_clock_status req timeouts with log msg [retrieveClockStatusesForUsersAcrossReplicaSet] Could not fetch clock values for wallets=${walletsOnReplica} on replica=${replica} ${errorMsg.toString()}