chore: introduced log & stats for retry of jobsDB execute & query methods

[saurav-malani]

Description

Here we have introduced logs & metrics for retry of query & execute methods of jobsDB. This is done by passing notify func while calling RetryWithNotify & QueryWithRetriesAndNotify. notify is called whenever an attempt fails.

Notion Ticket

https://www.notion.so/rudderstacks/Introduce-logs-and-alerts-for-jobsDB-retries-f20f44ddb5a7492a9c99fa49dba5cf3c

[codecov]

Codecov Report

Merging #2434 (a84216c) into master (12d3e59) will decrease coverage by 0.01%.
The diff coverage is 40.00%.

@@            Coverage Diff             @@
##           master    #2434      +/-   ##
==========================================
- Coverage   36.20%   36.18%   -0.02%     
==========================================
  Files         200      200              
  Lines       39663    39680      +17     
==========================================
  Hits        14360    14360              
- Misses      24399    24418      +19     
+ Partials      904      902       -2     

Impacted Files	Coverage Δ
jobsdb/jobsdb.go	68.80% <ø> (+0.29%)	⬆️
router/batchrouter/batchrouter.go	33.63% <17.07%> (-0.17%)	⬇️
services/multitenant/tenantstats.go	83.00% <28.57%> (-0.84%)	⬇️
processor/processor.go	71.26% <38.88%> (-1.16%)	⬇️
router/router.go	67.24% <52.63%> (-0.36%)	⬇️
utils/misc/retry.go	77.77% <93.33%> (+5.68%)	⬆️
config/backend-config/backend-config.go	69.88% <0.00%> (+0.56%)	⬆️
enterprise/reporting/reporting.go	9.82% <0.00%> (+1.44%)	⬆️
config/backend-config/namespace_config.go	78.62% <0.00%> (+2.29%)	⬆️
... and 1 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.