fix: gw transient errors crash

[saurav-malani]

Description

PR modifies maintenance loop in jobsdb to make handling of transient errors configurable.

Notion Ticket

https://www.notion.so/rudderstacks/Gateway-shall-not-panic-on-transient-DNS-errors-ff53bcb47f6045b1a409aecea927841c?pvs=4

Security

• The code changed/added as part of this pull request won't create any security issues with how the software is being used.

[codecov]

Codecov Report

Patch coverage: 43.10% and project coverage change: -0.16 ⚠️

Comparison is base (9e32802) 68.47% compared to head (6c099cf) 68.31%.

❗ Current head 6c099cf differs from pull request most recent head 3eda7a8. Consider uploading reports for the commit 3eda7a8 to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #3397      +/-   ##
==========================================
- Coverage   68.47%   68.31%   -0.16%     
==========================================
  Files         330      330              
  Lines       53042    53114      +72     
==========================================
- Hits        36319    36284      -35     
- Misses      14358    14433      +75     
- Partials     2365     2397      +32     

Impacted Files	Coverage Δ
app/apphandlers/gatewayAppHandler.go	14.28% <0.00%> (-0.33%)	⬇️
app/apphandlers/processorAppHandler.go	9.05% <0.00%> (-0.20%)	⬇️
jobsdb/unionQuery.go	88.03% <ø> (ø)
jobsdb/backup.go	70.39% <31.03%> (-4.65%)	⬇️
jobsdb/migration.go	70.31% <33.33%> (-5.61%)	⬇️
router/batchrouter/handle.go	62.91% <33.33%> (-0.28%)	⬇️
jobsdb/jobsdb_utils.go	89.28% <50.00%> (-3.09%)	⬇️
jobsdb/jobsdb.go	72.50% <55.11%> (-1.47%)	⬇️
app/apphandlers/embeddedAppHandler.go	77.69% <100.00%> (+0.42%)	⬆️

... and 5 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

[atzoum]

waiting to review changes after rebasing lint fixes introduced in master

[atzoum]

waiting to review changes after rebasing lint fixes introduced in master